Medical image registration method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention discloses a medical image registration method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of extracting a fixed feature map from a fixed image and extracting a moving feature map from a moving image through a feature extraction module; the image registration model comprises a feature extraction module, a forward deformation field prediction branch, a reverse deformation field prediction branch and a registration module; processing the fixed feature map through a reverse deformation field prediction branch to obtain a reverse deformation field; processing the reverse deformation field and the moving feature map through a forward deformation field prediction branch to obtain a target forward deformation field; and performing registration processing on the target forward deformation field and the moving image through a registration module to obtain a registration image. According to the embodiment of the invention, the prediction branches of the forward deformation field and the reverse deformation field are arranged in the model, and the reverse constraint and estimation information are added into the prediction branch of the forward deformation field, so that the forward deformation field with better reversibility is obtained, and the image registration precision is higher.

Description

Technical Field

[0001] This application belongs to the technical field of medical image processing, and particularly relates to a medical image registration method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Medical image registration can accurately spatially align medical images acquired at different times, from different perspectives, or by different imaging devices, aiming to make the same or corresponding anatomical structures in the images accurately correspond spatially, so as to achieve more effective comparison and comprehensive analysis. It is widely used in various clinical medical scenarios such as surgical planning, auxiliary diagnosis, and pathological analysis.

[0003] In traditional medical image registration methods, it is usually necessary to perform iterative optimization for each pair of images to solve the deformation field, and then use the deformation field to perform registration processing on the moving image to obtain the registered image. However, this iterative calculation method has a high computational cost.

[0004] With the continuous development of deep learning technology, medical image registration methods based on deep learning technology have emerged continuously. This method obtains the deformation field from the moving image to the fixed image through a pre-trained network model, and performs registration processing on the moving image according to the deformation field to obtain the registered image. However, this will lead to an asymmetric registration process and even make it difficult to ensure the geometric consistency of the registration result, thereby resulting in low image registration accuracy. Summary of the Invention

[0005] Embodiments of this application provide a medical image registration method, apparatus, electronic device, and computer-readable storage medium, which can solve the problem of low medical image registration accuracy.

[0006] In a first aspect, embodiments of this application provide a medical image registration method, including:

[0007] Obtain a fixed image and a moving image;

[0008] Extract at least one scale of fixed feature maps from the fixed image and at least one scale of moving feature maps from the moving image through the feature extraction module of a pre-trained image registration model; the image registration model includes a feature extraction module, a forward deformation field prediction branch, a reverse deformation field prediction branch, and a registration module;

[0009] Process at least one scale of fixed feature maps through the reverse deformation field prediction branch to obtain at least one scale of reverse deformation fields;

[0010] Process at least one scale of reverse deformation fields and at least one scale of moving feature maps through the forward deformation field prediction branch to obtain a target forward deformation field;

[0011] The target forward deformation field and the moving image are registered through a registration module to obtain a registered image.

[0012] As can be seen from the above, in the embodiment of the present application, by setting a forward deformation field prediction branch and a reverse deformation field prediction branch in the image registration model, the parameters learned by the image registration model during the training stage can take into account both the forward flow field and the reverse flow field. Furthermore, in the inference stage, the image registration model can obtain a target forward deformation field with better reversibility according to the reverse deformation field and the moving feature map. That is, in the forward deformation field prediction branch, the target forward deformation field is obtained according to the reverse deformation field and the moving feature map. In this way, through the network structure with a forward deformation field prediction branch and a reverse deformation field prediction branch, and by adding the constraint of reverse information and the estimation information during the calculation of the forward deformation field, the reversibility of the target forward deformation field is enhanced, and a symmetric registration result can be obtained, with higher image registration accuracy.

[0013] That is to say, compared with the image registration model that only focuses on the forward deformation field from the moving image to the fixed image, the image registration model of the embodiment of the present application not only focuses on the forward deformation field from the moving image to the fixed image, but also focuses on the reverse deformation field from the fixed image to the moving image. Moreover, the constraint of reverse information and the estimation information are added to the forward deformation field prediction branch. Therefore, a forward deformation field with better reversibility can be obtained, and the better reversibility of the forward deformation field can make the image registration accuracy higher.

[0014] In some possible implementation manners of the first aspect, the target forward deformation field is obtained by processing the reverse deformation field at at least one scale and the moving feature map at at least one scale through the forward deformation field prediction branch, including:

[0015] For each scale, a forward estimated deformation field is obtained according to the first deformation field obtained by taking the negative of the reverse deformation field and the forward deformation field; a warping operation is performed on the forward estimated deformation field and the moving feature map to obtain an estimated information feature map; a second deformation field is obtained according to the estimated information feature map and the moving feature map;

[0016] The target forward deformation field is obtained based on the second deformation field;

[0017] Wherein, the forward deformation field is obtained according to the moving feature map or the second deformation field of the previous scale.

[0018] In some possible implementation manners of the first aspect, the feature extraction module is used to extract the moving feature maps at n scales and the fixed feature maps at n scales, where n is greater than or equal to 2;

[0019] The forward deformation field prediction branch includes n layers of first iterative correction modules and a first upsampling module disposed between two adjacent layers of first iterative correction modules; each layer of first iterative correction module includes a registration estimation sub-module, a first warping sub-module, and a first optimization fusion sub-module;

[0020] Processing the inverse deformation field at at least one scale and the moving feature map at at least one scale through the forward deformation field prediction branch to obtain the target forward deformation field, including:

[0021] In the m-th layer of first iterative correction module, the first warping sub-module performs a warping operation on the moving feature map at the k-th scale and the forward deformation field at the k-th scale to obtain a first warped feature map, where m and k are greater than or equal to 1 and less than or equal to n;

[0022] After the registration estimation sub-module performs a forward deformation field estimation operation on the forward deformation field at the k-th scale and the first deformation field at the k-th scale to obtain the forward estimated deformation field at the k-th scale, a warping operation is performed on the forward estimated deformation field at the k-th scale and the moving feature map at the k-th scale to obtain the estimated information feature map at the k-th scale, and the first deformation field at the k-th scale is obtained by taking the negative of the inverse deformation field at the k-th scale;

[0023] The first optimization fusion sub-module performs an optimization fusion process on the first warped feature map at the k-th scale, the estimated information feature map at the k-th scale, and the moving feature map at the k-th scale to obtain the second deformation field at the k-th scale;

[0024] When k is greater than or equal to 2, the first upsampling module between the m-th layer and the m + 1-th layer upsamples the second deformation field at the k-th scale to obtain the forward deformation field at the k - 1-th scale;

[0025] When k = 1, the second deformation field at the 1st scale is determined as the target forward deformation field;

[0026] When m = 1, k = n; when m = n, k = 1; when m = 1, the forward deformation field at the n-th scale is generated based on the moving feature map at the n-th scale. The forward deformation fields at other scales are obtained based on the second deformation field at the previous scale

[0027] In this implementation, for each scale, preliminary estimation and correction processing are performed, and the deformation field obtained in the previous layer is upsampled and used as the deformation field input to the next layer, and iterative correction is performed layer by layer. Compared with the method of directly allowing the network model to output a final deformation field, the embodiment of the present application can make the deformation field continuously iterate to approach the optimal solution, making the image registration process more accurate and more robust.

[0028] In addition, the information obtained by estimating the forward deformation field and warping based on the inverse deformation field and the moving feature map is also provided to the forward deformation field prediction branch to achieve the estimation of the forward and inverse registration information and information interaction, which can further improve the accuracy and robustness of the image registration process.

[0029] In some possible implementation manners of the first aspect, the inverse deformation field prediction branch includes n layers of second iterative correction modules and a second upsampling module disposed between two adjacent layers of second iterative correction modules;

[0030] Each layer of the second iterative correction module includes a second warping sub-module and a second optimization and fusion sub-module;

[0031] Processing the fixed feature map of at least one scale through the inverse deformation field prediction branch to obtain the inverse deformation field of at least one scale, including:

[0032] In the m-th layer of the second iterative correction module, the second warping sub-module performs a warping operation on the fixed feature map of the k-th scale and the inverse deformation field of the k-th scale to obtain the second warped feature map of the k-th scale;

[0033] The second optimization and fusion sub-module performs an optimization and fusion process on the second warped feature map of the k-th scale and the fixed feature map of the m-th scale to obtain the third deformation field of the k-th scale;

[0034] When k is greater than or equal to 2, the second upsampling module between the m-th layer and the m + 1-th layer upsamples the third deformation field of the k-th scale to obtain the inverse deformation field at the k - 1-th scale;

[0035] Wherein, when m = 1, the inverse deformation field at the n-th scale is generated according to the fixed feature map of the n-th scale.

[0036] In this implementation manner, for each scale, preliminary estimation and correction processing are performed, and the deformation field obtained in the previous layer is used as the deformation field input to the next layer after upsampling, and iterative correction is performed layer by layer. Compared with the method of directly allowing the network model to output a final deformation field, the embodiment of the present application can make the deformation field continuously iterate to approach the optimal solution, making the image registration process more accurate and more robust.

[0037] In some possible implementation manners of the first aspect, the first optimization and fusion sub-module and the second optimization and fusion sub-module include a channel aggregation module. The channel aggregation module can enable the network model to dynamically focus on more valuable channel features and improve the discriminative power of the model.

[0038] In some possible implementation manners of the first aspect, the feature extraction module includes n - 1 layers of downsampling residual convolution modules;

[0039] Extract at least one scale of fixed feature maps from a fixed image and at least one scale of moving feature maps from a moving image through the feature extraction module of the image registration model, including:

[0040] In the downsampling convolutional module of the j-th layer, after obtaining the moving feature map of the j-th scale, perform a downsampling operation and a residual convolution operation on the moving feature map of the j-th scale to obtain the moving feature map of the (j + 1)-th scale;

[0041] In the downsampling convolutional module of the j-th layer, after obtaining the fixed feature map of the j-th scale, perform a downsampling operation and a residual convolution operation on the fixed feature map of the j-th scale to obtain the fixed feature map of the (j + 1)-th scale; j is greater than or equal to 1 and less than or equal to n - 1;

[0042] When j = 1, the moving feature map of the j-th scale is extracted from the moving image, and the fixed feature map of the j-th scale is extracted from the fixed image.

[0043] In this implementation, it is possible to capture global deformations using a large receptive field at low resolution and retain local detail information at a higher resolution.

[0044] In some possible implementations of the first aspect, the downsampling residual convolution module includes a downsampling sub-module and a residual convolution sub-module; the residual convolution sub-module includes at least two parallel branch channels, and each branch channel uses a three-dimensional convolution kernel of a different size.

[0045] In this implementation, by using separable multi-branch channel convolution, features of different scales in the horizontal, vertical, and depth directions can be obtained, and finally, the outputs of each branch are concatenated in the channel dimension to obtain a feature map that fuses multi-directional receptive fields.

[0046] In some possible implementations of the first aspect, the training process of the image registration model includes:

[0047] Obtain training sample data, where the training sample data includes at least one image pair, and the image pair includes a fixed image sample and a moving image sample;

[0048] Input the fixed image sample and the moving image sample into the image registration model to obtain the final forward deformation field and the final inverse deformation field output by the image registration model;

[0049] According to the final forward deformation field, the fixed image sample, and the moving image sample, calculate the forward loss using the forward loss function;

[0050] According to the final inverse deformation field, the fixed image sample, and the moving image sample, calculate the inverse loss using the inverse loss function;

[0051] Obtain the total loss according to the forward loss and the backward loss;

[0052] Adjust the parameters of the image registration model according to the total loss;

[0053] Iterate multiple times until the total loss tends to be stable, and obtain the trained image registration model.

[0054] In this implementation manner, by setting a forward deformation field prediction branch and a backward deformation field prediction branch on the network model, and using a forward loss function and a backward loss function for model training in the network training stage, the network model can learn parameters that take into account both the forward flow field and the backward flow field during the training stage. In this way, in the inference stage, the forward deformation field obtained by the image registration model has the characteristic of taking into account the backward flow field, with an enhancement of reversibility.

[0055] In a second aspect, an embodiment of the present application provides a medical image registration device, including:

[0056] An image acquisition module, configured to acquire a fixed image and a moving image;

[0057] A feature extraction module, configured to extract fixed feature maps of at least one scale from the fixed image and moving feature maps of at least one scale from the moving image through the feature extraction module of the image registration model; the image registration model includes a feature extraction module, a forward deformation field prediction branch, a backward deformation field prediction branch, and a registration module;

[0058] A backward deformation field prediction module, configured to process the fixed feature maps of at least one scale through the backward deformation field prediction branch to obtain backward deformation fields of at least one scale;

[0059] A forward deformation field prediction module, configured to process the backward deformation fields of at least one scale and the moving feature maps of at least one scale through the forward deformation field prediction branch to obtain a target forward deformation field;

[0060] A registration processing module, configured to perform registration processing on the target forward deformation field and the moving image through the registration module to obtain a registered image.

[0061] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method according to any one of the first aspects above is implemented.

[0062] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the first aspects above is implemented.

[0063] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the method described in any one of the above first aspects.

[0064] It can be understood that for the beneficial effects of the above second aspect to fifth aspect, reference can be made to the relevant descriptions in the above first aspect, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic block diagram of a model structure of an image registration model provided by an embodiment of the present application;

[0066] Figure 2A It is a schematic diagram of the extraction process of n scale moving feature maps provided by an embodiment of the present application;

[0067] Figure 2B It is a schematic diagram of the extraction process of n scale fixed feature maps provided by an embodiment of the present application;

[0068] Figure 3 It is a schematic diagram of the structure of a large kernel depthwise separable convolution feature extraction module provided by an embodiment of the present application;

[0069] Figure 4 It is a schematic block diagram of the structures of a forward deformation field prediction branch and a reverse deformation field prediction branch provided by an embodiment of the present application;

[0070] Figure 5 It is a schematic diagram of the iterative correction process of the m-th layer provided by an embodiment of the present application;

[0071] Figure 6 It is a schematic diagram of the overall multi-scale layer-by-layer pyramid network provided by an embodiment of the present application;

[0072] Figure 7 It is a schematic block diagram of the structure of a CA module provided by an embodiment of the present application;

[0073] Figure 8 It is a schematic block diagram of the training process of an image registration model provided by an embodiment of the present application;

[0074] Figure 9 It is a schematic block diagram of the process of a medical image registration method provided by an embodiment of the present application;

[0075] Figure 10 It is a schematic block diagram of the structure of a medical image registration device provided by an embodiment of the present application;

[0076] Figure 11 It is a schematic block diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0077] In the following description, specific details such as specific system architectures, technologies, etc. are presented for purposes of illustration and not limitation, so as to provide a thorough understanding of the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details from obscuring the description of the present application.

[0078] It should be understood that when used in the specification and claims of the present application, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their combinations.

[0079] It should also be understood that the term "and / or" as used in the specification and claims of the present application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0080] As used in the specification and claims of the present application, the term "if" can be interpreted as "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if determined" or "if [the described condition or event] is detected" can be interpreted as meaning "once determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]" depending on the context.

[0081] In addition, in the description of the specification and claims of the present application, the terms "first", "second", "third", etc. are only used for differentiating descriptions and cannot be understood as indicating or implying relative importance.

[0082] The reference to "one embodiment" or "some embodiments" or the like described in the specification of the present application means that a specific feature, structure, or characteristic described in connection with that embodiment is included in one or more embodiments of the present application. Thus, statements such as "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "comprising", "including", "having", and their variants all mean "including but not limited to", unless otherwise specifically emphasized in another way.

[0083] In the field of medical imaging, medical image registration can be applied to multiple scenarios ranging from clinical diagnosis to medical research. For example, through medical image registration, images from different imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET) are precisely aligned to more comprehensively display the anatomical information, functional information, and metabolic activities of tissues and organs, providing support for disease diagnosis and treatment decisions. For example, in tumor resection surgery, through medical image registration, medical images of different modalities (such as DTI and MRI) are precisely registered to distinguish the positional relationship between nerve bundles and tumor tissues. Another example is in the postoperative stage, through medical image registration to compare preoperative and postoperative images to evaluate the surgical efficacy, timely monitor complications, and formulate subsequent treatment plans, etc.

[0084] In the related art, medical image registration methods only focus on the forward deformation field from the moving image to the fixed image, lacking modeling of the reverse deformation field from the fixed image to the moving image. This can easily lead to asymmetry in the registration process and even make it difficult to ensure the geometric consistency of the registration result, thereby resulting in low medical image registration accuracy. Among them, the moving image can also be called the floating image, and the fixed image can also be called the reference image.

[0085] Ideally, if the forward deformation field and the reverse deformation field can be mutually constrained or reversible, the accuracy and robustness of image registration will be significantly improved. However, in actual engineering, how to make the network model learn both the forward deformation field and the reverse deformation field and maintain symmetry is one of the key problems that have not been fully solved yet.

[0086] To address the related technical problems mentioned above, in the embodiments of this application, by explicitly setting a forward deformation field prediction branch and a reverse deformation field prediction branch in the image registration model, the parameters learned by the image registration model during the training stage can take into account both the forward flow field and the reverse flow field. As a result, the image registration model can obtain a target forward deformation field with better reversibility during the inference stage. In addition, constraints on reverse information and estimation information are added during the calculation of the forward deformation field in the forward deformation field prediction branch. This can strengthen the reversibility of the target forward deformation field, thereby obtaining a registration result with better symmetry and higher image registration accuracy. That is, the embodiments of this application not only focus on the forward deformation field from the moving image to the fixed image but also on the reverse deformation field from the fixed image to the moving image.

[0087] See Figure 1 , which is a schematic block diagram of a model structure of the image registration model provided by the embodiments of this application. The image registration model may include a feature extraction module 100, a forward deformation field prediction branch 200, a reverse deformation field prediction branch 300, and a registration module 400.

[0088] The feature extraction module 100 is used to extract a moving feature map from a moving image and a fixed feature map from a fixed image.

[0089] The inverse deformation field prediction branch 300 is used to obtain an inverse deformation field according to the fixed feature map.

[0090] The forward deformation field prediction branch 200 is used to obtain a forward deformation field according to the inverse deformation field and the moving feature map.

[0091] In the model training stage, the registration module 400 is used to perform forward registration processing on the moving image according to the forward deformation field to obtain a registered image, so as to calculate a forward loss value according to the registered image and the fixed image; it is also used to perform inverse registration processing on the fixed image according to the inverse deformation field to obtain a registered image, so as to calculate an inverse loss value according to the registered image and the moving image.

[0092] In the model inference stage, the registration module 400 is used to perform forward registration processing on the moving image according to the forward deformation field to obtain a registered image. Of course, when choosing to output the inverse registered image, the registration module 400 can also perform inverse registration processing on the fixed image according to the inverse deformation field to obtain a registered image.

[0093] As Figure 1 shown, after the image registration model is trained, the moving image and the fixed image to be registered are input into the image registration model, and the image registration model performs inference according to the parameters learned in the training stage and outputs a registered image.

[0094] In some embodiments, the feature extraction module 100 can extract moving feature maps of n scales and fixed feature maps of n scales. n is greater than or equal to 2. For example, n is equal to 4. At this time, moving feature maps and fixed feature maps of different scales are extracted at the 1 / 1, 1 / 2, 1 / 4, and 1 / 8 resolution levels.

[0095] The feature extraction module 100 may include n - 1 layers of downsampling residual convolution modules. Each layer of the downsampling residual convolution module is used to perform a downsampling operation and a residual convolution operation on the input feature map, and output a feature map of the next scale. The n - 1 layers of downsampling residual convolution modules are cascaded in sequence. The input of the first layer of the downsampling residual convolution module is the moving feature map or the fixed feature map of the first scale, and the output is the moving feature map or the fixed feature map of the second scale; the output of the first layer of the downsampling residual convolution module serves as the input of the second layer of the downsampling residual convolution module. The output of the second layer of the downsampling residual convolution module is the moving feature map or the fixed feature map of the third scale, and the output of the second layer of the downsampling residual convolution module serves as the input of the third layer of the downsampling residual convolution module. And so on, the output of the (n - 1)th layer of the downsampling residual convolution module is the moving feature map or the fixed feature map of the nth scale. The moving feature map of the first scale is extracted from the moving image of the original resolution, and the fixed feature map of the first scale is extracted from the fixed image of the original resolution.

[0096] The feature extraction module 100 can perform feature extraction on the moving image and the fixed image respectively. Exemplarily, the following will separately combine Figure 2A and Figure 2B , and introduce the extraction process of the moving feature map and the fixed feature map.

[0097] Refer to Figure 2A the schematic diagram of the extraction process of the n-scale moving feature maps provided by the embodiment of the present application shown in, the feature extraction module 100 may include a convolution extraction module and n - 1 layers of downsampling residual convolution modules. The convolution extraction module is used to extract the moving feature map of the first scale from the moving image of the original resolution. The n - 1 layers of downsampling residual convolution modules are connected in a cascaded manner.

[0098] In the jth layer of the downsampling convolution module, after obtaining the moving feature map of the jth scale, perform a downsampling operation and a residual convolution operation on the moving feature map of the jth scale to obtain the moving feature map of the (j + 1)th scale; j is greater than or equal to 1 and less than or equal to n - 1; when j = 1, the moving feature map of the first scale is extracted from the moving image of the original resolution.

[0099] For example, when n is equal to 4, the first scale is 1 / 1 resolution, the second scale is 1 / 2 resolution, the third scale is 1 / 4 resolution, and the fourth scale is 1 / 8 resolution. At this time, j is equal to 3, that is, there are 3 cascaded downsampling residual convolution modules. The moving image with the original resolution is input into the convolution extraction module. After the convolution extraction module extracts features from the moving image with the original resolution, it outputs a moving feature map at the 1 / 1 resolution level (i.e., the moving feature map of the first scale). The moving feature map of the first scale is input into the first-layer downsampling residual convolution module. After the first-layer downsampling residual convolution module performs downsampling operation and residual convolution operation on the moving feature map of the first scale, it outputs a moving feature map at the 1 / 2 resolution level (i.e., the moving feature map of the second scale).

[0100] Similarly, the moving feature map of the second scale is input into the second-layer downsampling residual convolution module, and the second-layer downsampling residual convolution module outputs a moving feature map at the 1 / 4 resolution level (i.e., the moving feature map of the third scale); the moving feature map of the third scale is input into the third-layer downsampling residual convolution module, and the third-layer downsampling residual convolution module outputs a moving feature map at the 1 / 8 resolution level (i.e., the moving feature map of the fourth scale). In this way, the feature extraction module 100 can extract moving feature maps of 4 different scales according to the input moving image with the original resolution.

[0101] See Figure 2B the schematic diagram of the extraction process of n-scale fixed feature maps provided by the embodiment of the present application shown. The convolution extraction module is used to extract the fixed feature map of the first scale from the fixed image with the original resolution. The n - 1 layer of downsampling residual convolution modules are connected in a cascaded manner. In the j-th layer downsampling convolution module, after obtaining the fixed feature map of the j-th scale, a downsampling operation and a residual convolution operation are performed on the fixed feature map of the j-th scale to obtain the fixed feature map of the j + 1-th scale; j is greater than or equal to 1 and less than or equal to n - 1; when j = 1, the fixed feature map of the first scale is extracted from the fixed image with the original resolution.

[0102] For example, when n is equal to 4, the first scale is 1 / 1 resolution, the second scale is 1 / 2 resolution, the third scale is 1 / 4 resolution, and the fourth scale is 1 / 8 resolution. At this time, j is equal to 3, that is, there are 3 cascaded downsampling residual convolution modules in sequence. The fixed image with the original resolution is input into the convolution extraction module. After the convolution extraction module extracts features from the fixed image with the original resolution, it outputs a fixed feature map at the 1 / 1 resolution level (i.e., the fixed feature map of the first scale). The fixed feature map of the first scale is input into the first-layer downsampling residual convolution module. After the first-layer downsampling residual convolution module performs downsampling operations and residual convolution operations on the fixed feature map of the first scale, it outputs a fixed feature map at the 1 / 2 resolution level (i.e., the fixed feature map of the second scale). Similarly, through the second-layer downsampling residual convolution module and the third-layer downsampling residual convolution module, a moving feature map at the 1 / 4 resolution level (i.e., the moving feature map of the third scale) and a moving feature map at the 1 / 8 resolution level (i.e., the moving feature map of the fourth scale) are obtained.

[0103] It should be noted that in the image registration methods of related technologies, registration is usually performed only at a single scale or a single resolution, that is, only a feature map at one resolution level is obtained. However, if only the feature map at the high-resolution level is used for registration, although more local details can be captured, it is difficult to effectively obtain global deformation information due to huge computational overhead and memory overhead. If only the feature map at the low-resolution level is used for registration, although global deformation information can be obtained, local fine feature information (such as lesion edges or fine anatomical structures) will be lost, and local fine registration cannot be achieved.

[0104] In the embodiments of the present application, the feature extraction module 100 extracts feature maps at at least two resolution levels, that is, extracts the features of the moving image and the fixed image at multiple resolution levels (such as 1 / 2, 1 / 4, 1 / 8 and other resolution levels), and obtains feature maps of multiple scales. In this way, both the large receptive field of the low resolution can be used to capture global deformation information, and local detail information can be retained at a relatively high resolution, taking into account both global large deformations and local fine registration.

[0105] In some embodiments, each layer of the downsampling residual convolution module may include a downsampling sub-module and a residual convolution sub-module. The residual convolution sub-module may include at least two parallel branch channels, and each branch channel uses three-dimensional convolution kernels of different sizes. For example, the three-dimensional convolution kernels on the three parallel branch channels are 3×3×3, 1×11×1, and 11×1×1 respectively.

[0106] It should be noted that in the image registration methods of related technologies, a single type of convolution kernel (such as a 3×3×3 convolution kernel) is usually used for feature extraction. However, the inventors found during the research process that organ deformation does not necessarily have the same feature distribution in all dimensions or all directions of the three-dimensional space. Therefore, it is difficult for a single type of convolution kernel to capture multi-directional and multi-scale information. Based on this, the feature extraction module of large kernel depthwise separable convolution is adopted in the embodiments of the present application for feature extraction, so as to utilize three-dimensional convolution kernels of different sizes in at least two parallel branch channels to capture features of different scales in the horizontal, vertical, and depth directions. In addition, there are larger convolution kernels (such as 1×11×1 and 11×1×1, etc.) on the branch channels. The large kernel depth convolution can expand the receptive field, so as to better capture the global feature information and long-range dependence information in the image. That is, through at least two parallel convolution branch channels, and the convolution kernels on multiple branch channels have different sizes, there are both large convolution kernels and smaller convolution kernels, so that the model can take into account the large kernel to capture global mutual information and the small kernel to focus on local texture during the feature extraction stage, and realize more robust cross-modal registration.

[0107] Among them, the receptive field is the range of the input area that a convolution kernel in a convolutional neural network can cover.

[0108] Exemplarily, referring to Figure 3 the structural schematic diagram of the large kernel depthwise separable convolution feature extraction module provided by the embodiments of the present application shown in, the input is a fixed image or a moving image, and the output is feature maps of different scales (i.e., Figure 3 the Output in). When N = 1, the moving feature map or the fixed feature map of the first scale is output. When N = 2, the moving feature map or the fixed feature map of the second scale is output, and so on.

[0109] The large kernel depthwise separable convolution feature extraction module includes an X N module ( Figure 3 the dashed box in), a three-dimensional convolution layer (Cov3d), instance normalization (InstanceNorm3D), and a non-linear activation function (LeakyReLu). The X N module may include an average pooling layer (AvgPool3d), multiple three-dimensional convolution layers (Cov3d), multiple instance normalization layers (InstanceNorm3D), a large kernel depthwise separable convolution layer, and a non-linear activation function (LeakyReLu). For the specific connection relationship of each layer, please refer to Figure 3 .

[0110] The large-kernel depthwise separable convolution layer may include a split module, multiple parallel branch channels, and a concatenate module, and the multiple parallel branch channels adopt three-dimensional convolution kernels of different sizes. For example, Figure 3 shows the depthwise separable convolution 11×1×1 (DWconv11×1×1), the depthwise separable convolution 1×11×1 (DWconv1×11×1), the depthwise separable convolution 1×1×11 (DWconv1×1×11), and the depthwise separable convolution 3×3×3 (DWconv3×3×3).

[0111] After passing through the split module, it can be separated into multiple parallel branch channels; after the depthwise separable convolution processing of the multiple parallel branch channels, the outputs of different branch channels are input into the concatenate module to concatenate the features of different branch channels.

[0112] X N The average pooling layer in the module can perform average pooling operations on the input feature map to reduce the spatial resolution, that is, perform downsampling operations.

[0113] For example, Figure 3 the processing process of the feature extraction module can be as follows:

[0114] Obtain the moving image or fixed image with the original resolution. Use 3D convolution (Cov3d) to extract features from the input data at the original resolution, and then combine instance normalization and non-linear activation functions to enhance the initial feature expression ability. After passing through LeakyReLu, the feature map at the 1 / 1 resolution level (moving feature map or fixed feature map) can be output.

[0115] The feature map at the 1 / 1 resolution level is input into the X N module (at this time N = 2). After passing through the average pooling layer (AvgPool3d), the feature map at the 1 / 2 resolution level is output; the feature map at the 1 / 2 resolution level is then subjected to 3D convolution (Cov3d) and instance normalization (InstanceNorm3d) to extract local features; then, the large-kernel depthwise separable convolution layer is used to enhance the computational efficiency and receptive field range to obtain the convolution output features output by the large-kernel depthwise separable convolution layer; finally, the convolution output features and the initial features (i.e., the features output by cov3d in the xn module) are fused using the residual (i.e., the "+" sign in the xn module) to avoid gradient disappearance and improve the feature modeling ability. The fused features output through the residual connection are successively subjected to operations such as 3D convolution, instance normalization, and non-linear activation functions, and finally the feature map at the 1 / 2 resolution level is output through the output module.

[0116] The feature map at the 1 / 2 resolution level output by the Output module is then input into the XN module (where N = 3 at this time). After the downsampling operation and residual convolution operation of the feature map at the 1 / 2 resolution level pass through the X N module, the output module outputs the final feature map at the 1 / 4 resolution level.

[0117] Similarly, the feature map at the 1 / 4 resolution level output by the output module is input into the X N module (where N = 4 at this time). After the downsampling operation and residual convolution operation of the feature map at the 1 / 4 resolution level pass through the xn module, the output module outputs the final feature map at the 1 / 8 resolution level. And so on, until feature maps at all scales (such as 1 / 1, 1 / 2, 1 / 4, 1 / 8 resolutions) are obtained, providing rich multi-scale information for subsequent image registration.

[0118] It can be seen that multiple xn modules adopt a cascading operation. At each level, the resolution of the input feature map is halved, and the number of channels is doubled, thereby gradually extracting deeper features. Figure 2A and Figure 2B the downsampling residual convolution module in Figure 3 can be the xn module in Figure 3 and the convolution extraction module can include

[0119] cov3d, InstanceNorm3d, LeakyReLu, etc. outside the xn module in

[0120] In some embodiments, when the feature extraction module can extract moving and fixed feature maps at n scales, the forward deformation field prediction branch 200 and the backward deformation field prediction branch 300 can iteratively process the feature maps at different scales layer by layer to approximate the optimal solution by continuously iterating the deformation field. Figure 4 Referring to the structural schematic block diagrams of the forward deformation field prediction branch and the backward deformation field prediction branch shown in

[0121] the forward deformation field prediction branch 200 is used to output the final forward deformation field according to the moving feature map and the backward deformation field; the backward deformation field prediction branch 300 is used to output the final backward deformation field according to the fixed feature map, and is also used to provide the required backward deformation field to the forward deformation field prediction branch 200.

[0122] The forward deformation field prediction branch 200 includes n layers of first iterative correction modules, and a first upsampling module disposed between two layers of first iterative correction modules.

[0123] In the second iterative correction module of the m-th layer, a warping operation and an optimization fusion operation are performed based on the fixed feature map of the k-th scale and the inverse deformation field of the k-th scale to obtain the third deformation field of the k-th scale; when it is greater than or equal to 2, after the third deformation field of the k-th scale passes through the second upsampling module, the inverse deformation field of the k-1-th scale can be obtained. The inverse deformation field of the k-1-th scale is input as the inverse deformation field of the second iterative correction module of the next layer. When k = 1, the third deformation field of the k-th scale is determined as the final inverse deformation field. In this way, the inverse deformation field is continuously iteratively corrected to obtain the final inverse deformation field (i.e., the inverse deformation field of the 1st scale).

[0124] Among them, in the second iterative correction module of the first layer, the inverse deformation field of the n-th scale is generated according to the fixed feature map of the n-th scale. In the second iterative correction modules of the second layer and subsequent layers, the input inverse deformation fields are all obtained by upsampling the third deformation field output by the previous layer.

[0125] In addition, the inverse deformation field of the k-th scale can be provided to the first iterative correction module of the m-th layer, so that the first iterative correction module outputs the second deformation field of the k-th scale according to the inverse deformation field of the k-th scale, the forward deformation field of the k-th scale, and the moving feature map of the k-th scale. When k is greater than or equal to 2, after the second deformation field of the k-th scale passes through the first upsampling module, the forward deformation field of the k-1-th scale is output. The forward deformation field of the k-1-th scale is input as the forward deformation field of the first iterative correction module of the next layer. When k = 1, the second deformation field of the k-th scale is determined as the final forward deformation field. In this way, the forward deformation field is continuously iteratively corrected to obtain the final forward deformation field.

[0126] Among them, in the first iterative correction module of the first layer, the forward deformation field of the n-th scale is generated according to the moving feature map of the n-th scale. In the first iterative correction modules of the second layer and subsequent layers, the input forward deformation fields are all obtained by upsampling the second deformation field output by the previous layer.

[0127] Among them, m and k are greater than or equal to 1 and less than or equal to n. When m = 1, k = n; when m = n, k = 1.

[0128] For example, n = 4, and the 1st scale is 1 / 1 resolution, the 2nd scale is 1 / 2 resolution, the 3rd scale is 1 / 4 resolution, and the 4th scale is 1 / 8 resolution.

[0129] When m = 1, k = 4. In the second iterative correction module of the first layer, a warping operation and an optimization fusion operation are performed based on the fixed feature map at the 4th scale (i.e., 1 / 8 resolution) and the inverse deformation field at the 4th scale to obtain the third deformation field at the 4th scale. Since k is greater than or equal to 2 at this time, after the third deformation field at the 4th scale passes through the second upsampling module, the inverse deformation field at the 3rd scale (i.e., 1 / 4 resolution) can be obtained. The inverse deformation field at the 3rd scale is input as the inverse deformation field of the second iterative correction module of the next layer (i.e., the second layer).

[0130] In the first iterative correction module of the first layer, the second deformation field at the 4th scale is output based on the inverse deformation field at the 4th scale, the forward deformation field at the 4th scale, and the moving feature map at the 4th scale. Since k is greater than or equal to 2 at this time, after the second deformation field at the 4th scale passes through the first upsampling module, the forward deformation field at the 3rd scale is output. The forward deformation field at the 3rd scale is input as the forward deformation field of the first iterative correction module of the next layer.

[0131] When m = 2, k = 3. In the second iterative correction module of the second layer, a warping operation and an optimization fusion operation are performed based on the fixed feature map at the 3rd scale and the inverse deformation field at the 3rd scale to obtain the third deformation field at the 3rd scale. Since k is greater than or equal to 2 at this time, after the third deformation field at the 3rd scale passes through the second upsampling module, the inverse deformation field at the 2nd scale can be obtained. The inverse deformation field at the 2nd scale is input as the inverse deformation field of the second iterative correction module of the next layer (i.e., the third layer).

[0132] In the first iterative correction module of the second layer, the second deformation field at the 3rd scale is output based on the inverse deformation field at the 3rd scale, the forward deformation field at the 3rd scale, and the moving feature map at the 3rd scale. Since k is greater than or equal to 2 at this time, after the second deformation field at the 3rd scale passes through the first upsampling module, the forward deformation field at the 2nd scale is output. The forward deformation field at the 2nd scale is input as the forward deformation field of the first iterative correction module of the next layer.

[0133] When m = 3, k = 2. In the second iterative correction module of the third layer, a warping operation and an optimization fusion operation are performed based on the fixed feature map at the 2nd scale and the inverse deformation field at the 2nd scale to obtain the third deformation field at the 2nd scale. Since k is greater than or equal to 2 at this time, after the third deformation field at the 2nd scale passes through the second upsampling module, the inverse deformation field at the 1st scale can be obtained. The inverse deformation field at the 1st scale is input as the inverse deformation field of the second iterative correction module of the next layer (i.e., the fourth layer).

[0134] In the first iterative correction module of the third layer, according to the inverse deformation field of the second scale, the forward deformation field of the second scale, and the moving feature map of the second scale, the second deformation field of the second scale is output. Since k is greater than or equal to 2 at this time, after passing through the first upsampling module, the second deformation field of the second scale outputs the forward deformation field of the first scale. The forward deformation field of the first scale is input as the forward deformation field of the first iterative correction module of the next layer.

[0135] When m = 4, k = 1. In the second iterative correction module of the fourth layer, a warping operation and an optimization fusion operation are performed according to the fixed feature map of the first scale and the inverse deformation field of the first scale to obtain the third deformation field of the first scale. Since k is equal to 1 at this time, the third deformation field of the first scale is determined as the final inverse deformation field.

[0136] In the first iterative correction module of the fourth layer, according to the inverse deformation field of the first scale, the forward deformation field of the first scale, and the moving feature map of the first scale, the second deformation field of the first scale is output. Since k is equal to 1 at this time, the second deformation field of the first scale is determined as the final forward deformation field.

[0137] To better introduce the processing flows of the first iterative correction module and the second iterative correction module of each layer, the processing modules of the first iterative correction module and the second iterative correction module of the m-th layer will be introduced and explained below in combination with Figure 5 the schematic diagram of the iterative correction process of the m-th layer shown in

[0138] As Figure 5 shown, the first iterative correction module of the m-th layer includes a registration estimation sub-module, a first warping sub-module, and a first optimization fusion sub-module. A first upsampling module is provided between the first iterative correction module of the m-th layer and the first iterative correction module of the (m + 1)-th layer.

[0139] The second iterative correction module of the m-th layer includes a second warping sub-module and a second optimization fusion sub-module. A second upsampling module is provided between the second iterative correction module of the m-th layer and the second iterative correction module of the (m + 1)-th layer.

[0140] As Figure 5 shown, in the first iterative correction module of the m-th layer, a warping operation (Warping) is performed on the moving feature map of the k-th scale and the forward deformation field of the k-th scale through the first warping sub-module to obtain the first warped feature map. When m = 1, k = n, and the forward deformation field of the n-th scale is generated according to the moving feature map of the n-th scale.

[0141] The forward deformation field estimation operation (Estimating Forward field) is performed on the forward deformation field of the k-th scale and the first deformation field of the k-th scale by the registration estimation sub-module to obtain the forward estimated deformation field of the k-th scale; then, the warping operation (Warping) is performed on the forward estimated deformation field of the k-th scale and the moving feature map of the k-th scale to obtain the estimated information feature map. The first deformation field of the k-th scale is obtained by taking the negative operation on the reverse deformation field of the k-th scale. The reverse deformation field of the k-th scale is provided by the reverse deformation field prediction branch 300.

[0142] The first optimization fusion sub-module performs optimization fusion on the first warped feature map of the k-th scale, the estimated information feature map of the k-th scale, and the moving feature map of the k-th scale to obtain the second deformation field of the k-th scale.

[0143] After the first iterative correction module of the m-th layer outputs the second deformation field of the k-th scale, when k is greater than or equal to 2, the second deformation field of the k-th scale is input to the first upsampling module between the m-th layer and the m + 1-th layer. The first upsampling module upsamples the second deformation field of the k-th scale to obtain the forward deformation field of the k - 1-th scale. The forward deformation field of the k - 1-th scale is input to the first iterative correction module of the m + 1-th layer.

[0144] When k is equal to 1, the second deformation field of the k-th scale is determined as the final forward deformation field.

[0145] As Figure 5 shown, in the second iterative correction module of the m-th layer, the warping operation is performed on the fixed feature map of the k-th scale and the reverse deformation field of the k-th scale by the second warping sub-module to obtain the second warped feature map of the k-th scale;

[0146] The second optimization fusion sub-module performs optimization fusion processing on the second warped feature map of the k-th scale and the fixed feature map of the k-th scale to obtain the third deformation field of the k-th scale;

[0147] When k is greater than or equal to 2, the third deformation field of the k-th scale is upsampled by the second upsampling module between the m-th layer and the m + 1-th layer to obtain the reverse deformation field at the k - 1-th scale. The reverse deformation field at the k - 1-th scale is input to the second iterative correction module of the m + 1-th layer.

[0148] When k is equal to 1, the third deformation field of the k-th scale is determined as the final forward deformation field.

[0149] When m = 1 and k = n, the reverse deformation field of the n-th scale is generated according to the fixed feature map of the n-th scale.

[0150] It should be noted that each layer of the first iterative correction module includes a registration estimation submodule, a first distortion submodule and a first optimization fusion submodule; each layer of the second iterative correction module includes a second distortion submodule and a second optimization fusion submodule. The processing of the first iterative correction module and the second iterative correction module of each layer can be as follows: Figure 5 shown.

[0151] It should be noted that in the image registration methods of related technologies, a single-step prediction method is usually adopted, that is, only one deformation field is obtained. However, in complex medical image registration scenarios, if the initially estimated deformation field is overfitted or distorted, the conventional single-step prediction method may stagnate at the local optimal solution and it is difficult to correct it spontaneously in the subsequent process. The lack of a mechanism for multi-step feedback and iterative fusion will lead to low image registration accuracy and robustness. How to integrate multiple interactive corrections in the network structure so that the deformation field can continuously iterate and approach a better solution is a technical problem that needs to be solved urgently.

[0152] In the embodiment of the present application, each scale is equipped with a preliminary estimation module and a refined correction module, that is, after the deformation field is obtained by preliminary estimation at each scale, a second correction is made in combination with the distorted features; in addition, the deformation field obtained in the previous layer is upsampled as the deformation field input to the next layer, and then it can be iteratively corrected layer by layer. In this way, by allowing the deformation field to continuously iterate to approach the optimal solution, the image registration process is more accurate and more robust.

[0153] In addition, in the forward deformation field prediction branch, deformation field operations and warping operations are performed according to the inverse deformation field provided by the inverse deformation field prediction branch to achieve forward and inverse registration information estimation and information interaction, which can also further improve the accuracy and robustness of the image registration process.

[0154] In order to better introduce the network structure of the image registration model provided in the embodiment of the present application, Figure 6 The overall schematic diagram of the multi-scale layer-by-layer pyramid network is shown for introduction and explanation.

[0155] exist Figure 6 middle, I m is a moving image, I f is a fixed image. M1~M4 are moving feature maps of 4 scales, and F1~F4 are fixed feature maps of 4 scales. Among them, M1 and F1 are feature maps of the first scale, which are feature maps extracted at the 1 / 1 resolution level; M2 and F2 are feature maps of the second scale, which are feature maps extracted at the 1 / 2 resolution level; M3 and F3 are feature maps of the third scale, which are feature maps extracted at the 1 / 4 resolution level; M4 and F4 are feature maps of the fourth scale, which are feature maps extracted at the 1 / 8 resolution level.

[0156] Among them, Rφ4~Rφ1 are reverse deformation fields at different scales. Fφ1~Fφ4 are forward deformation fields at different scales. Fφ is the final forward deformation field, and Rφ is the final reverse deformation field. Eφ1~Eφ4 are forward estimated deformation fields at different scales, and -Rφ4~-Rφ1 are deformation fields obtained by taking the negative operation on the reverse deformation fields Rφ4~Rφ1 (i.e., the first deformation field above).

[0157] Figure 6 In, "W" is the warping operation (i.e., warping), "E" is the forward deformation field estimation operation (i.e., Estimating Forward flow field), "F" is the fused deformation field operation (i.e., Fused flow field), and "up" is the upsampling operation. Among them, the reverse deformation field prediction branch can refer to the bottom row (i.e., the row where I f is located); the forward deformation field prediction branch can include the top row (i.e., the row where I m is located) and the middle row (i.e., the row where Eφ1 is located).

[0158] In Figure 6 , starting from the coarsest resolution (i.e., 1 / 8 resolution), the deformation field is generated layer by layer, and the deformation field prediction result of each layer provides a preliminary reference for the next layer. The specific process can be as follows:

[0159] First, generate M1~M4 according to the moving image I m , and generate F1~F4 according to the fixed image I f . For example, through the large kernel depthwise separable convolution feature extraction module shown in Figure 3 , 4 different-scale moving feature maps and fixed feature maps can be extracted.

[0160] For M4 and F4 (i.e., the first layer), generate the 4th-scale forward deformation field Fφ4 according to the 4th-scale moving feature map M4; generate the 4th-scale reverse deformation field Rφ4 according to the 4th-scale fixed feature map. Then take the negative of the 4th-scale reverse deformation field Rφ4 to obtain -Rφ4.

[0161] In the forward deformation field prediction branch, perform the warping operation according to Fφ4 and M4 to obtain the first warped feature map at the 4th scale (not shown in the figure); perform the forward deformation field estimation operation (i.e., "E") according to -Rφ4 and Fφ4 to obtain the forward estimated deformation field Eφ4, and then perform the warping operation according to the forward estimated deformation field Eφ4 and M4 to obtain the estimated information feature map (not shown in the figure); perform splicing and fusion according to the first warped feature map, the estimated information feature map, and M4 to obtain the first fused feature map at the 4th scale (i.e., the cuboid including three colors in the figure).

[0162] After performing an optimization process on the first fused feature map of the fourth scale and then through a fusion operation (i.e., "F"), the second deformation field of the fourth scale can be obtained. Among them, the processes of optimization and fusion can be: obtaining a deformation field based on the first fused feature map; obtaining a registration update value q based on this deformation field; performing a fusion operation (i.e., "F") based on this registration update value q and this deformation field to obtain the second deformation field, and this deformation field is a deformation field after update and adjustment.

[0163] After obtaining the second deformation fields of the 4 scales, through an upsampling operation (i.e., "up"), the forward deformation field Fφ3 of the third scale can be obtained.

[0164] It should be noted that the first optimized fusion sub-module of each layer above can include Figure 6 the process of obtaining the first fused feature map by splicing and fusing, the optimization process, and the fusion process (i.e., "F").

[0165] In the reverse deformation field prediction branch, a warping operation is performed based on Rφ4 and F4 to obtain the second warped feature map of the fourth scale (not shown in the figure); a splicing and fusion is performed based on the second warped feature map and F4 to obtain the second fused feature map of the fourth scale (i.e., the cuboid including two colors in the figure).

[0166] After performing an optimization process on the second fused feature map of the fourth scale and then through a fusion operation (i.e., "F"), the third deformation field of the fourth scale can be obtained. Among them, the processes of optimization and fusion can be: obtaining a deformation field based on the second fused feature map; obtaining a registration update value q_reverse based on this deformation field; performing a fusion operation (i.e., "F") based on this registration update value q_reverse and this deformation field to obtain the third deformation field, and this deformation field is a deformation field after update and adjustment.

[0167] It should be noted that the second optimized fusion sub-module of each layer above can include Figure 6 the process of obtaining the second fused feature map by splicing and fusing, the optimization process, and the fusion process (i.e., "F").

[0168] After obtaining the third deformation fields of the 4 scales, through an upsampling operation (i.e., "up"), the reverse deformation field Rφ3 of the third scale can be obtained.

[0169] For M3 and F3 (i.e., the second layer), taking the negative of the reverse deformation field Rφ3 of the third scale to obtain -Rφ3.

[0170] At this time, in the forward deformation field prediction branch, a warping operation is performed according to Fφ3 and M3 to obtain the first warped feature map at the third scale (not shown in the figure); after performing a forward deformation field estimation operation (i.e., "E") according to -Rφ3 and Fφ3 to obtain the forward estimated deformation field Eφ3, a warping operation is then performed according to the forward estimated deformation field Eφ3 and M3 to obtain the estimated information feature map (not shown in the figure); the first fused feature map at the third scale (i.e., the cuboid including three colors in the figure) is obtained by splicing and fusing the first warped feature map, the estimated information feature map and M3. After performing an optimization process on the first fused feature map at the third scale, and then through a fusion operation (i.e., "F"), the second deformation field at the third scale can be obtained. After obtaining the second deformation fields at 4 scales, through an upsampling operation (i.e., "up"), the forward deformation field Fφ2 at the second scale can be obtained.

[0171] In the reverse deformation field prediction branch, a warping operation is performed according to Rφ3 and F3 to obtain the second warped feature map at the third scale (not shown in the figure); the second fused feature map at the third scale (i.e., the cuboid including two colors in the figure) is obtained by splicing and fusing the second warped feature map and F3. After performing an optimization process on the second fused feature map at the third scale, and then through a fusion operation (i.e., "F"), the third deformation field at the third scale can be obtained. After obtaining the third deformation fields at 3 scales, through an upsampling operation (i.e., "up"), the reverse deformation field Rφ2 at the second scale can be obtained.

[0172] For M2 and F2 (i.e., the third layer), the processing process is similar to that of the second layer. For specific details, please refer to the relevant content of the second layer and will not be elaborated here.

[0173] For M1 and F1 (i.e., the fourth layer), the negative value of the reverse deformation field Rφ1 at the first scale is taken to obtain -Rφ1.

[0174] At this time, in the forward deformation field prediction branch, a warping operation is performed according to Fφ1 and M1 to obtain the first warped feature map of the first scale (not shown in the figure); after performing a forward deformation field estimation operation (i.e., "E") according to -Rφ1 and Fφ1 to obtain the forward estimated deformation field Eφ1, a warping operation is then performed according to the forward estimated deformation field Eφ1 and F1 to obtain the estimated information feature map (not shown in the figure); the first fused feature map of the first scale (i.e., the cuboid including three colors in the figure) is obtained by splicing and fusing the first warped feature map, the estimated information feature map and M1. After performing an optimization process on the first fused feature map of the first scale and then through a fusion operation (i.e., "F"), the second deformation field of the first scale can be obtained. At this time, since it is at the last layer, an upsampling operation is not required anymore, and the second deformation field of the first scale is determined as the final forward deformation field Fφ.

[0175] In the reverse deformation field prediction branch, a warping operation is performed according to Rφ1 and F1 to obtain the second warped feature map of the first scale (not shown in the figure); the second fused feature map of the first scale (i.e., the cuboid including two colors in the figure) is obtained by splicing and fusing the second warped feature map and F1. After performing an optimization process on the second fused feature map of the first scale and then through a fusion operation (i.e., "F"), the third deformation field of the first scale can be obtained. The third deformation field of the first scale is determined as the final reverse deformation field Rφ.

[0176] It should be noted that three-dimensional medical images usually contain a large number of voxels, and there are significant size and deformation differences between different anatomical structures. The overall shape of organs (such as the liver, heart) corresponds to a larger scale; tissue details (such as blood vessels, tumor edges) are at a smaller scale. Thus, it is required that the image registration algorithm can capture both global deformations (such as large-scale displacements of organs) and local differences (such as precise alignment of lesion areas). In the related art, feature extraction or registration is only performed at a single resolution. On the one hand, the computational cost is huge at high resolution and it is easy to get stuck in local details and difficult to capture global deformations. On the other hand, key information at small lesions or edges will be lost at too low resolution. However, the image registration model with a multi-scale layer-by-layer pyramid network structure provided by the embodiments of the present application can generate a deformation field layer by layer starting from the coarsest resolution, and the deformation field prediction result of each layer provides a preliminary reference for the next layer. Such a "coarse-to-fine" strategy avoids the local optimum problem that may occur when directly estimating the deformation field from a high-resolution image. Compared with generating a deformation field through a network model at one time, the embodiments of the present application can obtain a smoother deformation field.

[0177] Of course, in some other embodiments, the feature extraction module 100 may also extract a moving feature map of only one scale and a fixed feature map of only one scale. In this case, the forward deformation field prediction branch may include only one layer of the first iterative correction module, and the reverse deformation field prediction branch may include only one layer of the second iterative correction module. Refer to Figure 5 , in this case, after obtaining the second deformation field output by the first optimization fusion sub-module and the third deformation field output by the second optimization fusion sub-module, the second deformation field is determined as the final forward deformation field, and the third deformation field is determined as the final reverse deformation field. For example, refer to Figure 6 , the image registration model in this case may include only the fourth layer.

[0178] In this case, in the forward deformation field prediction branch, a deformation field estimation operation and a warping operation are also performed according to the reverse deformation field provided by the reverse deformation field prediction branch to achieve forward and reverse registration information estimation and information interaction, which can also further improve the accuracy and robustness of the image registration process.

[0179] In some embodiments, the first optimization fusion sub-module and the second optimization fusion sub-module may include a Channel Aggregation (CA) module.

[0180] It should be noted that during the image registration process, the fixed image and the moving image or the features after intermediate deformation need to be stitched together and then deeply fused. However, if the channels are simply added or stitched together, it is often difficult to fully exploit the complementary information between channels, and there is also a lack of a mechanism for screening redundant or irrelevant channels. In response to this, the embodiments of the present application use the CA module to enable the network model to dynamically focus on more valuable channel features and improve the discriminative power of the model.

[0181] Refer to Figure 7 The schematic block diagram of the CA module provided by the embodiments of the present application shown, which may include multiple network modules such as ChannelConcatenate, Conv1×1×1, GELU Activation, Channel Aggregation, Adjustment, and Conv1×1×1. The input of the CA module includes a moving feature map and a fixed feature map. σ is a coefficient.

[0182] Channel Concatenate is used for channel connection. Conv1×1×1 is used to transform the channel dimension and effectively integrate the difference information of the "fixed and moving" features.

[0183] After introducing the network structure of the image registration model provided in the embodiments of the present application, the model training stage and the inference stage of the image registration model will be introduced separately below.

[0184] See Figure 8 , which is a schematic block diagram of the training process of the image registration model provided in the embodiments of the present application. The training process may include the following steps:

[0185] Step S801: Obtain training sample data. The training sample data includes at least one pair of images, and the pair of images includes a fixed image sample and a moving image sample.

[0186] The fixed image sample and the moving image sample may be three-dimensional medical images.

[0187] Step S802: Input the fixed image sample and the moving image sample into the image registration model to obtain the final forward deformation field and the final inverse deformation field output by the image registration model.

[0188] The image registration model may first extract one or more scales of fixed feature maps from the fixed image sample and one or more scales of moving feature maps from the moving image sample; then, through the forward deformation field prediction branch and the inverse deformation field prediction branch, process the one or more scales of fixed feature maps and moving feature maps to obtain the final forward deformation field and the final inverse deformation field.

[0189] For example, when the network structure of the image registration model is Figure 6 the structure shown, the final forward deformation field is Fφ, and the final inverse deformation field is Rφ.

[0190] For the specific structure and internal processing flow of the image registration model, please refer to the foregoing, and details are not described herein again.

[0191] Step S803: Calculate the forward loss using the forward loss function according to the final forward deformation field, the fixed image sample, and the moving image sample.

[0192] Step S804: Calculate the inverse loss using the inverse loss function according to the final inverse deformation field, the fixed image sample, and the moving image sample.

[0193] Step S805: Obtain the total loss according to the forward loss and the inverse loss, and adjust the parameters of the image registration model according to the total loss.

[0194] Exemplarily, the total loss function of the embodiments of the present application is shown in Equation 1 below.

[0195]

[0196]

[0197] Among them, represents the similarity metric before and after image registration, represents the regularization constraint on the deformation field. λ1 to λ4 are hyperparameters.

[0198] Forward loss function is Backward loss function is

[0199] Among them, φ is the deformation field, φ m→f is the forward deformation field, φ m→f is the backward deformation field.

[0200] The local normalized cross-correlation is adopted as the similarity metric, as shown in Equation 2 below. Among them and represent the average voxel value within a local window of size n 3 centered at voxel p.

[0201]

[0202] Among them, p i is the i-th voxel.

[0203] The spatial gradient is adopted as the regularization constraint on the deformation field is represented. Among them, u(p) is the spatial gradient of the deformation field u. The spatial gradient is approximated using forward differences, that is p {x,y,z} is the three-dimensional coordinate of voxel p.

[0204] Step S806: Iterate multiple times until the total loss tends to be stable, and obtain the trained image registration model.

[0205] It should be noted that in the model, a forward deformation field prediction branch and a backward deformation field prediction branch are explicitly set, so that parameters that take into account both the forward flow field and the backward flow field can be learned during the model training phase. In addition, separate similarity metric loss functions are used to perform constraint learning in the two branches to achieve the accurate registration effect of each branch.

[0206] After introducing the model training phase, the model inference phase will be introduced and described below in combination with Figure 9 the flowchart of the medical image registration method shown in

[0207] As Figure 9 shown, the medical image registration method may include the following process steps:

[0208] Step S901: Obtain a fixed image and a moving image.

[0209] Step S902: Extract at least one scale of fixed feature maps from the fixed image and at least one scale of moving feature maps from the moving image through the feature extraction module of the image registration model; the image registration model includes a feature extraction module, a forward deformation field prediction branch, a reverse deformation field prediction branch, and a registration module.

[0210] The feature extraction module can extract one or more scales of fixed feature maps and moving feature maps.

[0211] Step S903: Process at least one scale of fixed feature maps through the reverse deformation field prediction branch to obtain at least one scale of reverse deformation fields.

[0212] In the model inference stage, if there is no need to output the reverse registered image, the final reverse deformation field can be selected not to be output. Only the intermediate deformation field needs to be output. For example, in Figure 6 , only reverse deformation fields such as Rφ4 to Rφ1 can be obtained, without obtaining the final reverse deformation field Rφ. Of course, it is also possible to select to output Rφ. This is not limited here. For the specific structure and processing process of the reverse deformation field prediction branch, please refer to the previous text and will not be elaborated here.

[0213] Step S904: Process at least one scale of reverse deformation fields and at least one scale of moving feature maps through the forward deformation field prediction branch to obtain the target forward deformation field.

[0214] In some embodiments, in the forward deformation field prediction branch, for each scale, after obtaining the forward estimated deformation field based on the first deformation field obtained by taking the negative of the reverse deformation field and the forward deformation field, a warping operation is performed on the forward estimated deformation field and the moving feature maps to obtain the estimated information feature maps; based on the estimated information feature maps and the moving feature maps, a second deformation field is obtained, and the target forward deformation field is obtained based on the second deformation field.

[0215] The forward deformation field is obtained based on the moving feature maps or the second deformation field of the previous scale.

[0216] For example, in Figure 6 , the target forward deformation field is Fφ. For the specific structure and processing process of the forward deformation field prediction branch, please refer to the previous text and will not be elaborated here. For the specific process of obtaining the second deformation field, please refer to the previous text and will not be elaborated here.

[0217] Step S905: Perform registration processing on the target forward deformation field and the moving image through the registration module to obtain the registered image.

[0218] For example, in Figure 6Among them, the target forward deformation field is Fφ, and the registration module performs a warping operation on Fφ and Im to obtain a forward registered image (i.e., Warped Moving). Figure 6 The Warped Fixed in it is a reverse registered image and is an optional output result.

[0219] The image registration model in the embodiments of the present application not only focuses on the forward deformation field from the moving image to the fixed image, but also focuses on the reverse deformation field from the fixed image to the moving image, and thus can obtain a forward deformation field with better reversibility. The forward deformation field with better reversibility can make the image registration accuracy higher.

[0220] In some possible implementation manners, the feature extraction module is used to extract moving feature maps of n scales and fixed feature maps of n scales, where n is greater than or equal to 2;

[0221] The forward deformation field prediction branch includes n layers of first iterative correction modules and a first upsampling module disposed between two adjacent layers of first iterative correction modules; each layer of first iterative correction module includes a registration estimation sub-module, a first warping sub-module, and a first optimization fusion sub-module;

[0222] Processing at least one scale of the reverse deformation field and at least one scale of the moving feature maps through the forward deformation field prediction branch to obtain the target forward deformation field, including:

[0223] In the m-th layer of the first iterative correction module, the first warping sub-module performs a warping operation on the moving feature map of the k-th scale and the forward deformation field of the k-th scale to obtain a first warped feature map, where m and k are greater than or equal to 1 and less than or equal to n;

[0224] After the registration estimation sub-module performs a forward deformation field estimation operation on the forward deformation field of the k-th scale and the first deformation field of the k-th scale to obtain the forward estimated deformation field of the k-th scale, a warping operation is performed on the forward estimated deformation field of the k-th scale and the moving feature map of the k-th scale to obtain the estimated information feature map of the k-th scale. The first deformation field of the k-th scale is obtained by taking the negative of the reverse deformation field of the k-th scale;

[0225] The first optimization fusion sub-module performs an optimization fusion process on the first warped feature map of the k-th scale, the estimated information feature map of the k-th scale, and the moving feature map of the k-th scale to obtain the second deformation field of the k-th scale;

[0226] When k is greater than or equal to 2, the first upsampling module between the m-th layer and the m + 1-th layer upsamples the second deformation field of the k-th scale to obtain the forward deformation field of the k - 1-th scale;

[0227] When k = 1, the second deformation field of the first scale is determined as the target forward deformation field;

[0228] When m = 1, k = n; when m = n, k = 1; when m = 1, the forward deformation field of the nth scale is generated according to the moving feature map of the nth scale.

[0229] In some possible implementation manners, the reverse deformation field prediction branch includes n layers of second iterative correction modules and a second upsampling module disposed between two adjacent layers of second iterative correction modules;

[0230] Each layer of second iterative correction module includes a second warping sub-module and a second optimization fusion sub-module;

[0231] Processing the fixed feature maps of at least one scale through the reverse deformation field prediction branch to obtain the reverse deformation fields of at least one scale, including:

[0232] In the mth layer of second iterative correction module, the fixed feature map of the kth scale and the reverse deformation field of the kth scale are warped through the second warping sub-module to obtain the second warped feature map of the kth scale;

[0233] The second warped feature map of the kth scale and the fixed feature map of the mth scale are optimized and fused through the second optimization fusion sub-module to obtain the third deformation field of the kth scale;

[0234] When k is greater than or equal to 2, the third deformation field of the kth scale is upsampled through the second upsampling module between the mth layer and the m + 1th layer to obtain the reverse deformation field at the k - 1th scale;

[0235] Wherein, when m = 1, the reverse deformation field at the nth scale is generated according to the fixed feature map of the nth scale.

[0236] In some possible implementation manners, the first optimization fusion sub-module and the second optimization fusion sub-module include a channel aggregation module.

[0237] In some possible implementation manners, the feature extraction module includes n - 1 layers of downsampling residual convolution modules;

[0238] At this time, the process of extracting the fixed feature maps of at least one scale from the fixed image and the moving feature maps of at least one scale from the moving image through the feature extraction module of the pre-trained image registration model may include:

[0239] In the jth layer of downsampling convolution module, after obtaining the moving feature map of the jth scale, a downsampling operation and a residual convolution operation are performed on the moving feature map of the jth scale to obtain the moving feature map of the j + 1th scale;

[0240] In the downsampling convolutional module of the j-th layer, after obtaining the fixed feature map of the j-th scale, perform a downsampling operation and a residual convolution operation on the fixed feature map of the j-th scale to obtain the fixed feature map of the (j + 1)-th scale; j is greater than or equal to 1 and less than or equal to n - 1;

[0241] When j = 1, the moving feature map of the j-th scale is extracted from the moving image, and the fixed feature map of the j-th scale is extracted from the fixed image.

[0242] In some possible implementation manners, the downsampling residual convolution module includes a downsampling sub-module and a residual convolution sub-module; the residual convolution sub-module includes at least two parallel branch channels, and each branch channel uses a three-dimensional convolution kernel of a different size.

[0243] It should be understood that the magnitudes of the sequence numbers of the steps in the above embodiments do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0244] Corresponding to the medical image registration method described in the above embodiments, Figure 10 The structural schematic diagram of the medical image registration device provided by the embodiments of the present application is shown. For the convenience of description, only the parts related to the embodiments of the present application are shown.

[0245] Refer to Figure 10 , the medical image registration device includes:

[0246] An image acquisition module 1010, configured to acquire a fixed image and a moving image;

[0247] A feature extraction module 1020, configured to extract at least one scale of fixed feature maps from the fixed image and at least one scale of moving feature maps from the moving image through the feature extraction module of the image registration model; the image registration model includes a feature extraction module, a forward deformation field prediction branch, a reverse deformation field prediction branch, and a registration module;

[0248] A reverse deformation field prediction module 1030, configured to process at least one scale of fixed feature maps through the reverse deformation field prediction branch to obtain at least one scale of reverse deformation fields;

[0249] A forward deformation field prediction module 1040, configured to process at least one scale of reverse deformation fields and at least one scale of moving feature maps through the forward deformation field prediction branch to obtain a target forward deformation field;

[0250] A registration processing module 1050, configured to perform registration processing on the target forward deformation field and the moving image through the registration module to obtain a registered image.

[0251] It should be noted that for the information interaction, execution process, etc. between the above-mentioned devices / modules, since they are based on the same concept as the method embodiments of the present application, for their specific functions and the technical effects brought about, reference may be specifically made to the method embodiment section, and details will not be elaborated here.

[0252] Figure 11 This is a schematic block diagram of the structure of the electronic device provided by the embodiments of the present application. As Figure 11 shown, the electronic device 11 of this embodiment includes: at least one processor 110 ( Figure 11 only one is shown in the figure), a memory 111, and a computer program 112 stored in the memory 111 and executable on the at least one processor 110. When the processor 110 executes the computer program 112, the steps in any of the above method embodiments are implemented.

[0253] The electronic device may include, but is not limited to, a processor 110 and a memory 111. Those skilled in the art can understand that Figure 11 this is only an example of the electronic device 11, and does not constitute a limitation on the electronic device 11. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0254] The so-called processor 110 may be a central processing unit (CPU), and the processor 110 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0255] In some embodiments, the memory 111 may be an internal storage unit of the electronic device 11, such as a hard disk or memory of the electronic device 11. In other embodiments, the memory 111 may also be an external storage device of the electronic device 11, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 11. Further, the memory 111 may also include both the internal storage unit and the external storage device of the electronic device 11. The memory 111 is used to store an operating system, application programs, a BootLoader, data, and other programs, such as program codes of the computer program. The memory 111 may also be used to temporarily store data that has been output or will be output.

[0256] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0257] In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0258] The embodiments of the present application further provide an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the foregoing method embodiments are implemented.

[0259] The embodiments of the present application further provide a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the foregoing method embodiments can be implemented.

[0260] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, it enables the electronic device to execute and implement the steps in the above-mentioned method embodiments.

[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-mentioned method embodiments of the present application, a computer program can be used to instruct the relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps in the above-mentioned method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the photographing device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disc, etc. In some jurisdictions, according to legislation and patent practice, the computer-readable medium cannot be an electrical carrier signal and a telecommunication signal.

[0262] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0263] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0264] In the embodiments provided in the present application, it should be understood that the disclosed devices, electronic devices, and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0265] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0266] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A medical image registration method, characterized in that: include: Acquire fixed and moving images; Extracting at least one fixed feature map of a scale from the fixed image and extracting at least one mobile feature map of a scale from the mobile image through a feature extraction module of an image registration model; the image registration model comprises the feature extraction module, a forward deformation field prediction branch, a reverse deformation field prediction branch and a registration module; Processing the fixed feature map of at least one scale through the inverse deformation field prediction branch to obtain an inverse deformation field of at least one scale; Processing the inverse deformation field of at least one scale and the movement feature map of at least one scale through the forward deformation field prediction branch to obtain a target forward deformation field; The registration module performs registration processing on the target forward deformation field and the moving image to obtain a registered image.

2. The method according to claim 1, characterized in that Processing the inverse deformation field of at least one scale and the movement feature map of at least one scale through the forward deformation field prediction branch to obtain a target forward deformation field includes: For each scale, a forward estimated deformation field is obtained according to a first deformation field obtained by negating the inverse deformation field and a forward deformation field; a warping operation is performed on the forward estimated deformation field and the movement feature map to obtain an estimated information feature map; a second deformation field is obtained according to the estimated information feature map and the movement feature map; Obtaining the target forward deformation field based on the second deformation field; The forward deformation field is obtained according to the movement feature map or the second deformation field of the previous scale.

3. The method according to claim 1 or 2, characterized in that The feature extraction module is used to extract the mobile feature maps of n scales and the fixed feature maps of n scales, where n is greater than or equal to 2; The forward deformation field prediction branch includes n layers of first iterative correction modules and a first upsampling module arranged between two adjacent layers of the first iterative correction modules; each layer of the first iterative correction module includes a registration estimation submodule, a first distortion submodule and a first optimization fusion submodule; Processing the inverse deformation field of at least one scale and the movement feature map of at least one scale through the forward deformation field prediction branch to obtain a target forward deformation field includes: In the first iterative correction module of the mth layer, the first distortion submodule performs a distortion operation on the kth scale movement feature map and the kth scale forward deformation field to obtain a first distortion feature map, where m and k are greater than or equal to 1 and less than or equal to n; Performing a forward deformation field estimation operation on the forward deformation field of the k-th scale and the first deformation field of the k-th scale through the registration estimation submodule, after obtaining the forward estimated deformation field of the k-th scale, performing a distortion operation on the forward estimated deformation field of the k-th scale and the movement feature map of the k-th scale to obtain the estimated information feature map of the k-th scale, wherein the first deformation field of the k-th scale is obtained by performing a negative operation on the inverse deformation field of the k-th scale; The first distortion feature map at the k-th scale, the estimated information feature map at the k-th scale, and the movement feature map at the k-th scale are optimized and fused by the first optimization fusion submodule to obtain a second deformation field at the k-th scale; When k is greater than or equal to 2, upsampling the second deformation field of the kth scale by the first upsampling module between the mth layer and the m+1th layer to obtain a forward deformation field of the k-1th scale; When k=1, the second deformation field at the first scale is determined as the target forward deformation field; When m=1, k=n; when m=n, k=1; when m=1, the forward deformation field of the nth scale is generated based on the moving feature map of the nth scale, and the forward deformation fields of other scales are obtained based on the second deformation field of the previous scale.

4. The method according to claim 3, characterized in that The inverse deformation field prediction branch includes n layers of second iterative correction modules and a second up-sampling module arranged between two adjacent layers of the second iterative correction modules; The second iterative correction module of each layer includes a second distortion submodule and a second optimization fusion submodule; Processing the fixed feature map of at least one scale through the inverse deformation field prediction branch to obtain an inverse deformation field of at least one scale includes: In the second iterative correction module of the mth layer, the fixed feature map of the kth scale and the inverse deformation field of the kth scale are distorted by the second distortion submodule to obtain a second distorted feature map of the kth scale; Performing an optimization fusion process on the second distorted feature map at the kth scale and the fixed feature map at the mth scale through the second optimization fusion submodule to obtain a third deformation field at the kth scale; When k is greater than or equal to 2, upsampling the third deformation field at the kth scale through the second upsampling module between the mth layer and the m+1th layer to obtain an inverse deformation field at the k-1th scale; When m=1, the inverse deformation field of the nth scale is generated according to the fixed feature map of the nth scale.

5. The method according to claim 4, characterized in that The feature extraction module includes an n-1 layer down-sampling residual convolution module; Extracting at least one fixed feature map of a scale from the fixed image and extracting at least one mobile feature map of a scale from the mobile image by a feature extraction module of an image registration model, comprising: In the downsampling convolution module of the jth layer, after obtaining the motion feature map of the jth scale, a downsampling operation and a residual convolution operation are performed on the motion feature map of the jth scale to obtain a motion feature map of the j+1th scale; In the downsampling convolution module of the jth layer, after obtaining the fixed feature map of the jth scale, a downsampling operation and a residual convolution operation are performed on the fixed feature map of the jth scale to obtain a fixed feature map of the j+1th scale; j is greater than or equal to 1 and less than or equal to n-1; When j=1, the j-th scale moving feature map is extracted from the moving image, and the j-th scale fixed feature map is extracted from the fixed image.

6. The method according to claim 5, characterized in that The down-sampling residual convolution module includes a down-sampling submodule and a residual convolution submodule; the residual convolution submodule includes at least two parallel branch channels, each branch channel uses a three-dimensional convolution kernel of a different size; the first optimization fusion submodule and the second optimization fusion submodule include a channel aggregation module.

7. The method according to claim 1, characterized in that The training process of the image registration model includes: Acquire training sample data, wherein the training sample data includes at least one image pair, and the image pair includes a fixed image sample and a moving image sample; Inputting the fixed image sample and the moving image sample into the image registration model to obtain a final forward deformation field and a final inverse deformation field output by the image registration model; Calculating a forward loss using a forward loss function according to the final forward deformation field, the fixed image samples, and the moving image samples; Calculating an inverse loss using an inverse loss function according to the final inverse deformation field, the fixed image samples and the moving image samples; Obtaining a total loss according to the forward loss and the reverse loss; Adjusting parameters of the image registration model according to the total loss; Iterate multiple times until the total loss tends to be stable, and obtain the trained image registration model.

8. A medical image registration device, characterized in that: include: An image acquisition module, used for acquiring fixed images and moving images; A feature extraction module, used to extract a fixed feature map of at least one scale from the fixed image and a moving feature map of at least one scale from the moving image through the feature extraction module of the image registration model; the image registration model includes the feature extraction module, a forward deformation field prediction branch, a reverse deformation field prediction branch and a registration module; An inverse deformation field prediction module, used for processing the fixed feature map of at least one scale through the inverse deformation field prediction branch to obtain an inverse deformation field of at least one scale; A forward deformation field prediction module, used for processing the reverse deformation field of at least one scale and the movement feature map of at least one scale through the forward deformation field prediction branch to obtain a target forward deformation field; The registration processing module is used to perform registration processing on the target forward deformation field and the moving image through the registration module to obtain a registered image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images