Image registration method, terminal device, and computer-readable storage medium

Abstract

The application discloses an image registration method, a terminal device and a computer readable storage medium, relates to the technical field of image processing, and discloses an image registration method, which comprises the following steps: acquiring an image to be registered, wherein the image to be registered comprises a reference image and a moving image; performing at least one discrete wavelet transform on the image to be registered to obtain the image to be registered at at least two different scales; inputting the image to be registered at multiple different scales into a pre-trained deformation field generation model to generate a target deformation field corresponding to the moving image; and registering the moving image according to the target deformation field to generate a registered image corresponding to the moving image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image registration method, a terminal device, and a computer-readable storage medium. Background Technology

[0002] Image registration refers to the process of precisely aligning multiple images or different parts of an image. This process is widely used in fields such as computer vision, medical image processing, and remote sensing image processing.

[0003] In related technologies, image registration involves directly inputting a moving image and a reference image into a registration network. The registration network extracts global features from both the moving and reference images, and then generates a deformation field of the moving image based on these global features. This deformation field is then fused with the moving image to obtain a registered image. However, directly inputting images into the registration network leads to a gradual loss of details and features during the registration process, resulting in a decrease in the quality and accuracy of the registered image.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide an image registration method, terminal device, and computer-readable storage medium, aiming to solve the technical problem of low quality and accuracy of registered images.

[0006] To achieve the above objectives, this application proposes an image registration method, which includes:

[0007] Acquire the image to be registered, which includes a reference image and a moving image;

[0008] Perform at least one discrete wavelet transform on the image to be registered to obtain at least two images of different scales;

[0009] Multiple images to be registered at different scales are input into a pre-trained deformation field generation model to generate the target deformation field corresponding to the moving image;

[0010] The moving image is registered based on the target deformation field to generate a registered image corresponding to the moving image.

[0011] In one embodiment, the deformation field generation model includes at least two levels of sub-models; the target deformation field is the output of the last-level sub-model; the sub-model includes a pre-registration module, a deformation field generation module, and an upsampling module; the output of the pre-registration module is connected to the input of the deformation field generation module, and the output of the deformation field generation module is connected to the input of the upsampling module; the input of the pre-registration module is connected to the output of the upsampling module of the previous-level sub-model; the input of the sub-model is the image to be registered at the corresponding scale and / or the sub-deformation field output by the upsampling module of the previous-level sub-model; wherein,

[0012] The pre-registration module is used to perform a first pre-registration on the moving image in the image to be registered based on the sub-deformation field output by the upsampling module of the previous level sub-model, so as to obtain the moving image after the first pre-registration.

[0013] The deformation field generation module is used to generate an initial sub-deformation field based on the moving image after the first pre-registration and the reference image in the image to be registered;

[0014] The upsampling module is used to upsample the initial sub-deformation field to obtain the sub-deformation field.

[0015] In one embodiment, the sub-model further includes a first fusion module; the output of the first fusion module is connected to the input of the upsampling module, and the input of the first fusion module is connected to the output of the deformation field generation module and the output of the upsampling module of the previous-level sub-model; wherein,

[0016] The first fusion module is used to fuse the initial sub-deformation field and the sub-deformation field output by the upsampling module of the previous level sub-model to obtain a fused sub-deformation field;

[0017] The upsampling module is used to upsample the fused sub-deformation field to obtain the sub-deformation field.

[0018] In one embodiment, the output of the first fusion module is also connected to the input of the pre-registration module; wherein,

[0019] The pre-registration module is used to perform a second pre-registration on the moving image in the image to be registered or the moving image after the first pre-registration according to the fusion sub-deformation field, so as to obtain the moving image after the second pre-registration.

[0020] The deformation field generation module is used to generate a target sub-deformation field based on the moving image after the second pre-registration and the reference image in the image to be registered;

[0021] The upsampling module is used to upsample the target sub-deformation field to obtain the sub-deformation field.

[0022] In one embodiment, the deformation field generation module includes at least a local feature module, a global feature module, a frequency domain feature module, and a second fusion module; the outputs of the local feature module, the global feature module, and the frequency domain feature module are respectively connected to the input of the second fusion module; wherein,

[0023] The local feature module is used to generate a first sub-deformation field based on the local features of the moving image and the reference image in the image to be registered, which are output by the pre-registration module.

[0024] The global feature module is used to generate a second sub-deformation field based on the global features of the moving image and the reference image in the image to be registered, which are output by the pre-registration module after registration.

[0025] The frequency domain feature module is used to generate a third sub-deformation field and a fourth sub-deformation field based on the frequency domain features of the moving image and the reference image in the image to be registered, which are output by the pre-registration module after registration.

[0026] The second fusion module is used to fuse the first sub-deformation field, the second sub-deformation field, the third sub-deformation field, and the fourth sub-deformation field to obtain the initial sub-deformation field.

[0027] In one embodiment, the frequency domain feature module includes a transform module, a first model, and a second model; the output of the transform module is connected to the inputs of the first model and the second model, respectively; wherein,

[0028] The transformation module is used to perform Fourier transform on the moving image output by the pre-registration module to obtain the amplitude image and phase image corresponding to the moving image, and to perform Fourier transform on the reference image in the image to be registered to obtain the amplitude image and phase image corresponding to the reference image, wherein the amplitude image corresponding to the moving image and the amplitude image corresponding to the reference image are output to the first model, and the phase image corresponding to the moving image and the phase image corresponding to the reference image are output to the second model;

[0029] The first model is used to generate the third sub-deformation field based on the amplitude features of the amplitude image corresponding to the moving image and the amplitude image corresponding to the reference image output by the transformation module.

[0030] The second model is used to generate the fourth sub-deformation field based on the phase features of the phase image corresponding to the moving image and the phase image corresponding to the reference image output by the transformation module.

[0031] In one embodiment, the local feature module includes at least a partitioning module, a third model, and a third fusion module; the output of the partitioning module is connected to the input of the third model, and the output of the third model is connected to the input of the third fusion module; wherein,

[0032] The partitioning module is used to determine the number of blocks based on the partitioning parameters and dimensions of the image to be registered;

[0033] Used to segment the image to be registered into block images corresponding to the number of blocks;

[0034] The third model is used to generate sub-deformation fields for each of the segmented images based on the local features of each segmented image.

[0035] The third fusion module is used to fuse the various sub-deformation fields to generate the first sub-deformation field.

[0036] In one embodiment, before the step of inputting multiple images to be registered at different scales into a pre-trained deformation field generation model to generate the target deformation field corresponding to the moving image, the method further includes:

[0037] Obtain the registration image to be trained, the registration image including the moving image and the reference image;

[0038] Perform the step of performing at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales until the target deformation field of the moving image is obtained;

[0039] Acquire the moving image, reference image, sub-deformation field, and the moving image after the second pre-registration of the sub-model at each level in the deformation field generation model;

[0040] Based on the moving image, the reference image, the sub-deformation field, and the moving image after the second pre-registration, determine the loss function value of the sub-model at each level;

[0041] Determine the sum of the loss function values ​​for each level of the sub-model;

[0042] If the change value of the sum within a preset time period is greater than a preset range, the model parameters of the sub-model at each level are backpropagated, and the model parameters of the sub-model at each level are shared to update the model parameters of the sub-model at each level.

[0043] Continue with the step of obtaining the registration image to be trained;

[0044] If the change in the sum within a preset time period is less than or equal to a preset range, the training of the deformation field generation model ends.

[0045] In addition, to achieve the above objectives, this application also proposes a terminal device, the terminal device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image registration method as described above.

[0046] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the image registration method described above.

[0047] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the image registration method described above.

[0048] This application obtains an image to be registered, which includes a reference image and a moving image. Then, it performs at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales. Subsequently, it inputs the multiple images to be registered at different scales into a pre-trained deformation field generation model to generate a target deformation field corresponding to the moving image. Based on the target deformation field, it registers the moving image to generate a registered image corresponding to the moving image. This can improve the quality and accuracy of the registered image by improving the accuracy of the target deformation field. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating an embodiment of the image registration method involved in this application.

[0052] Figure 2 A deformation field generation model diagram for the image registration method involved in this application;

[0053] Figure 3 This is a schematic diagram of the deformation field generation module of the image registration method involved in this application;

[0054] Figure 4 This is a schematic diagram of the frequency domain feature module of the image registration method involved in this application;

[0055] Figure 5 This is a schematic diagram of the local feature module of the image registration method involved in this application;

[0056] Figure 6 This is a flowchart illustrating Embodiment 2 of the image registration method involved in this application;

[0057] Figure 7 This is a schematic diagram illustrating the training of the deformation field generation model for the image registration method involved in this application;

[0058] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the image registration method in the embodiments of this application.

[0059] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0061] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0062] The main solution of this application embodiment is as follows: by acquiring the image to be registered, which includes a reference image and a moving image, and then performing at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales, and then inputting the multiple images to be registered at different scales into a pre-trained deformation field generation model to generate a target deformation field corresponding to the moving image, and registering the moving image according to the target deformation field to generate a registered image corresponding to the moving image. This can improve the quality and accuracy of the registered image by improving the accuracy of the target deformation field.

[0063] In this embodiment, for ease of description, the following description will focus on the identification terminal device as the execution subject.

[0064] In related technologies, image registration directly inputs the moving image and a reference image into a registration network. The registration network extracts global features from both the moving and reference images, and then generates a deformation field of the moving image based on these global features. This deformation field is then fused with the moving image to obtain the registered image. However, directly inputting the image into the registration network causes a gradual loss of details and features during the registration process, leading to a decrease in the quality and accuracy of the registered image.

[0065] This application provides a solution that improves the quality and accuracy of registered images by increasing the precision of the target deformation field.

[0066] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or a terminal device capable of performing the above functions. The following description uses a terminal device as an example to illustrate this embodiment and the subsequent embodiments.

[0067] Based on this, embodiments of this application provide an image registration method, referring to... Figure 1 , Figure 1 This is a schematic flowchart of the first embodiment of the image registration method of this application.

[0068] In this embodiment, the image registration method includes steps S10 to S40:

[0069] Step S10: Obtain the image to be registered, which includes a reference image and a moving image;

[0070] In this embodiment, the moving image and the reference image can be images acquired by the same acquisition device or different acquisition devices for the same scene or different scenes. The moving image and the reference image can be medical images or other images; this embodiment does not specifically limit them.

[0071] It should be noted that the moving image and the reference image in the image to be registered are of the same scale.

[0072] It is understandable that the moving image is the image that needs to be registered, and the reference image is the registration target of the moving image. That is, the registered image obtained after registering the moving image should be aligned with the reference image.

[0073] Step S20: Perform at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales;

[0074] In this embodiment, the scale of the image to be registered will be reduced after performing Discrete Wavelet Transform (DWT) on the image to be registered.

[0075] It should be noted that performing at least one discrete wavelet transform on the image to be registered means performing a discrete wavelet transform on the original image to be registered, and then performing a discrete wavelet transform on the image to be registered obtained after the discrete wavelet transform. The image to be registered obtained after each discrete wavelet transform and the original image to be registered are the images to be registered at least at two different scales.

[0076] Optionally, this application does not limit the scale change value of the discrete wavelet transform; users can choose the scale change value according to their needs. For example, the scale change value of the discrete wavelet transform can be set to 2, then a discrete wavelet transform function that can reduce the image by 2 scales can be selected to perform the discrete wavelet transform on the image to be registered. The discrete wavelet transform function can also be called the wavelet basis function.

[0077] It should be noted that using discrete wavelet transform to scale the image to be registered can preserve the complete original information of the scaled image, thereby avoiding the loss of details and features in the image in the deformation field generation model, so as not to reduce the accuracy of the registered image.

[0078] Step S30: Input multiple images to be registered at different scales into a pre-trained deformation field generation model to generate the target deformation field corresponding to the moving image;

[0079] In this embodiment, the deformation field generation model includes at least two levels of sub-models. The number of sub-models is the same as the number of images to be registered. Images to be registered at corresponding scales are used as inputs to the corresponding level of the sub-model. The sub-models at each level are interconnected, forming a recursive network. After the first-level sub-model outputs a sub-deformation field, this sub-deformation field is used as input to the second-level sub-model for further processing, thus outputting the sub-deformation field of the second-level sub-model. This process continues until the last level of the sub-model is reached, and the initial sub-deformation field output by the last-level model is the target deformation field.

[0080] Optionally, the images to be registered can be assigned to each level of the sub-model in ascending order of sub-model level, and then in ascending order of scale. Therefore, since the original images to be registered have not undergone discrete wavelet transform and have the largest scale, they will be assigned to the last level of the sub-model, while the images to be registered with the smallest scale will be assigned to the first level of the sub-model.

[0081] For example, the images to be registered include a first image, a second image, and a third image, with scales decreasing sequentially. The sub-models include a first-level sub-model, a second-level sub-model, and a third-level sub-model. The ascending order of the sub-model levels is first level, second level, and third level. Therefore, starting from the ascending order of the sub-model levels, the images to be registered are assigned to each level of the sub-model in ascending order of scale. Specifically, the third image, with the smallest scale, is assigned to the first-level sub-model; the second image, with the second smallest scale, is assigned to the second-level sub-model; and the first image, with the largest scale, is assigned to the third-level sub-model.

[0082] Optionally, the number of sub-models in the deformation field generation model is preferably set to 4, that is, the deformation field generation model is preferably composed of sub-models at 4 levels. Since the number of images to be registered corresponds one-to-one with the number of sub-models in the deformation field generation model, the number of images to be registered is also preferably 4. When implementing the image registration method disclosed in this application, those skilled in the art can modify the number of sub-models in the deformation field generation model according to actual needs, and this implementation does not specifically limit this.

[0083] Optionally, the number of sub-models in the deformation field generation model can be set according to the scale of the original image to be registered. If the scale of the original image to be registered is large, the number of sub-models in the deformation field generation model can be increased; if the scale of the original image to be registered is small, the number of sub-models in the deformation field generation model can be reduced. This embodiment does not impose specific limitations on this.

[0084] Figure 2 This is a model diagram of the deformation field involved in this application. (Refer to...) Figure 2Figure (a) shows the deformation field generation model of this application, which includes at least two levels of sub-models. The target deformation field is the output of the last level of the sub-model. The sub-model includes a pre-registration module, a deformation field generation module, and an upsampling module. The output of the pre-registration module is connected to the input of the deformation field generation module, the output of the deformation field generation module is connected to the input of the upsampling module, and the input of the pre-registration module is connected to the output of the upsampling module of the previous level of the sub-model. The input of the sub-model is the image to be registered at the corresponding scale and / or the sub-deformation field output by the upsampling module of the previous level of the sub-model. The processing flow of each level of the sub-model is as follows: the image to be registered and / or the sub-deformation field output by the upsampling module of the previous level of the sub-model sequentially pass through the pre-registration module, the deformation field generation module, and the upsampling module. The pre-registration module is used to perform the first pre-registration of the moving image in the image to be registered based on the sub-deformation field output by the upsampling module of the previous level sub-model, so as to obtain the moving image after the first pre-registration; the deformation field generation module is used to generate the initial sub-deformation field based on the moving image after the first pre-registration and the reference image in the image to be registered; and the upsampling module is used to upsample the initial sub-deformation field to obtain the sub-deformation field.

[0085] Optionally, the number of levels of the sub-models in the deformation field generation model of this application can be preferably set to 4. That is, the deformation field generation model includes a first-level sub-model, a second-level sub-model, a third-level sub-model, and a fourth-level sub-model. See Appendix [link to Appendix]. Figure 2 Figure (a) in the middle.

[0086] The network architecture of each sub-model level is the same, differing only in the number of sampling layers in the deformation field generation module. Since the scale of the images to be registered differs at each sub-model level, the number of sampling layers in the deformation field generation module can be set according to the scale of the images to be registered, in order to fully extract information from these images. That is, the larger the scale of the image to be registered, the more sampling layers the deformation field generation module should have; the smaller the scale of the image to be registered, the fewer sampling layers the deformation field generation module should have.

[0087] Optionally, since the first-level sub-model has the smallest scale of the image to be registered, and the scale of the image to be registered increases with the level, the number of sampling layers of the deformation field generation module of each level of sub-model can be uniformly set. Since the image to be registered in the last level is the original image, the number of sampling layers of the deformation field generation module of the last level of sub-model can be set first, and then each level can be traversed from the last level downwards. The number of sampling layers of the deformation field generation module of each level of sub-model is the result value obtained by decreasing the number of sampling layers of the deformation field generation module of the previous level of sub-model by a preset value.

[0088] For example, suppose the deformation field generation model of this application includes four levels of sub-models: a first-level sub-model, a second-level sub-model, a third-level sub-model, and a fourth-level sub-model. First, set the sampling layer number of the deformation field generation module of the fourth-level sub-model to 5, with a default value of 1. Then, the result value 4 is obtained by subtracting 1 from the sampling layer number of the deformation field generation module of the third-level sub-model (5 minus 1). The result value 3 is obtained by subtracting 1 from the sampling layer number of the deformation field generation module of the third-level sub-model (4). And so on, with the sampling layer number of the deformation field generation module of the first-level sub-model being 2.

[0089] Optionally, the preset value can be set to the scale change value of the discrete wavelet transform, or it can be set by the user. This embodiment does not impose specific limitations on this. The scale change value of the discrete wavelet transform refers to the amount of scale change of the image to be registered after the discrete wavelet transform. For example, if the scale of the image to be registered is 3, and the scale after the discrete wavelet transform is 2, then the scale change value of the discrete wavelet transform is 1.

[0090] It should be noted that the upsampling module in this application is used to switch the scale of the initial sub-deformation field generated by the deformation field generation module of the current level sub-model to the same scale as the image to be registered for the corresponding sub-model of the next level, thereby obtaining the sub-deformation field of the current level sub-model. This allows the sub-deformation field generated by the deformation field generation module of the current level sub-model to be used in the pre-registration module of the next level sub-model. However, since there are no other sub-models after the last level sub-model, there is no need to set up an upsampling module.

[0091] It should be noted that if the registration module of the current level sub-model does not receive the sub-deformation field output by the upsampling module of the previous level sub-model, the image to be registered is directly input into the deformation field generation module. The deformation field generation module directly generates the initial sub-deformation field based on the moving image and reference image in the image to be registered. Since there are no other sub-models before the first level sub-model, the pre-registration module of the first level sub-model will not receive the sub-deformation field output by the upsampling module of the previous level sub-model. Therefore, the pre-registration module of the first level sub-model directly inputs the image to be registered into the deformation field generation module, which directly generates the initial sub-deformation field based on the moving image and reference image in the image to be registered.

[0092] Optionally, the deformation field generation module of this application can be built based on the U-Net network.

[0093] Optionally, refer to Figure 3 , Figure 3This is a schematic diagram of the deformation field generation module involved in this application. The deformation field generation module includes at least a local feature module, a global feature module, a frequency domain feature module, and a second fusion module. The outputs of the local feature module, global feature module, and frequency domain feature module are respectively connected to the input of the second fusion module. Specifically, the local feature module generates a first sub-deformation field based on the local features of the moving image output by the pre-registration module and the reference image in the image to be registered. The global feature module generates a second sub-deformation field based on the global features of the moving image output by the pre-registration module and the reference image in the image to be registered. The frequency domain feature module generates a third and a fourth sub-deformation field based on the frequency domain features of the moving image output by the pre-registration module and the reference image in the image to be registered. The second fusion module fuses the first, second, third, and fourth sub-deformation fields to obtain an initial sub-deformation field.

[0094] Furthermore, referring to Figure 4 , Figure 4 This is a schematic diagram of the frequency domain feature module involved in this application. The frequency domain feature module includes a transform module, a first model, and a second model. The output of the transform module is connected to the inputs of the first and second models, respectively. Specifically, the transform module performs a Fast Fourier Transform (FFT) on the moving image output by the pre-registration module to obtain the amplitude and phase images corresponding to the moving image, and performs a Fourier Transform on the reference image in the image to be registered to obtain the amplitude and phase images corresponding to the reference image. The amplitude images corresponding to the moving image and the reference image are output to the first model, and the phase images corresponding to the moving image and the reference image are output to the second model. The first model generates a third sub-deformation field based on the amplitude features of the amplitude images corresponding to the moving image and the reference image output by the transform module. The second model generates a fourth sub-deformation field based on the phase features of the phase images corresponding to the moving image and the reference image output by the transform module.

[0095] Furthermore, referring to Figure 5 , Figure 5 This is a schematic diagram of the local feature modules involved in this application. It should be noted that... Figure 5The diagram shows the local feature modules in each level of the sub-model. Each local feature module includes at least a partitioning module, a third model, and a third fusion module. The output of the partitioning module is connected to the input of the third model, and the output of the third model is connected to the input of the third fusion module. Specifically, the partitioning module determines the number of blocks based on the partitioning parameters and dimensions of the image to be registered, dividing the image into blocks corresponding to the number of blocks. The third model generates sub-deformation fields for each block based on its local features. The third fusion module fuses these sub-deformation fields to generate the first sub-deformation field.

[0096] The expression for calculating the number of blocks can be expressed as:

[0097]

[0098] Where S is the number of blocks, C is the partitioning parameter, and n is the dimension of the image to be registered.

[0099] Optionally, the partitioning parameter of the local feature module of the last-level sub-model can be set to 1 / 16. Starting from the last level, each level is traversed downwards, and the partitioning parameter of the local feature module of each level's sub-model is twice that of the previous level. For example, if the partitioning parameter of the local feature module of the last-level sub-model is 1 / 16, then the partitioning parameter of the local feature module of the second-to-last level sub-model is 2 * 1 / 16 = 1 / 8. (See reference...) Figure 5 .

[0100] Furthermore, the global feature module includes a fourth model.

[0101] It should be noted that the first, second, third, and fourth models are all built on the U-Net network. The difference lies in that the first model generates the third sub-deformation field by extracting the amplitude features of the input, the second model generates the fourth sub-deformation field by extracting the phase features of the input, the third model generates the first sub-deformation field by extracting local features of the input, and the fourth model generates the second sub-deformation field by extracting global features of the input. The second fusion module fuses the first, second, third, and fourth sub-deformation fields extracted based on different features, thereby obtaining an initial sub-deformation field that covers all information of the image to be registered. This significantly improves the accuracy of the final generated target deformation field, thus improving the quality and accuracy of the registered image.

[0102] It should be noted that the number of sampling layers of the deformation field generation module of the sub-model mentioned in the first embodiment refers to the number of sampling layers of the first model, the second model, the third model and the fourth model in the deformation field generation module.

[0103] Step S40: Register the moving image according to the target deformation field to generate a registered image corresponding to the moving image.

[0104] In this embodiment, the initial sub-deformation field output by the last level sub-model in the deformation field generation model is the target deformation field. This application registers the moving image in the original image to be registered (without discrete wavelet transform) based on the target deformation field, thereby generating a registered image corresponding to the moving image.

[0105] It is understandable that the input image to be registered in the last level sub-model is the original image to be registered without discrete wavelet transform. Therefore, the scale of the initial sub-deformation field generated by the deformation field generation module in the last level is the same as the scale of the original image to be registered. It can be directly used to register the moving image in the original image to be registered, thereby obtaining the registered image corresponding to the moving image.

[0106] It should be noted that the registration mentioned in this application is achieved by adding the deformation field and the moving image pixel by pixel. The image registration method in the prior art can also be used. This embodiment does not make a specific limitation on this.

[0107] In the technical solution provided in this embodiment, by acquiring the image to be registered, which includes a reference image and a moving image, and then performing at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales, the multiple images to be registered at different scales are then input into a pre-trained deformation field generation model to generate a target deformation field corresponding to the moving image. The moving image is then registered according to the target deformation field to generate a registered image corresponding to the moving image. This can improve the quality and accuracy of the registered image by improving the accuracy of the target deformation field.

[0108] In another alternative implementation, refer to Figure 2 As shown in Figure (b), the sub-model of this application further includes a first fusion module. The output of the first fusion module is connected to the input of the upsampling module, and the input of the first fusion module is connected to the output of the deformation field generation module and the output of the upsampling module of the previous-level sub-model. Specifically, the first fusion module is used to fuse the initial sub-deformation field and the sub-deformation field output by the upsampling module of the previous-level sub-model to obtain a fused sub-deformation field. The upsampling module is used to upsample the fused sub-deformation field to obtain a sub-deformation field. Therefore, the fused sub-deformation field output by the first fusion module of the last-level sub-model is used as the target deformation field.

[0109] Understandably, after adding the first fusion module to the sub-model, the upsampling module changes from upsampling the initial sub-deformation field output by the deformation field generation module to upsampling the fused sub-deformation field output by the first fusion module to obtain the sub-deformation field.

[0110] It should be noted that since there are no other sub-models before the first-level sub-model, the first fusion module of the first-level sub-model directly outputs the initial sub-deformation field as the fused sub-deformation field after receiving the initial sub-deformation field output by the deformation field generation module, without processing the initial sub-deformation field.

[0111] Understandably, this application adds a first fusion module to fuse the initial sub-deformation field output by the deformation field generation module of the current level sub-model with the sub-deformation field output by the upsampling module of the previous level sub-model. This allows the fused sub-deformation field to be applied to the next level sub-model and fused with the initial sub-deformation field output by the deformation field generation module of the next level sub-model. By continuously and recursively fusing the initial sub-deformation fields of each level sub-model, the accuracy of the final generated target deformation field is further improved.

[0112] In another alternative implementation, refer to Figure 2 Figure (c) shows the first fusion module's output, which is also connected to the input of the pre-registration module. The pre-registration module is further configured to perform a second pre-registration on the moving image in the image to be registered or the moving image after the first pre-registration, based on the fused sub-deformation field, to obtain a second pre-registered moving image. The deformation field generation module is further configured to generate a target sub-deformation field based on the second pre-registered moving image and a reference image in the image to be registered. The upsampling module is configured to upsample the target sub-deformation field to obtain the sub-deformation field.

[0113] It is understandable that connecting the inputs of the first fusion module and the pre-registration module indicates that the pre-registration module, deformation field generation module, and first fusion module of each level of sub-model in this application need to be run twice. During the first run, the pre-registration module outputs the first registered moving image and the reference image in the image to be registered, the deformation field generation module outputs the initial sub-deformation field, and the first fusion module fuses the initial sub-deformation field with the sub-deformation field of the previous level sub-model to obtain the fused sub-deformation field. During the second run, the pre-registration module outputs the second registered moving image and the reference image in the image to be registered, the deformation field generation module outputs the target sub-deformation field, and the first fusion module fuses the target sub-deformation field with the sub-deformation field of the previous level sub-model to obtain the fused sub-deformation field again. Only the fused sub-deformation field generated after the final run is applied to the next level sub-model.

[0114] It should be noted that since there are no other sub-models after the last level of sub-model, the first fusion module of the last level does not need to be connected to the input of the pre-registration module. Therefore, the fused sub-deformation field obtained by the first fusion module after fusing the initial sub-deformation field and the sub-deformation field of the previous level can be directly used as the target deformation field. Alternatively, the first fusion module of the last level can also be connected to the input of the pre-registration module. In this case, the fused sub-deformation field obtained by the first fusion module after fusing the target sub-deformation field and the sub-deformation field of the previous level can be used as the target deformation field. This embodiment does not specifically limit this approach.

[0115] Understandably, this application connects the output of the first fusion module with the input of the pre-registration module, enabling each level of sub-model to run twice, which can further improve the accuracy of the target deformation field, thereby improving the quality and accuracy of the registered image.

[0116] In the technical solution provided in this embodiment, by adding a first fusion module to the sub-model, the sub-deformation fields between sub-models at different levels can be fused, thereby further improving the accuracy of the final generated target deformation field, and thus further improving the quality and accuracy of the registered image.

[0117] Reference Figure 6 In the second embodiment, based on any of the above embodiments, before step S30, the method further includes steps S50 to S120:

[0118] Step S50: Obtain the registration image to be trained, the registration image including the moving image and the reference image;

[0119] Step S60: Perform the step of performing at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales until the target deformation field of the moving image is obtained;

[0120] Step S70: Obtain the moving image, reference image, sub-deformation field, and moving image after the second pre-registration of the sub-model at each level in the deformation field generation model;

[0121] Step S80: Determine the loss function value of the sub-model at each level based on the moving image, reference image, sub-deformation field, and the moving image after the second pre-registration;

[0122] In this embodiment, the application sets a loss function for each level of sub-model separately, so that each level of sub-model can update its own loss function value, enabling each level of sub-model to be trained well, resulting in balanced performance of each level of sub-model, thereby improving the accuracy of deformation field generation by each level of sub-model, and thus improving the accuracy of the target deformation field.

[0123] The loss function can be expressed as:

[0124]

[0125] in, For the sub-deformation field output by the sub-model, I Α For moving images, I Β For reference image, This is the moving image after the second pre-registration, where v is the voxel and S is the image size. and I respectively Α I Β , The size of the voxel is p 3 The average voxel value within the local window, p = 9.

[0126] As can be understood, a voxel refers to a pixel in an image in three-dimensional space. Voxels describe the content of an image, while image size describes the dimensions of the image in each dimension.

[0127] Furthermore, for regularization, the L2 norm of the deformation field gradient is used, i.e. To ensure the reasonableness of deformation of each level of sub-model:

[0128]

[0129] in, This represents the gradient difference.

[0130] It should be noted that, It should be approximately 0.

[0131] Step S90: Determine the sum of the loss function values ​​of the sub-models at each level;

[0132] In this embodiment, when the loss function value of each level of sub-model is obtained, the sum of the loss function values ​​of all levels of sub-model is calculated, and the convergence of the deformation field generation model is determined based on the change of the sum.

[0133] Step S100: If the change value of the sum within a preset time period is greater than a preset range, backpropagate the model parameters corresponding to the sub-model at each level, share the model parameters of the sub-models at each level, and update the model parameters of the sub-model at each level.

[0134] In this embodiment, if the change in the total value within a preset time period is greater than a preset range, it indicates that the deformation field generation model has not yet converged. At this time, in addition to backpropagating the model parameters corresponding to each level of sub-model within its own model, the model parameters of each level of sub-model are shared sequentially from the first level upwards. This allows each level of sub-model to be further updated through shared model parameters, thereby accelerating the convergence speed of each level of sub-model and thus improving the convergence speed of the deformation field generation model. (See also...) Figure 7 , Figure 7 This is a schematic diagram illustrating the training of the deformation field generation model involved in this application.

[0135] It should be noted that the model parameters used for sharing in each level of the sub-model in this application refer to all model parameters of the third model of the local feature module and the model parameters generated after the downsampling network in the fourth model of the global feature module.

[0136] Step S110: Continue to execute the step of obtaining the registration image to be trained;

[0137] Step S120: If the change value of the sum within a preset time period is less than or equal to a preset range, the training of the deformation field generation model ends.

[0138] In this embodiment, if the change in the sum within a preset time period is less than or equal to a preset range, it indicates that the deformation field generation model has converged, and the training of the deformation field generation model ends.

[0139] It is understandable that the convergence of the deformation field generation model indicates that the sub-models at each level of the deformation field generation model have converged.

[0140] In the technical solution provided in this embodiment, by acquiring a registration image to be trained, which includes a moving image and a reference image, the step of performing at least one discrete wavelet transform on the image to be trained to obtain at least two images to be trained at different scales is performed until the target deformation field of the moving image is obtained. Then, the moving image, reference image, sub-deformation field, and the moving image after the second pre-registration of each level of the deformation field generation model are acquired. Based on the moving image, reference image, sub-deformation field, and the moving image after the second pre-registration, the loss function value of each level of the sub-model is determined. Then, by determining the sum of the loss function values ​​of each level of the sub-model, if the change value of the sum within a preset time period is greater than a preset range, the model parameters of each level of the sub-model are backpropagated, and the model parameters of the sub-models at each level are shared to update the model parameters of each level of the sub-model. The step of acquiring the registration image to be trained continues. If the change value of the sum within a preset time period is less than or equal to a preset range, the training of the deformation field generation model ends. The training method of the deformation field generation model in this application can improve the convergence speed and performance of the deformation field generation model, thereby improving the accuracy of the target deformation field.

[0141] This application provides a terminal device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the image registration method in Embodiment 1 above.

[0142] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a terminal device suitable for implementing embodiments of this application. The terminal device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 8As shown, the terminal device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the terminal device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows terminal devices to communicate wirelessly or wiredly with other devices to exchange data. Although terminal devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0144] The terminal device provided in this application, employing the image registration method described in the above embodiments, can solve the technical problem of low quality and accuracy of registered images. Compared with the prior art, the beneficial effects of the terminal device provided in this application are the same as those of the image registration method provided in the above embodiments, and other technical features of this terminal device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0145] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0146] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0147] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image registration method in the above embodiments.

[0148] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0149] The aforementioned computer-readable storage medium may be included in the terminal device; or it may exist independently and not assembled into the terminal device.

[0150] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by a terminal device, the terminal device causes the terminal device to: acquire an image to be registered, the image to be registered including a reference image and a moving image; perform at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales; input the multiple images to be registered at different scales into a pre-trained deformation field generation model to generate a target deformation field corresponding to the moving image; and register the moving image according to the target deformation field to generate a registered image corresponding to the moving image.

[0151] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0154] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for performing the above-described image registration method, which can solve the technical problem of low quality and accuracy of registered images. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the image registration method provided in the above embodiments, and will not be repeated here.

[0155] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An image registration method, characterized in that, The image registration method includes: Acquire the image to be registered, which includes a reference image and a moving image; Perform at least one discrete wavelet transform on the image to be registered to obtain at least two images of different scales; Multiple images to be registered at different scales are input into a pre-trained deformation field generation model to generate the target deformation field corresponding to the moving image; The moving image is registered according to the target deformation field to generate a registered image corresponding to the moving image; The deformation field generation model includes at least two levels of sub-models; the target deformation field is the output of the last-level sub-model; each sub-model includes a pre-registration module, a deformation field generation module, and an upsampling module; the output of the pre-registration module is connected to the input of the deformation field generation module, and the output of the deformation field generation module is connected to the input of the upsampling module; when the sub-model is a first-level sub-model, the input of the sub-model is the image to be registered at the corresponding scale; when the sub-model is not a first-level sub-model, the input of the pre-registration module is connected to the output of the upsampling module of the previous-level sub-model, and its input is the image to be registered at the corresponding scale and the sub-deformation field output by the upsampling module of the previous-level sub-model; wherein, The pre-registration module for non-first-level sub-models is used to perform a first pre-registration on the moving image in the image to be registered based on the sub-deformation field output by the upsampling module of the previous-level sub-model, so as to obtain the moving image after the first pre-registration. The deformation field generation module is used to generate an initial sub-deformation field based on the moving image after the first pre-registration and the reference image in the image to be registered; The upsampling module is used to upsample the initial sub-deformation field to obtain the sub-deformation field; The deformation field generation module includes at least a local feature module, a global feature module, a frequency domain feature module, and a second fusion module; the outputs of the local feature module, the global feature module, and the frequency domain feature module are respectively connected to the input of the second fusion module; wherein, The local feature module is used to generate a first sub-deformation field based on the local features of the moving image and the reference image in the image to be registered, which are output by the pre-registration module. The global feature module is used to generate a second sub-deformation field based on the global features of the moving image and the reference image in the image to be registered, which are output by the pre-registration module after registration. The frequency domain feature module is used to generate a third sub-deformation field and a fourth sub-deformation field based on the frequency domain features of the moving image output by the pre-registration module and the reference image in the image to be registered. The frequency domain feature module includes a transformation module, a first model, and a second model. The output of the transformation module is connected to the inputs of the first model and the second model, respectively. The transformation module is used to perform a Fourier transform on the moving image output by the pre-registration module to obtain the amplitude image and phase image corresponding to the moving image, and to perform a Fourier transform on the reference image in the image to be registered to obtain the amplitude image and phase image corresponding to the reference image. The image is a bit image, wherein the amplitude image corresponding to the moving image and the amplitude image corresponding to the reference image are output to the first model, and the phase image corresponding to the moving image and the phase image corresponding to the reference image are output to the second model; the first model is used to generate the third sub-deformation field based on the amplitude features of the amplitude image corresponding to the moving image and the amplitude image corresponding to the reference image output by the transformation module; the second model is used to generate the fourth sub-deformation field based on the phase features of the phase image corresponding to the moving image and the phase image corresponding to the reference image output by the transformation module. The second fusion module is used to fuse the first sub-deformation field, the second sub-deformation field, the third sub-deformation field, and the fourth sub-deformation field to obtain the initial sub-deformation field.

2. The method as described in claim 1, characterized in that, When the sub-model is not a first-level sub-model, the sub-model further includes a first fusion module. The output of the first fusion module is connected to the input of the upsampling module. The input of the first fusion module is connected to the output of the deformation field generation module and the output of the upsampling module of the previous-level sub-model. The first fusion module is used to fuse the initial sub-deformation field and the sub-deformation field output by the upsampling module of the previous level sub-model to obtain a fused sub-deformation field; The upsampling module is used to upsample the fused sub-deformation field to obtain the sub-deformation field.

3. The method as described in claim 2, characterized in that, The output of the first fusion module is also connected to the input of the pre-registration module; wherein, The pre-registration module is used to perform a second pre-registration on the moving image in the image to be registered or the moving image after the first pre-registration according to the fusion sub-deformation field, so as to obtain the moving image after the second pre-registration. The deformation field generation module is used to generate a target sub-deformation field based on the moving image after the second pre-registration and the reference image in the image to be registered; The upsampling module is used to upsample the target sub-deformation field to obtain the sub-deformation field.

4. The method as described in claim 1, characterized in that, The local feature module includes at least a partitioning module, a third model, and a third fusion module; the output of the partitioning module is connected to the input of the third model, and the output of the third model is connected to the input of the third fusion module; wherein, The partitioning module is used to determine the number of blocks based on the partitioning parameters and dimensions of the image to be registered; Used to segment the image to be registered into block images corresponding to the number of blocks; The third model is used to generate sub-deformation fields for each of the segmented images based on the local features of each segmented image. The third fusion module is used to fuse the various sub-deformation fields to generate the first sub-deformation field.

5. The method as described in claim 3, characterized in that, Before the step of inputting multiple images to be registered at different scales into a pre-trained deformation field generation model to generate the target deformation field corresponding to the moving image, the method further includes: Obtain the registration image to be trained, the registration image including the moving image and the reference image; Perform the step of performing at least one discrete wavelet transform on the image to be registered to obtain at least two images to be registered at different scales until the target deformation field of the moving image is obtained; Acquire the moving image, reference image, sub-deformation field, and the moving image after the second pre-registration of the sub-model at each level in the deformation field generation model; Based on the moving image, the reference image, the sub-deformation field, and the moving image after the second pre-registration, determine the loss function value of the sub-model at each level; Determine the sum of the loss function values ​​for each level of the sub-model; If the change value of the sum within a preset time period is greater than a preset range, the model parameters of the sub-model at each level are backpropagated, and the model parameters of the sub-model at each level are shared to update the model parameters of the sub-model at each level. Continue with the step of obtaining the registration image to be trained; If the change in the sum within a preset time period is less than or equal to a preset range, the training of the deformation field generation model ends.

6. A terminal device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image registration method as described in any one of claims 1 to 5.

7. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the image registration method as described in any one of claims 1 to 5.

Images