Image registration method and device, electronic equipment and storage medium

Abstract

The invention relates to an image registration method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first image and a second image obtained through image collection of a target physiological region, the first image data being a three-dimensional image, and the second image being a two-dimensional image; and determining a deformation field describing a mapping relation between the first image and the second image, and performing nonlinear deformation on the first image according to the deformation field to obtain a deformed image. And projecting the deformed image to obtain a two-dimensional third image capable of measuring a registration effect with the second image. According to the embodiment of the invention, through the architecture for image deformation and projection and image registration based on the deep learning automatic learning deformation field, the elastic registration between the images can be simply, quickly and accurately realized. Meanwhile, the image registration mode is realized by depending on a software algorithm without depending on the improvement of a hardware structure, so that the cost is low.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image registration method, apparatus, electronic device and storage medium. Background Technology

[0002] Computed tomographic angiography (CTA) can visualize arteries and veins throughout the body, from those serving the brain to those supplying blood to the lungs, kidneys, arms, and legs, and is commonly used for preoperative examinations. Digital subtraction angiography (DSA) is a diagnostic procedure used to examine the inner surface of blood vessels, also known as the lumen, and can be used to examine arteries, veins, and ventricles, widely used in interventional radiology. Registering CTA and DSA images can guide clinical treatment strategies, provide real-time intraoperative navigation, and facilitate postoperative follow-up. However, the registration techniques for CTA and DSA images suffer from low efficiency, high hardware costs, poor adaptability to complex scenarios, and high dependence on specialists. Therefore, a computationally efficient, robust, and hardware-independent image registration strategy is urgently needed to enhance its practical application value. Summary of the Invention

[0003] In view of this, this disclosure proposes an image registration method, apparatus, electronic device and storage medium, aiming to provide an image registration strategy that is computationally efficient, robust and has low hardware dependency.

[0004] According to a first aspect of this disclosure, an image registration method is provided, the method comprising:

[0005] A first image and a second image are obtained by image acquisition of the target physiological region, wherein the first image is a three-dimensional image and the second image is a two-dimensional image;

[0006] Determine the deformation fields corresponding to the first image and the second image, wherein the deformation fields are used to describe the mapping relationship between the first image and the second image;

[0007] The first image is nonlinearly deformed according to the deformation field to obtain a deformed image;

[0008] The deformed image is projected to obtain a third image registered with the second image, and the third image is a two-dimensional image.

[0009] In one possible implementation, acquiring the first and second images obtained by image acquisition of the target physiological region includes:

[0010] Acquire a first image and a sequence of candidate images obtained by image acquisition of the target physiological region, wherein the sequence of candidate images includes multiple two-dimensional candidate images in temporal order;

[0011] The candidate image sequence is subjected to dynamic frame fusion to obtain the second image.

[0012] In one possible implementation, the step of performing dynamic frame fusion on the candidate image sequence to obtain the second image includes:

[0013] The first frame image in the candidate image sequence is determined as the target candidate image, and the images other than the target candidate image are other candidate images;

[0014] Calculate the difference between each of the other candidate images in the candidate image sequence and the target candidate image to obtain the corresponding silhouette image;

[0015] The second image is obtained by averaging each of the silhouette images.

[0016] In one possible implementation, determining the deformation fields corresponding to the first image and the second image includes:

[0017] The first image feature is extracted from the first image by the first image encoder. The first image feature is a three-dimensional image feature.

[0018] The second image features are extracted from the second image using a second image encoder. The second image features are two-dimensional image features.

[0019] The second image features are extended to obtain a three-dimensional third image feature;

[0020] The fourth image feature is obtained by stitching together the first image feature and the third image feature;

[0021] The fourth image features are encoded and decoded using a codec to obtain the deformation fields corresponding to the first and second images.

[0022] In one possible implementation, the method further includes:

[0023] A training set is determined, comprising multiple groups of sample images, wherein the groups of sample images include a first sample image and a second sample image.

[0024] The first image encoder, the second image encoder, and the codec are trained based on the training set.

[0025] In one possible implementation, training the first image encoder, the second image encoder, and the codec based on the training set includes:

[0026] By inputting the first sample image and the second sample image in each sample image group into the first image encoder and the second image encoder respectively, the corresponding predicted deformation field is determined;

[0027] The first sample image is subjected to nonlinear deformation based on the predicted deformation field to obtain a predicted deformation image;

[0028] The predicted deformed image is projected to obtain a predicted third image registered with the sample second image;

[0029] The prediction loss is determined based on the predicted third image and the sample first image;

[0030] The parameters of the first image encoder, the second image encoder, and the encoder / decoder are adjusted based on the predicted loss.

[0031] In one possible implementation, determining the prediction loss based on the predicted third image and the sample first image includes:

[0032] Determine at least one of the following losses between the predicted third image and the sample first image: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and key point registration loss;

[0033] The prediction loss is obtained by calculating the weighted sum of at least one of the following losses: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and keypoint registration loss.

[0034] According to a second aspect of this disclosure, an image registration apparatus is provided, the apparatus comprising:

[0035] The image acquisition module is used to acquire a first image and a second image obtained by image acquisition of the target physiological region, wherein the first image is a three-dimensional image and the second image is a two-dimensional image;

[0036] A deformation field determination module is used to determine the deformation fields corresponding to the first image and the second image, wherein the deformation fields are used to describe the mapping relationship between the first image and the second image;

[0037] A deformation module is used to perform nonlinear deformation on the first image according to the deformation field to obtain a deformed image;

[0038] The projection module is used to project the deformed image to obtain a third image registered with the second image, wherein the third image is a two-dimensional image.

[0039] In one possible implementation, the image acquisition module is further configured to:

[0040] Acquire a first image and a sequence of candidate images obtained by image acquisition of the target physiological region, wherein the sequence of candidate images includes multiple two-dimensional candidate images in temporal order;

[0041] The candidate image sequence is subjected to dynamic frame fusion to obtain the second image.

[0042] In one possible implementation, the image acquisition module is further configured to:

[0043] The first frame image in the candidate image sequence is determined as the target candidate image, and the images other than the target candidate image are other candidate images;

[0044] Calculate the difference between each of the other candidate images in the candidate image sequence and the target candidate image to obtain the corresponding silhouette image;

[0045] The second image is obtained by averaging each of the silhouette images.

[0046] In one possible implementation, the deformation field determining module is further configured to:

[0047] The first image feature is extracted from the first image by the first image encoder. The first image feature is a three-dimensional image feature.

[0048] The second image features are extracted from the second image using a second image encoder. The second image features are two-dimensional image features.

[0049] The second image features are extended to obtain a three-dimensional third image feature;

[0050] The fourth image feature is obtained by stitching together the first image feature and the third image feature;

[0051] The fourth image features are encoded and decoded using a codec to obtain the deformation fields corresponding to the first and second images.

[0052] In one possible implementation, the device further includes:

[0053] A training set determination module is used to determine a training set comprising multiple groups of sample images, wherein the groups of sample images include a first sample image and a second sample image.

[0054] The model training module is used to train the first image encoder, the second image encoder, and the codec based on the training set.

[0055] In one possible implementation, the model training module is further configured to:

[0056] The corresponding predicted deformation field is determined by inputting the first sample image and the second sample image in each sample image group into the first image encoder and the second image encoder, respectively.

[0057] The first sample image is subjected to nonlinear deformation based on the predicted deformation field to obtain a predicted deformation image;

[0058] The predicted deformed image is projected to obtain a predicted third image registered with the sample second image;

[0059] The prediction loss is determined based on the predicted third image and the sample first image;

[0060] The parameters of the first image encoder, the second image encoder, and the encoder / decoder are adjusted based on the predicted loss.

[0061] In one possible implementation, the model training module is further configured to:

[0062] Determine at least one of the following losses between the predicted third image and the sample first image: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and key point registration loss;

[0063] The prediction loss is obtained by calculating the weighted sum of at least one of the following losses: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and keypoint registration loss.

[0064] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing instructions stored in the memory.

[0065] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0066] According to a fifth aspect of this disclosure, a computer program product is provided, including computer-readable code or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0067] In this embodiment, the method acquires a first image and a second image obtained by image acquisition of a target physiological region. The first image is a three-dimensional image, and the second image is a two-dimensional image. A deformation field describing the mapping relationship between the first and second images is determined. The first image is nonlinearly deformed according to the deformation field to obtain a deformed image. The deformed image is projected to obtain a two-dimensional third image that can be used to measure the registration effect with the second image. This embodiment adopts an architecture for image registration based on deep learning to automatically learn the deformation field for image deformation and projection, which can achieve elastic registration between images simply, quickly, and accurately. Furthermore, this image registration method relies on software algorithms and does not require hardware structure improvements, resulting in low cost.

[0068] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0069] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0070] Figure 1 A schematic diagram showing a first image and a second image according to an embodiment of the present disclosure;

[0071] Figure 2 A flowchart illustrating an image registration method according to an embodiment of the present disclosure is shown;

[0072] Figure 3 A schematic diagram illustrating an image registration process according to an embodiment of the present disclosure is shown;

[0073] Figure 4 A schematic diagram of the architecture of an image registration algorithm according to an embodiment of the present disclosure is shown;

[0074] Figure 5 A schematic diagram illustrating an image registration network training process according to an embodiment of the present disclosure is shown.

[0075] Figure 6 A schematic diagram showing an image registration result according to an embodiment of the present disclosure;

[0076] Figure 7 A schematic diagram of an image registration apparatus according to an embodiment of the present disclosure is shown;

[0077] Figure 8 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0078] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0079] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0080] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0081] The image registration method of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal, such as a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the image registration method of this disclosure by having its processor call computer-readable instructions stored in its memory.

[0082] In this embodiment, the image registration method can be applied to the medical field, that is, to register different medical images in any scenario such as image diagnosis, surgical navigation, and radiotherapy. Alternatively, it can also be applied to any field such as remote sensing, computer vision, and industry to register different images in any scenario within the corresponding field.

[0083] Figure 1 A schematic diagram showing a first image and a second image according to an embodiment of the present disclosure is provided. Figure 1 As shown, in some embodiments, the embodiments of this application can be applied to register medical images, wherein the left image is a computed tomographic angiography (CTA) as the first image, and the right image is a digital subtraction angiography (DSA) as the second image.

[0084] In other words, when the image registration method of this application is applied to the medical field, it can be used to register images from both computed tomographic angiography (CTA) and digital subtraction angiography (DSA). CTA is a three-dimensional medical imaging technique that displays arteries and veins throughout the body, ranging from arteries serving the brain to blood-carrying arteries leading to the lungs, kidneys, arms, and legs; it is commonly used for preoperative examinations. DSA is a two-dimensional medical imaging technique used to visualize the inner surface of blood vessels, also known as a diagnostic procedure for assessing the lumen; it can be used to examine arteries, veins, and ventricles and is widely used in interventional radiology. Registering these two techniques can be used to guide clinical treatment strategies, provide real-time intraoperative navigation, and facilitate postoperative follow-up.

[0085] Currently, in the field of medical imaging, clinical practice has applied the 3D image fusion technology of magnetic resonance angiography (MRA) and digital subtraction angiography (DSA) to four endovascular interventional procedures, demonstrating that this real-time 3D navigation system can accurately guide the catheter manipulation path. Therefore, when using CTA-DSA cross-dimensional registration for real-time navigation during interventional procedures, it can accurately provide real-time path maps to guide the completion of the procedure, thereby reducing the use of contrast agents and radiation dose, ultimately achieving the goal of reducing surgical risks and improving surgical safety. The iterative development of 3D-2D image registration technology is driving vascular imaging technology towards a "double reduction and double increase" direction: through intelligent algorithm optimization, it is expected to achieve a radiation dose reduction of more than 50% and a contrast agent dosage reduction of 60% in the future, while simultaneously improving image resolution and navigation accuracy. This technological innovation not only reduces the risk of nephrotoxicity for patients undergoing angiography but also opens up new pathways for minimally invasive treatment of complex vascular lesions.

[0086] Image registration techniques primarily include feature-point-based, image intensity-based, and deep learning-based methods. Feature-point-based methods extract key feature points from 3D models and 2D images, such as SIFT, SURF, and ORB, to establish geometric correspondences and optimize spatial transformation parameters. They are commonly used for bone registration in orthopedic surgical navigation. Representative products include commercial software like ITK-SNAP and the registration module integrated into 3D Slicer, as well as hardware devices such as registration devices for optical positioning systems and CT / MRI images. Image intensity-based methods utilize image grayscale or gradient information and optimize similarity metrics, such as mutual information and normalized cross-correlation, to solve for elastic deformation fields. They are commonly used for patient positioning error correction in radiotherapy and real-time tracking of dynamic organs such as the lungs. Representative algorithms include the Demons algorithm and the B-spline free deformation model. Deep learning-based registration methods use convolutional neural networks (CNNs) or generative adversarial networks (GANs) to directly learn registration transformation parameters from image data. Representative studies include end-to-end registration frameworks such as VoxelMorph and DLIR.

[0087] The aforementioned methods suffer from various problems at both the algorithmic and hardware levels when performing image registration. For example, at the algorithmic level, feature-based registration methods are prone to failure in scenarios with missing structures, noise interference, or low contrast, such as soft tissue registration. Their high computational complexity, coupled with the need for traditional iterative optimization methods like the Demons algorithm to calculate similarity metrics multiple times, makes them time-consuming and unsuitable for real-time requirements such as intraoperative navigation. Intensity-based registration methods are sensitive to changes in parameters such as grayscale distribution, including image differences after contrast agent injection, and parameter tuning is highly dependent on experience. At the hardware level, existing systems often rely on external markers or high-resolution imaging equipment such as laser scanners, resulting in high hardware costs and a high demand for high-precision equipment. Multimodal registration requires the integration of multiple sensors, such as optical + electromagnetic positioning, which presents challenges in calibration and susceptibility to environmental interference. The high system complexity makes training difficult and necessitates a large number of specialized professionals.

[0088] Therefore, the technical problem that this application embodiment needs to solve is how to provide a flexible registration method that is computationally efficient, robust, and has low hardware dependency.

[0089] To address the aforementioned technical issues, this application provides an architecture for multimodal cross-dimensional image registration based on deep learning to automatically learn deformation fields for image deformation and projection. This architecture enables simple, fast, and accurate registration of multimedia cross-dimensional images. Furthermore, this image registration method relies on software algorithms, requiring no hardware structural improvements and thus is cost-effective.

[0090] Figure 1A flowchart illustrating an image registration method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the image registration method of this disclosure embodiment may include the following steps S10-S40.

[0091] The following description uses electronic devices as the executing entity. It can be understood that the electronic device is not limited to the electronic device itself, but also includes modules such as processors or processing chips in the electronic device that can perform computer tasks.

[0092] Step S10: The electronic device acquires the first and second images obtained by image acquisition of the target physiological region.

[0093] In one possible implementation, embodiments of this application acquire a first image and a second image that need to be registered via an electronic device. The first image and the second image can be different types of medical images obtained by acquiring the same target physiological region using different types of image acquisition. The target physiological region can be any region of a human body or part of an animal body, such as the head, limbs, and organs such as the heart and lungs.

[0094] Optionally, the first image can be a computed tomographic angiography (CTA) image obtained from medical imaging of veins or arteries in the target physiological region; this medical image is a three-dimensional image. The second image can be a digital subtraction angiography (DSA) image used to view the inner surface of blood vessels; this medical image is a two-dimensional image. The first and second images can be acquired by different medical imaging devices and then transmitted to an electronic device, or at least one of them can be directly acquired by an image acquisition device connected to the electronic device.

[0095] In some embodiments, when the first image is a CTA image, it can be directly acquired by medical imaging equipment, and then the vascular region can be segmented. When the second image is a DSA image, the medical imaging equipment acquires multiple images in temporal order during image acquisition, and the electronic device can determine the second image based on the acquired multiple images. For example, the electronic device can first acquire a first image obtained by image acquisition of the target physiological region and a sequence of candidate images, the candidate image sequence including multiple two-dimensional candidate images in temporal order. Then, the second image is obtained by dynamic frame fusion of the candidate image sequence.

[0096] For example, in this embodiment of the application, the process of the electronic device performing dynamic frame fusion on a candidate image sequence to obtain a second image can be as follows: First, determine the first frame image in the candidate image sequence as the target candidate image, and the images other than the target candidate image as other candidate images. Calculate the difference between each other candidate image and the target candidate image in the candidate image sequence to obtain the corresponding silhouette image. Average each silhouette image to obtain the second image.

[0097] In other words, this dynamic frame fusion method can involve performing digital subtraction on the candidate image sequence to subtract the first frame from each of the subsequent frames and remove the first frame. Then, all the subtracted frames are superimposed and averaged to obtain the synthesized second image. Optionally, embodiments of this application can further normalize the synthesized second image, mapping the pixels at each pixel location to the range of 0-255, and further reduce image noise through smoothing processes such as mean filtering.

[0098] Alternatively, embodiments of this application may perform any selectable image optimization processing, such as image filtering or image enhancement, on the first and second images to further improve the quality of the first and second images.

[0099] In some possible implementations, after determining the first image and the second image, the electronic device of this application embodiment can directly perform image registration based on the first image and the second image. Alternatively, it can segment the first image and the second image to obtain at least one set of three-dimensional image blocks and two-dimensional image blocks of the same size that represent the same region information, and further perform image registration based on the segmented image blocks. In the case of image registration through the segmented image blocks, the electronic device can stitch together the registration results of each set of image blocks to obtain the final registration result.

[0100] Step S20: The electronic device determines the deformation field corresponding to the first image and the second image.

[0101] In one possible implementation, after acquiring the first image and the second image, the electronic device can determine the deformation field corresponding to the first image and the second image. This deformation field, used to describe the mapping relationship between the first image and the second image, can be implemented using any existing image processing algorithm. For example, if the first image and the second image are images of the same dimension, the deformation field can be determined by calculating the global optical flow of the first image and the second image.

[0102] In some embodiments, the electronic device can determine the deformation fields of the first image and the second image based on a neural network model. For example, a neural network model for extracting the deformation fields of the first image and the second image can be pre-trained so that the corresponding deformation fields can be directly output by inputting the first image and the second image into the trained neural network model.

[0103] Optionally, the neural network model trained in this embodiment may include a first image encoder and a second image encoder. The neural network model extracts the deformation fields of the first and second images by: extracting first image features (three-dimensional features) from the first image using the first image encoder; extracting second image features (two-dimensional features) from the second image using the second image encoder; expanding the second image features to obtain three-dimensional third image features; and encoding and decoding the fourth image features obtained by concatenating the first and third image features to obtain the deformation fields corresponding to the first and second images.

[0104] In some embodiments, the first image encoder can be a 3D convolutional encoder (Conv3D), including downsampling layers, dense blocks, and transition layers, for extracting three-dimensional image features from the input first image to obtain first image features. The second encoder can be DenseNet-BC, including initial convolution, max pooling, and cascaded dense blocks, for the user to extract two-dimensional image features from the input second image, and outputting multi-scale second image features.

[0105] For example, the first image encoder may include a 3D convolutional layer with a kernel size of 3x3x3, a stride of (2,2,2), edge padding of 1 (padding1), followed by a batch normalization layer (BatchNorm3d) and an activation function layer (LeakyReLU). The function of this first image encoder is to downsample the input 3D first image, reduce the spatial dimension, and extract low-level encoded features as the first image features.

[0106] Optionally, in this embodiment, the first image encoder can be pre-trained. This pre-training can be achieved by upsampling the low-level encoded features extracted from the input first image using transposed 3D convolution to restore their size. The original first image volume data is reconstructed from the restored encoded features to obtain the reconstructed image corresponding to the first image. The voxel-level mean square error between the reconstructed image and the first image is calculated as the loss of the first image encoder, and the first image encoder is adjusted accordingly.

[0107] For example, the parameter settings for the autoencoder pre-trained on the first image in this embodiment can be as shown in the table below:

[0108]

[0109] For example, the second image encoder can be a variant based on the DenseNet-BC architecture, primarily processing two-dimensional images. Its structure includes: an initial convolutional layer: a 7x7 convolution with a stride of 1 and padding of 3, followed by a batch normalization layer (BatchNorm) and an activation function layer (LeakyReLU), then downsampling via a max-pooling layer. Multiple densely connected blocks (DenseBlocks) and transition layers: each DenseBlock contains multiple DenseLayers, each consisting of two convolutional layers (1x1 and 3x3) using dense connections. The transition layer includes 1x1 convolutions and average pooling for feature compression and dimensionality reduction. Finally, there is the BatchNorm and activation layer. The input is a two-dimensional second image, and the output is a high-dimensional feature map after passing through multiple dense blocks and transition layers, yielding the second image features.

[0110] The decoder consists of transposed convolutional layers that upsample the encoded features back to the original image size. The original DSA image is reconstructed from the encoded features, with the loss function being the pixel-level mean square error.

[0111] Optionally, in this embodiment, the second image encoder can be pre-trained. This pre-training of the second image decoder can be achieved by using a decoder composed of transposed convolutional layers to upsample the features extracted from the input second image to restore the original image size. The original second image is reconstructed from the encoded features of the restored size to obtain the reconstructed image corresponding to the second image. The pixel-level mean square error between the reconstructed image and the second image is calculated as the loss of the second image encoder, and the second image encoder is adjusted accordingly.

[0112] For example, the parameter settings for the second image pre-trained autoencoder in this application embodiment can be as shown in the following table:

[0113]

[0114] The training objective of the pre-training process for the first and second image encoders described above is to reconstruct the input image. Therefore, the first and second image encoders can learn effective feature representations through the pre-training process. Furthermore, in the embodiments of this application, the subsequent registration task can further fine-tune the parameters of the pre-trained first and second image encoders.

[0115] In some embodiments, after the electronic device extracts features from the first image and the second image using a first image encoder and a second image encoder respectively, it obtains first image features and second image features. Since the second image features are two-dimensional image features and the first image features are three-dimensional image features, the second image features can be further extended axially using the unsqueeze(1) function to extend the second image features into pseudo-three-dimensional image features as third image features.

[0116] Optionally, after determining the third 3D image features, the first and third 3D image features can be concatenated along the channel dimension to obtain the fourth image features. Then, by further encoding and decoding the fourth image features, the deformation fields corresponding to the first and second images are obtained. This encoding and decoding process can be implemented using an improved U-net network, where the fourth image features are input into a U-net-like network, and the network outputs a 3-channel deformation field (3D displacement vector field) through an encoder-decoder structure with skip connections. The encoder is used to capture the global context, and the decoder is used to recover spatial details.

[0117] Step S30: The electronic device performs nonlinear deformation on the first image according to the deformation field to obtain a deformation image.

[0118] In one possible implementation, after determining the deformation fields corresponding to the first and second images, the electronic device can perform nonlinear deformation on the first image based on the deformation fields to obtain a deformed image. This nonlinear deformation process can be implemented using Spatial Transformer Networks (STNs). That is, the electronic device encapsulating the STN software module can, after determining the deformation field, input the deformation field and the first image into the Spatial Transformer Network (STN) to apply the deformation field to the original first image for nonlinear deformation, thereby obtaining the deformed image.

[0119] Step S40: The electronic device projects the deformed image to obtain a third image registered with the second image.

[0120] In one possible implementation, after the electronic device obtains a deformed image by performing nonlinear deformation on the first image, it can project the deformed image to obtain a third image registered with the second image. The third image is a two-dimensional image. Optionally, in this embodiment, the electronic device can project the three-dimensional deformed image using any projection method to obtain a two-dimensional third image. For example, the projection method can include parallel projection methods such as orthographic projection and oblique projection, or perspective projection methods such as parallel perspective, angular perspective, and oblique perspective. Specifically, a suitable projection parameter model needs to be selected based on the projection transformation relationship between the second image and the first image.

[0121] In some embodiments, the electronic device of this application may also deploy a Digital Reconstruction Projection Module (DRR), a technology for medical image processing, primarily used to generate two-dimensional radiographic images (such as DSA X-ray image data) from three-dimensional images (such as CT scan data). In embodiments of this application, the DRR module can be used to simulate the imaging process of a second image, projecting the input deformed head image into a two-dimensional third image.

[0122] In some embodiments, the digital reconstruction projection module of this application can simulate the principle of X-ray imaging, integrate a differentiable ray projection algorithm, support end-to-end projection optimization, and project a three-dimensional deformed image into a two-dimensional third image for registration and alignment with the second image. The core process of this projection method includes: geometric parameter calculation → ray direction modeling → voxel accumulation projection → image normalization.

[0123] Specifically, the digital reconstruction projection module transforms the input 3D deformed image into a 2D third image by simulating the physical process of X-ray imaging. First, the physical coordinates of the input deformed image are converted to voxel space based on the voxel spacing. Then, the precise position of the X-ray source in the voxel coordinate system is calculated using rotation and translation parameters based on geometric transformations. Next, based on the source-image distance and source-object distance, a grid coordinate system matching the size of the second image is generated on the projection plane. Through 3D rotation and translation operations, each pixel on the projection plane is mapped to a 3D space aligned with the volume data. Then, the digital reconstruction projection module performs discretization sampling along the ray direction from the X-ray source to each pixel: by parameterizing the ray equation, the coordinates of the ray passing through the volume data are calculated layer by layer. Nearest neighbor interpolation is used to obtain the voxel intensity value of the current layer, and the intensity of all sampled points is accumulated and integrated to form an initial DRR image as a candidate image. During this process, coordinates exceeding the volume data boundary are mapped to a preset filling layer to ensure computational robustness. Finally, the module normalizes the intensity of the selected image, linearly scaling it to the [0,1] range to generate a synthetic projection image geometrically aligned with the second image, which serves as the third image. This projection process integrates 3D geometric transformation, ray tracing, and physical scale mapping, achieving end-to-end projection modeling through fully differentiable operations, providing a high-fidelity synthetic image foundation for multimodal medical image registration.

[0124] Figure 3 This diagram illustrates an image registration process according to an embodiment of the present disclosure, such as... Figure 3 As shown, the entire registration process in this embodiment is as follows: First, the first image and the second image to be registered are determined by an electronic device. Then, the three-dimensional first image features of the first image are extracted by a first image encoder, and the two-dimensional second image features of the second image are extracted by a second image encoder. The second image features are further extended to obtain three-dimensional third image features. Then, the fourth image features are obtained by stitching the first and third image features together, and the fourth image features are encoded and decoded to obtain the deformation fields corresponding to the first and second images. The first image is then subjected to nonlinear deformation based on the deformation fields to obtain a deformed image. Finally, the deformed image is projected to obtain a two-dimensional third image registered with the second image.

[0125] The first image encoder and the second image encoder can both obtain their initial parameters through pre-training of the autoencoder reconstruction task.

[0126] For example, embodiments of this application may pre-establish pre-training tasks for a first image encoder and a second image encoder, which are used to learn low-dimensional representations of the first and second images through an autoencoder network, providing a robust feature foundation for the subsequent multimodal image registration process.

[0127] Optionally, the preprocessing tasks of the first and second image encoders take the preprocessed first and second images as inputs, respectively, and use a symmetric encoding / decoding structure to compress and reconstruct the input images. The encoder progressively extracts spatial features and reduces dimensionality through multi-layer convolution, while the decoder inversely restores the original dimensions, ultimately outputting a reconstructed result with the same dimensions as the input. The network for this task is trained using the PyTorch framework on an NVIDIA GeForce RTX 3090 GPU for 1500 epochs, with a batch size of 8 and a learning rate of 1e-3. Mean squared error is used as the reconstruction loss function during training, and the network parameters are iteratively optimized using an SGD optimizer. Reconstruction quality is evaluated on the validation set every 50 epochs. This pre-training task can capture the anatomical feature distribution of CTA data and the spatial distribution characteristics of vascular structures in DSA images through unsupervised learning, providing highly representative initial network parameters for downstream registration tasks.

[0128] Figure 4 A schematic diagram of the architecture of an image registration algorithm according to an embodiment of the present disclosure is shown. Figure 4 As shown, the image registration algorithm of this application embodiment can be integrated into an image registration network, which may include a registration module, a spatial transformation module, and a projection module. When registering a first image and a second image, the electronic device can directly input the first and second images into the image registration network to output a registered third image. The registration module includes a first image encoder and a second image encoder, used to determine the deformation fields corresponding to the input first and second images. The spatial transformation module is used to perform nonlinear deformation on the first image based on the deformation field to obtain a deformed image. The projection module is used to project the three-dimensional deformed image to obtain a two-dimensional third image.

[0129] In some embodiments, the registration module extracts first image features from the first image using a first image encoder and second image features from the second image using a second image encoder. The first image features and the second image features, extended to obtain three-dimensional image features, are then fused by stitching together to obtain fourth image features. These fourth image features are then encoded and decoded using a codec to obtain the deformation fields corresponding to the first and second images.

[0130] For example, the codec in the registration module of this application embodiment is similar to that of the traditional U-Net, having multiple processing structures including convolutional layers and batch normalization processing layers, an encoding structure for performing convolutional downsampling operations in the encoder, a decoding structure for upsampling operations in the decoder, and a skip connection structure across each level.

[0131] Optionally, Figure 4 The codec in the codec can include multiple encoder layers (Enc1-Enc... N ) and multiple decoder layers (Dec1-Dec N The detailed structure can be as follows: An initial downsampling layer is a 3x3x3 3D ​​convolution with padding1, followed by BatchNorm3D and LeakyReLU to extract shallow features. This shallow feature extraction process does not reduce spatial resolution, and the output size is the same as the input size. After the initial downsampling layer, multiple ordinary downsampling layers are connected: each downsampling layer contains a 3x3x3 3D ​​convolution with stride 2, padding1, followed by BatchNorm3D and LeakyReLU. This ordinary downsampling layer halves the size of the input feature map through a convolution with stride 2, while increasing the number of channels. The upsampling layer structure uses ConvTranspose3D(3x3x3, stride=2, padding=1, output_padding=1) followed by BatchNorm3D and LeakyReLU to achieve a 2x upsampling using transposed convolution. The upsampling layer is concatenated with the corresponding encoder feature map, i.e., a skip connection. The post-processing module structure is used to adjust the number of channels and fuse features, further upsample through transposed convolution, and then generate the final output.

[0132] In some possible implementations, the image registration network can be pre-trained with two separate feature encoders, or it can be trained as a whole. Alternatively, the two encoders can be pre-trained first, and then the parameters of the module can be tuned as a whole.

[0133] For example, when an electronic device needs to further fine-tune the parameters of the first and second image encoders in an image registration network, it can first determine a training set comprising multiple sample image groups, including a first sample image and a second sample image. The first and second image encoders are then trained based on the training set. The first and second sample images can be images that have undergone image preprocessing. This preprocessing process may include normalizing and resizing the original image data, and generating a series of pre-training simulation data to increase the number of training samples. The resizing process can uniformly set the size of the first sample image to 256×256×256 with an isotropic resampling voxel interval of [2.0, 1.0, 1.0], and uniformly set the size of the second sample image to 512×512 with an isotropic resampling voxel interval of [0.75, 0.75]. Then, these larger image blocks are cropped from the center to generate image blocks of a preset size.

[0134] While acquiring the training set, the electronic device can also acquire a validation set and a test set with the same content style as the training set in the same way, for use in validation and testing after training the image registration network.

[0135] In some possible implementations, after acquiring the training set, the electronic device can determine the corresponding predicted deformation field by inputting the first and second sample images from each sample image group into a first image encoder and a second image encoder, respectively. Then, it performs nonlinear deformation on the first sample image based on the predicted deformation field to obtain a predicted deformed image. The predicted deformed image is then projected to obtain a predicted third image registered with the second sample image. The prediction loss is determined based on the predicted third image and the first sample image. Finally, the parameters of the first and second image encoders are adjusted based on the prediction loss.

[0136] Optionally, the training process of this image registration network can be implemented on the PyTorch framework and trained for 1000 epochs on an NVIDIA GeForce RTX 3090 GPU with a batch size of 1 and a learning rate of 1e-3 for the entire image registration network. The parameters of the deformation field prediction network are updated through backpropagation, and iterative optimization is performed until convergence. The SGD optimizer is used to update the network parameters, and the quality of the validation set is evaluated at each epoch.

[0137] In some embodiments, the prediction loss determined in this application may include a combined loss comprising multiple types of losses. For example, the mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and keypoint registration loss between the predicted third image and the sample first image may be determined first. Then, a weighted sum of the mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and keypoint registration loss is calculated to obtain the prediction loss, which is used to adjust the parameters of the first image encoder, the second image encoder, and the encoder / decoder of the deformation field prediction network based on the prediction loss.

[0138] In some embodiments, mutual information loss is used to maximize the statistical dependency between two images, enabling multimodal registration. This mutual information loss (L... MI It can be calculated using the following formula:

[0139] MI = H(I) + H(J) - H(I,J)

[0140] L MI =1-MI

[0141] Where I and J are the predicted third image and the sample first image, respectively, H is the entropy, which can be calculated by histogram and Gaussian approximation, and MI is the mutual information.

[0142] In some embodiments, gradient smoothing loss is used to constrain the spatial smoothness of the deformable field and avoid abrupt changes. This gradient smoothing loss (L...) smooth It can be calculated using the following formula:

[0143]

[0144] Where E[·] represents the expectation, and φ is the deformation field. Let x, y, z represent the spatial gradients of the deformation field in the x, y, z directions, respectively, and ‖·‖ represent the Euclidean distance.

[0145] In some embodiments, the deformation direction consistency loss is used to maintain the directional consistency of adjacent deformation vectors. This deformation direction consistency loss (L...) orient It can be calculated using the following formula:

[0146]

[0147] Where E[·] represents the expectation, θ represents the angle between the deformation vectors of adjacent voxels in the first sample image and the predicted deformation image in the x, y, and z directions, respectively, and cosθ reflects the consistency of the two vector directions (θ ranges from 0° to 360°; when θ equals 0° or 360°, cosθ = 1, indicating that the vectors in the two directions are completely consistent; when θ equals 180°, cosθ = -1, indicating that the vectors in the two directions are completely opposite).

[0148] In some embodiments, the keypoint registration loss is used to assess registration accuracy through the matching error of keypoints. In the case of registering medical images, keypoints can be vascular branch points. The calculation steps for this keypoint registration loss may include: extraction: extracting the vascular centerline through skeletonization → point finding: finding keypoints such as inflection points and bifurcation points → matching: finding the nearest neighbor for each point → filtering: removing the quantile with the largest error (e.g., retaining 80%) → calculation: calculating the average Euclidean distance of the retained points. This keypoint registration loss (L...) point It can be calculated using the following formula:

[0149]

[0150] Among them, pt1 i pt2 represents the i-th point in the first set of points (e.g., key points in the second sample image); i represents the i-th corresponding point in the second set of points (such as the key points in the predicted third image obtained after registration and projection); ‖·‖2 represents the Euclidean distance (i.e., L2 norm), the straight-line distance between two points; N represents the number of key points.

[0151] In some embodiments, volume conservation loss is used to constrain volume changes before and after deformation, preventing unreasonable shrinkage or expansion. This volume conservation loss (L...) volume It can be calculated using the following formula:

[0152]

[0153] Among them, V new V is the volume of the predicted deformed image after deformation. raw This represents the volume of the first image of the sample before deformation.

[0154] In some possible implementations, the predicted loss can be calculated by weighting and summing at least one of the aforementioned losses, without limitation. For example, when the predicted loss is determined by weighting and summing all the aforementioned losses, it can be obtained through the total loss function L. total =λ0L sim +λ1L smooth +λ2L orient +λ3L volume +λ4L point The prediction loss is calculated to balance similarity, smoothness, orientation consistency, and anatomical alignment. Where L... sim For the aforementioned mutual information loss L MI λ0, λ1, λ2, λ3, and λ4 are weights.

[0155] Figure 5 A schematic diagram illustrating an image registration network training process according to an embodiment of the present disclosure is shown. Figure 5 As shown, during the training of the image registration network, the electronic device can input the first and second sample images from each sample image group into the first and second image encoders, respectively. The first image encoder extracts the first image features of the first image, and the second image encoder extracts the second image features of the second image. Feature expansion is performed on the second image features to obtain three-dimensional third image features. The fourth image features, obtained by concatenating the first and third image features, are encoded and decoded by an encoder-decoder to determine the corresponding predicted deformation field. Then, the first sample image is nonlinearly deformed according to the predicted deformation field to obtain a predicted deformed image. The predicted deformed image is projected to obtain a predicted third image registered with the second sample image. The prediction loss is determined based on the predicted third image and the first sample image. The parameters of at least one network module in the image registration network, including the first image encoder, the second image encoder, and the encoder-decoder that processes the fourth image features, are then adjusted based on the prediction loss.

[0156] Figure 6 A schematic diagram illustrating an image registration result according to an embodiment of the present disclosure is shown. Figure 6 As shown, the upper left image is a two-dimensional image obtained by projecting the first image before registration, and the upper right image is a third image after image registration based on the embodiment of this application. It can be intuitively seen that the third image obtained after registration is more accurate. The lower image shows the registered third image and the second image, as well as a checkerboard image of the third image and the fixed image of the second image. Based on this image, it can be seen that the embodiment of this application has achieved excellent registration results.

[0157] In some embodiments, the relevant parameters determined by the first image and the second image before image registration, and the relevant parameters determined by the third image after registration and the second image can be compared as shown in the following table:

[0158]

[0159] As can be seen from the table above, the operation time before and after registration in this application embodiment is shortened from tens of minutes to less than 1 second, and the registration accuracy is also guaranteed.

[0160] Based on the aforementioned technical features, this application embodiment enables image registration through an architecture based on image deformation and projection using a deformation field. This architecture allows for simple, fast, and accurate image registration with a highly automated process. The entire image registration process, from data input to result output, requires no manual intervention (traditional methods require manual annotation of 5-7 key points), reducing operation time from 30 minutes to within 5 seconds while maintaining registration accuracy. Therefore, it eliminates the need for professional medical personnel, removing the bottleneck of manual intervention and reducing the consumption of medical resources. Furthermore, this image registration method relies on software algorithms, requiring no hardware structural improvements, resulting in low cost.

[0161] Furthermore, the image registration method of this application embodiment is highly adaptable to complex scenarios and is applicable to scenarios such as coronary artery disease, abdominal aortic aneurysm, and peripheral vascular intervention in the field of medical image registration. It is also compatible with CTA / DSA equipment from mainstream manufacturers such as Siemens and Philips, breaking the closed ecosystem limitations of traditional solutions. In addition, the core algorithm can be packaged into an independent SDK (interface response time <10ms), seamlessly integrating with existing PACS systems or surgical navigation platforms.

[0162] Figure 7 A schematic diagram of an image registration apparatus according to an embodiment of the present disclosure is shown. Figure 7 As shown, an image registration apparatus according to an embodiment of this application may include:

[0163] The image acquisition module 70 is used to acquire a first image and a second image obtained by image acquisition of the target physiological region, wherein the first image is a three-dimensional image and the second image is a two-dimensional image;

[0164] The deformation field determination module 71 is used to determine the deformation fields corresponding to the first image and the second image, wherein the deformation fields are used to describe the mapping relationship between the first image and the second image;

[0165] Deformation module 72 is used to perform nonlinear deformation on the first image according to the deformation field to obtain a deformed image;

[0166] The projection module 73 is used to project the deformed image to obtain a third image registered with the second image, wherein the third image is a two-dimensional image.

[0167] In one possible implementation, the image acquisition module 70 is further configured to:

[0168] Acquire a first image and a sequence of candidate images obtained by image acquisition of the target physiological region, wherein the sequence of candidate images includes multiple two-dimensional candidate images in temporal order;

[0169] The candidate image sequence is subjected to dynamic frame fusion to obtain the second image.

[0170] In one possible implementation, the image acquisition module 70 is further configured to:

[0171] The first frame image in the candidate image sequence is determined as the target candidate image, and the images other than the target candidate image are other candidate images;

[0172] Calculate the difference between each of the other candidate images in the candidate image sequence and the target candidate image to obtain the corresponding silhouette image;

[0173] The second image is obtained by averaging each of the silhouette images.

[0174] In one possible implementation, the deformation field determination module 71 is further configured to:

[0175] The first image feature is extracted from the first image by the first image encoder. The first image feature is a three-dimensional image feature.

[0176] The second image features are extracted from the second image using a second image encoder. The second image features are two-dimensional image features.

[0177] The second image features are extended to obtain a three-dimensional third image feature;

[0178] The fourth image feature is obtained by stitching together the first image feature and the third image feature;

[0179] The fourth image features are encoded and decoded using a codec to obtain the deformation fields corresponding to the first and second images.

[0180] In one possible implementation, the device further includes:

[0181] A training set determination module is used to determine a training set comprising multiple groups of sample images, wherein the groups of sample images include a first sample image and a second sample image.

[0182] The model training module is used to train the first image encoder, the second image encoder, and the codec based on the training set.

[0183] In one possible implementation, the model training module is further configured to:

[0184] The corresponding predicted deformation field is determined by inputting the first sample image and the second sample image in each sample image group into the first image encoder and the second image encoder, respectively.

[0185] The first sample image is subjected to nonlinear deformation based on the predicted deformation field to obtain a predicted deformation image;

[0186] The predicted deformed image is projected to obtain a predicted third image registered with the sample second image;

[0187] The prediction loss is determined based on the predicted third image and the sample first image;

[0188] The parameters of the first image encoder, the second image encoder, and the encoder / decoder are adjusted based on the predicted loss.

[0189] In one possible implementation, the model training module is further configured to:

[0190] Determine at least one of the following losses between the predicted third image and the sample first image: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and key point registration loss;

[0191] The prediction loss is obtained by calculating the weighted sum of at least one of the following losses: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and keypoint registration loss.

[0192] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0193] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0194] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0195] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0196] Figure 8 A schematic diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0197] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). Electronic device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM Mac OS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0198] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0199] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0200] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0201] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0202] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0203] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0204] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0205] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0206] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0207] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image registration method, characterized in that, The method includes: A first image and a second image are obtained by image acquisition of the target physiological region, wherein the first image is a three-dimensional image and the second image is a two-dimensional image; Determine the deformation fields corresponding to the first image and the second image, wherein the deformation fields are used to describe the mapping relationship between the first image and the second image; The first image is nonlinearly deformed according to the deformation field to obtain a deformed image; The deformed image is projected to obtain a third image registered with the second image, and the third image is a two-dimensional image.

2. The method according to claim 1, characterized in that, The acquisition of the first and second images obtained by image acquisition of the target physiological region includes: Acquire a first image and a sequence of candidate images obtained by image acquisition of the target physiological region, wherein the sequence of candidate images includes multiple two-dimensional candidate images in temporal order; The candidate image sequence is subjected to dynamic frame fusion to obtain the second image.

3. The method according to claim 2, characterized in that, The step of performing dynamic frame fusion on the candidate image sequence to obtain the second image includes: The first frame image in the candidate image sequence is determined as the target candidate image, and the images other than the target candidate image are other candidate images; Calculate the difference between each of the other candidate images in the candidate image sequence and the target candidate image to obtain the corresponding silhouette image; The second image is obtained by averaging each of the silhouette images.

4. The method according to claim 1, characterized in that, Determining the deformation fields corresponding to the first image and the second image includes: The first image feature is extracted from the first image by the first image encoder. The first image feature is a three-dimensional image feature. The second image features are extracted from the second image using a second image encoder. The second image features are two-dimensional image features. The second image features are extended to obtain a three-dimensional third image feature; The fourth image feature is obtained by stitching together the first image feature and the third image feature; The fourth image features are encoded and decoded using a codec to obtain the deformation fields corresponding to the first and second images.

5. The method according to claim 4, characterized in that, The method further includes: A training set is determined, comprising multiple groups of sample images, wherein the groups of sample images include a first sample image and a second sample image. The first image encoder, the second image encoder, and the codec are trained based on the training set.

6. The method according to claim 5, characterized in that, The step of training the first image encoder, the second image encoder, and the codec based on the training set includes: By inputting the first sample image and the second sample image in each sample image group into the first image encoder and the second image encoder respectively, the corresponding predicted deformation field is determined; The first sample image is subjected to nonlinear deformation based on the predicted deformation field to obtain a predicted deformation image; The predicted deformed image is projected to obtain a predicted third image registered with the sample second image; The prediction loss is determined based on the predicted third image and the sample first image; The parameters of the first image encoder, the second image encoder, and the encoder / decoder are adjusted based on the predicted loss.

7. The method according to claim 6, characterized in that, The step of determining the prediction loss based on the predicted third image and the sample first image includes: Determine at least one of the following losses between the predicted third image and the sample first image: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and key point registration loss; The prediction loss is obtained by calculating the weighted sum of at least one of the following losses: mutual information loss, gradient smoothing loss, deformation direction consistency loss, volume conservation loss, and keypoint registration loss.

8. An image registration device, characterized in that, The device includes: The image acquisition module is used to acquire a first image and a second image obtained by image acquisition of the target physiological region, wherein the first image is a three-dimensional image and the second image is a two-dimensional image; A deformation field determination module is used to determine the deformation fields corresponding to the first image and the second image, wherein the deformation fields are used to describe the mapping relationship between the first image and the second image; A deformation module is used to perform nonlinear deformation on the first image according to the deformation field to obtain a deformed image; The projection module is used to project the deformed image to obtain a third image registered with the second image, wherein the third image is a two-dimensional image.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7 when executing instructions stored in the memory.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images