Registration method and device based on multi-modal image, equipment and storage medium

Abstract

The invention discloses a registration method and device based on a multi-modal image, equipment and a storage medium, and the method comprises the steps: inputting a preprocessed target 3D-C arm image, a target preoperative CT image, a first mask and a second mask into a diffusion model module, and obtaining a high-resolution CT image and a third mask; inputting the preprocessed target 3D-SPACE MR image and the preprocessed high-resolution CT image into a multi-affine matrix estimation module to obtain a multi-scale feature image; and inputting the multi-scale feature image into an affine elastic fusion module to obtain a first rigid deformation field and a first elastic deformation field, and inputting the first rigid deformation field and the first elastic deformation field into a local rigid constraint module to obtain a target registration image. The rigid-elastic hybrid registration network and the diffusion model module are combined, the 3D-C arm image and the MR image are automatically registered, and compared with the scheme that operation navigation is conducted only through the 3D-C arm image or the CT image, the comprehensive and accurate lumbosacral intervertebral foramen 3D structure can be visually displayed in the operation.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of image processing technology, and in particular to a registration method, device, equipment, and storage medium based on multimodal images. Background Art

[0002] The anatomical structure of the lumbar intervertebral foramen of the human body is complex, containing important anatomical structures such as nerve roots, arteriovenous vessels, and intervertebral foraminal ligaments. The inner side is the dura mater and the walking nerve roots. The thickening of the yellow ligament and the hyperplasia of the articular process joints caused by degeneration can further change the morphology of the intervertebral foramen. These factors increase the difficulty of intervertebral foraminal puncture. Based on this, the prior art proposes to use a surgical navigation system guided by 3D-C-arm images or CT images to assist doctors in rapid positioning and puncture to reduce the difficulty of surgical operations and reduce surgical time. Although these navigation systems have reduced the difficulty of surgical operations and improved surgical efficiency to a certain extent, 3D-C-arm images and CT images cannot clearly display important soft tissue anatomical structures such as lumbar nerve roots, dura mater, and intervertebral discs. Therefore, it is still a challenge to intuitively, comprehensively, and in detail visualize the 3D structure of the patient's lumbar intervertebral foramen during surgery. Summary of the invention

[0003] The embodiments of the present application provide a multimodal image-based registration method, apparatus, device, and storage medium, which can provide an effective data basis for visually displaying a more comprehensive and accurate 3D structure of the patient's lumbar intervertebral foramen during surgery.

[0004] In a first aspect, an embodiment of the present application provides a registration method based on a multimodal image, comprising:

[0005] Acquiring an initial 3D-C-arm image, an initial preoperative CT image, and an initial 3D-SPACE MR image, wherein the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image are images of the lumbar sacral region of the target object;

[0006] Performing image preprocessing on the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image to obtain a preprocessing result;

[0007] Acquire a preset image registration model, wherein the image registration model includes a diffusion model module, a multi-affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module connected in sequence;

[0008] A target 3D-C-arm image, a target preoperative CT image, a first mask, and a second mask are obtained from the preprocessing results, and the target 3D-C-arm image, the target preoperative CT image, the first mask, and the second mask are input into the diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, wherein the first mask is a two-dimensional mask corresponding to the target 3D-C-arm image for each vertebral structure and each soft tissue structure in the lumbar sacral region, the second mask is a two-dimensional mask corresponding to the target preoperative CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region, and the third mask is a two-dimensional mask corresponding to the high-resolution CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region;

[0009] Acquire a target 3D-SPACE MR image from the preprocessing result, and input the high-resolution CT image and the target 3D-SPACE MR image into the multi-affine matrix estimation module for feature extraction to obtain a multi-scale feature image;

[0010] Inputting the multi-scale feature image into the affine elastic fusion module for image stitching processing to obtain a first rigid deformation field and a first elastic deformation field;

[0011] The first rigid deformation field and the first elastic deformation field are input into the local rigid constraint module to obtain a target registration image.

[0012] In some embodiments, the diffusion model module includes a U-Net model, and the target 3D-C arm image, the target preoperative CT image, the first mask, and the second mask are input into the diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, including:

[0013] Using a preset first Markov chain, a preset number of Gaussian noise addition operations are performed on the target 3D-C-arm image and the first mask to obtain a pure noise image, wherein each Gaussian noise addition operation corresponds to obtaining a first intermediate noise image;

[0014] Using the U-Net model to obtain noise distribution information of each of the first intermediate noise images;

[0015] Based on a preset second Markov chain and each of the noise distribution information, performing a preset number of noise signal subtraction operations on the pure noise image to obtain a second intermediate noise image;

[0016] The high-resolution CT image and the third mask are obtained by pairing the second intermediate noise image with the target preoperative CT image for training.

[0017] In some embodiments, the multi-affine matrix estimation module includes a plurality of first convolution blocks, a flat layer, a preset number of dense layers and the preset number of affine dense layers connected in sequence, each of the dense layers is connected to an affine dense layer, each of the dense layers corresponds to different vertebral structures and different soft tissue structures in the lumbar sacral region, and the high-resolution CT image and the target 3D-SPACE MR image are input into the multi-affine matrix estimation module for feature extraction to obtain a multi-scale feature image, including:

[0018] Inputting the high-resolution CT image and the target 3D-SPACE MR image into a plurality of the first convolution blocks to perform multi-scale image feature extraction to obtain a first intermediate feature image;

[0019] Inputting the first intermediate feature image into the flat layer and the preset number of dense layers to obtain the preset number of affine matrices;

[0020] Inputting each of the affine matrices into a corresponding affine dense layer to obtain each second intermediate feature image;

[0021] Each of the second intermediate feature images is connected with preset size information to obtain each of the multi-scale feature images.

[0022] In some embodiments, the preprocessing result further includes a plurality of fourth masks, each of which is a two-dimensional mask corresponding to each vertebral structure and each soft tissue structure in the lumbar sacral region of the target 3D-SPACE MR image, and the method further includes:

[0023] Inputting each of the fourth masks into a corresponding affine dense layer, and outputting a first rigid transformation parameter corresponding to each of the vertebral structures;

[0024] Using a fully connected network to estimate the second rigid transformation parameters of each of the third masks;

[0025] Calculating a target rigid deformation field of each of the vertebral structures based on a preset cost function, each of the first rigid transformation parameters, and each of the second rigid transformation parameters;

[0026] Performing distance-weighted average calculation on all the target rigid deformation fields in sequence to obtain an initial rigid deformation field corresponding to each vertebral structure;

[0027] All of the initial rigid deformation fields are upsampled to obtain initial deformation fields corresponding to each of the vertebral structures.

[0028] In some embodiments, the affine elastic fusion module includes a plurality of combination units, a plurality of second convolution blocks and an integrated block connected in sequence, the combination unit includes a third convolution block and a first upsampling layer, the second convolution block includes a convolution layer and an activation function layer, the integrated block includes a vertical layer and a scaling layer, the multi-scale feature image is input into the affine elastic fusion module for image stitching processing, and a first rigid deformation field and a first elastic deformation field are obtained, including:

[0029] Inputting the multi-scale feature image into a plurality of the combination units for feature fusion to obtain a third intermediate feature image;

[0030] The third intermediate feature image is input into a plurality of the second convolution blocks and the integrated block for multiple convolution and scaling conversion processing to obtain the first rigid deformation field and the first elastic deformation field, wherein the first rigid deformation field and the first elastic deformation field are respectively the rigid deformation field corresponding to each of the bony structures and the elastic deformation field corresponding to each of the soft tissue structures in the lumbar sacral region of the target object.

[0031] In some embodiments, the method further comprises:

[0032] splicing the fourth mask with the first rigid deformation field and the first elastic deformation field to obtain a second rigid deformation field and a second elastic deformation field;

[0033] Calculating a mutual information loss cost value between the high-resolution CT image and the second rigid deformation field and the second elastic deformation field;

[0034] Calculating elastic DICE loss values ​​between each of the initial deformation field and the second rigid deformation field and the second elastic deformation field, wherein the elastic DICE loss value is used to indicate the DICE cost of the segmented area corresponding to the rigid area of ​​each vertebral structure;

[0035] Calculating an initial registration image based on a preset spatial transformation network module, the second rigid deformation field, and the second elastic deformation field;

[0036] The parameters of the affine elastic fusion module are adjusted based on the initial registration image, the mutual information loss cost value and the elastic DICE loss value.

[0037] In some embodiments, inputting the first rigid deformation field and the first elastic deformation field into the local rigid constraint module to obtain a target registration image includes:

[0038] Using the third mask, performing least square regression on all of the first rigid deformation fields and the first elastic deformation fields to obtain a third rigid deformation field;

[0039] Calculating an error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field;

[0040] When the error value is greater than or equal to the preset threshold, adjusting the network parameters of the image registration model, inputting the target 3D-C-arm image, the first mask and the target 3D-SPACE MR image into the adjusted image registration model, recalculating the first rigid deformation field and the first elastic deformation field, re-performing least squares regression on all of the first rigid deformation field and the first elastic deformation field using the third mask to obtain the third rigid deformation field, and recalculating the error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field until the error value is less than the preset threshold;

[0041] When the error value is less than a preset error threshold, the target registration image is calculated based on a preset spatial transformation network module, the first rigid deformation field and the first elastic deformation field.

[0042] In a second aspect, an embodiment of the present application provides a control device, comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the multimodal image-based registration method as described in the first aspect.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, comprising the control device of the second aspect.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the multimodal image-based registration method as described in the first aspect.

[0045] The embodiment of the present application provides a registration method, apparatus, device, and storage medium based on multimodal images, the method comprising: acquiring an initial 3D-C-arm image, an initial preoperative CT image, and an initial 3D-SPACE MR image, wherein the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image are images of the lumbar sacral region of the target object; performing registration of the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE The MR image is preprocessed to obtain a preprocessing result; a preset image registration model is obtained, and the image registration model includes a diffusion model module, a multi-affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module connected in sequence; a target 3D-C arm image, a target preoperative CT image, a first mask and a second mask are obtained from the preprocessing result, and the target 3D-C arm image, the target preoperative CT image, the first mask and the second mask are input into the diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, wherein the first mask is a two-dimensional mask corresponding to the target 3D-C arm image for each vertebral structure and each soft tissue structure in the lumbar sacral region, the second mask is a two-dimensional mask corresponding to the target preoperative CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region, and the third mask is a two-dimensional mask corresponding to the high-resolution CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region; the target 3D-SPACE is obtained from the preprocessing result. MR image, and input the high-resolution CT image and the target 3D-SPACE MR image into the multi-affine matrix estimation module for feature extraction to obtain a multi-scale feature image; input the multi-scale feature image into the affine elastic fusion module for image stitching processing to obtain a first rigid deformation field and a first elastic deformation field; input the first rigid deformation field and the first elastic deformation field into the local rigid constraint module to obtain a target registration image. According to the solution provided in the embodiment of the present application, a diffusion model module is added to the rigid-elastic hybrid registration network to realize automatic registration of 3D-C-arm images and MR images. Compared with the solution of surgical navigation only through original 3D-C-arm images or CT images, the present application can provide an effective data basis for visually displaying a more comprehensive and accurate 3D structure of the patient's lumbar intervertebral foramen during surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flowchart of a multimodal image registration method provided in one embodiment of the present application;

[0047] Figure 2 is a structural diagram of a control device provided by another embodiment of the present application;

[0048] Figure 3 It is a structural diagram of an image registration model provided by another embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] It is understood that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flow chart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0051] The anatomical structure of the lumbar intervertebral foramen of the human body is complex, containing important anatomical structures such as nerve roots, arteriovenous vessels, and intervertebral foraminal ligaments. The inner side is the dura mater and the walking nerve roots. The thickening of the yellow ligament and the hyperplasia of the articular process joints caused by degeneration can further change the morphology of the intervertebral foramen. These factors increase the difficulty of intervertebral foraminal puncture. Based on this, the prior art proposes to use a surgical navigation system guided by 3D-C-arm images or CT images to assist doctors in rapid positioning and puncture to reduce the difficulty of surgical operations and reduce surgical time. Although these navigation systems have reduced the difficulty of surgical operations and improved surgical efficiency to a certain extent, 3D-C-arm images and CT images cannot clearly display important soft tissue anatomical structures such as lumbar nerve roots, dura mater, and intervertebral discs. Therefore, it is still a challenge to intuitively, comprehensively, and in detail visualize the 3D structure of the patient's lumbar intervertebral foramen during surgery.

[0052] In order to solve the above-mentioned problems, the embodiments of the present application provide a registration method, apparatus, device, and storage medium based on multimodal images, the method comprising: acquiring an initial 3D-C-arm image, an initial preoperative CT image, and an initial 3D-SPACE MR image, wherein the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image are images of the lumbar sacral region of the target object; performing registration of the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image; The MR image is preprocessed to obtain a preprocessing result; a preset image registration model is obtained, and the image registration model includes a diffusion model module, a multi-affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module connected in sequence; a target 3D-C arm image, a target preoperative CT image, a first mask and a second mask are obtained from the preprocessing result, and the target 3D-C arm image, the target preoperative CT image, the first mask and the second mask are input into the diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, wherein the first mask is a two-dimensional mask corresponding to the target 3D-C arm image for each vertebral structure and each soft tissue structure in the lumbar sacral region, the second mask is a two-dimensional mask corresponding to the target preoperative CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region, and the third mask is a two-dimensional mask corresponding to the high-resolution CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region; the target 3D-SPACE is obtained from the preprocessing result. MR image, and input the high-resolution CT image and the target 3D-SPACE MR image into the multi-affine matrix estimation module for feature extraction to obtain a multi-scale feature image; input the multi-scale feature image into the affine elastic fusion module for image stitching processing to obtain a first rigid deformation field and a first elastic deformation field; input the first rigid deformation field and the first elastic deformation field into the local rigid constraint module to obtain a target registration image. According to the solution provided in the embodiment of the present application, a diffusion model module is added to the rigid-elastic hybrid registration network to realize automatic registration of 3D-C-arm images and MR images. Compared with the solution of surgical navigation only through original 3D-C-arm images or CT images, the present application can provide an effective data basis for visually displaying a more comprehensive and accurate 3D structure of the patient's lumbar intervertebral foramen during surgery.

[0053] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0054] refer to Figure 1 , Figure 1 Yes, the embodiment of the present application provides a registration method based on multimodal images, which includes but is not limited to the following steps:

[0055] Step S110, acquiring an initial 3D-C-arm image, an initial preoperative CT image, and an initial 3D-SPACE MR image, wherein the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image are images of the lumbar sacral region of the target object;

[0056] Step S120, performing image preprocessing on the initial 3D-C-arm image, the initial preoperative CT image and the initial 3D-SPACE MR image to obtain a preprocessing result;

[0057] Step S130, obtaining a preset image registration model, the image registration model comprising a diffusion model module, a multi-affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module connected in sequence;

[0058] Step S140, obtaining a target 3D-C-arm image, a target preoperative CT image, a first mask, and a second mask from the preprocessing results, and inputting the target 3D-C-arm image, the target preoperative CT image, the first mask, and the second mask into a diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, wherein the first mask is a two-dimensional mask corresponding to the target 3D-C-arm image for each vertebral structure and each soft tissue structure in the lumbar sacral region, the second mask is a two-dimensional mask corresponding to the target preoperative CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region, and the third mask is a two-dimensional mask corresponding to the high-resolution CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region;

[0059] Step S150, obtaining a target 3D-SPACE MR image from the preprocessing result, and inputting the high-resolution CT image and the target 3D-SPACE MR image into a multi-affine matrix estimation module for feature extraction to obtain a multi-scale feature image;

[0060] Step S160, inputting the multi-scale feature image into an affine elastic fusion module for image stitching processing to obtain a first rigid deformation field and a first elastic deformation field;

[0061] Step S170: input the first rigid deformation field and the first elastic deformation field into a local rigid constraint module to obtain a target registration image.

[0062] It should be noted that the present embodiment does not limit the specific method of acquiring multimodal images, namely, the initial 3D-C-arm image, the initial 3D-SPACE MR image and the initial preoperative CT image. For example, for the initial 3D-C-arm image, the present embodiment first determines the standard lumbar spine AP and lateral perspective images (the spinous process in the AP image is located in the center of the pedicle line, the vertebral end plates in the lateral image are parallel to each other, and the surgical segment is located in the center of the perspective image), and then performs a 3D scan to obtain the 3D-C-arm image of the L4-S1 lumbar spine segment of the target object (i.e., the target patient), and exports the initial 3D-C-arm image in DICOM format; for the initial 3D-SPACE MR image, the present embodiment uses the 3D-SPACE sequence of the superconducting MR machine to obtain the lumbar and sacral spine segment image data of the target object, and exports the initial 3D-SPACE in DICOM format. MR images, specifically, the scanning range of the superconducting MR machine is the L4-S1 lumbar segment, the direction perpendicular to the long axis of the target object is the scanning baseline, and the specific parameters of the 3D-SPACE sequence are: repetition time / echo time is 2800.0ms / 189.0ms, flip angle is 45 degrees, field of view is 240×240mm, matrix size is 320×320, slice thickness is 0.8mm, bandwidth is 579kHz, and the final initial 3D-SPACE The image resolution of the MR image is 0.8×0.8×0.8mm; for the initial preoperative CT image, this embodiment uses a dual-source CT device (dual 128 layers, dynamic 800 layers) to obtain CT lumbar sacral image data, and uses the direction perpendicular to the long axis of the target object as the scanning baseline to scan the target object layer by layer without intervals. The scanning field of view is 180mm, the layer thickness is 1mm, and the transverse CT imaging data of the target object is collected and exported in DICOM format. In addition, the imaging of the target object in the supine position is performed in this embodiment.

[0063] It should be noted that this embodiment does not limit the specific content of the preprocessing results, and may include the preprocessed target 3D-C-arm image, the target 3D-SPACE MR image and the target preoperative CT image, as well as a first mask corresponding to each vertebral structure and each soft tissue structure corresponding to the target 3D-C-arm image, a second mask corresponding to each vertebral structure and each soft tissue structure corresponding to the target preoperative CT image, and a fourth mask corresponding to each vertebral structure and each soft tissue structure corresponding to the target 3D-SPACE MR image. Furthermore, this embodiment does not limit the image preprocessing method for the initial 3D-C-arm image and the initial 3D-SPACE MR image. This embodiment can be to perform N4 algorithm to remove offset field, grayscale normalization and other preprocessing operations on all initial 3D-C-arm images and initial 3D-SPACE MR images, which can effectively improve the quality and consistency of the image and provide an accurate and reliable data basis for subsequent image registration processing; for the first mask, the second mask and the fourth mask, they can be obtained by manually outlining the L4 vertebra, L4 / L5 intervertebral disc, L5 vertebra, L4 nerve root, L5 nerve root, dura mater, skin and intervertebral disc herniation lesion target on the axial multimodal image data on the annotation software.

[0064] Understandably, reference Figure 3 The image registration model of this embodiment includes a diffusion model module 300, a multi-affine matrix estimation module 400, an affine elastic fusion module 500 and a local rigid constraint module 600 which are connected in sequence. The working principle of the model is to firstly pass the 3D-C arm image through the forward denoising and reverse denoising process of the diffusion model module to remove the artifacts and other noise in the 3D-C arm image and improve the imaging quality of the bone structure, so as to synthesize the 3D-C arm image into a high-resolution CT image; then extract the image to be registered (i.e., the high-resolution CT image synthesized in the above steps and the 3D-SPACE The multi-scale image features of the MRI image) are used, and the initial deformation field of each scale in the image to be registered (i.e., each vertebral structure and each soft tissue structure corresponding to the image to be registered) is obtained by using the rigid transformation estimation parameters. Then, the image features of each scale are spliced ​​with the initial deformation field of each structure obtained by the rigid transformation estimation through the affine elastic fusion module, the rigid / elastic deformation field is calculated, and the target registered image is output. In addition, the local rigid constraint module of the present application can perform strict local rigid constraints on the vertebral structure corresponding to each feature image in the model operation process, so as to reduce the possibility of unreasonable elastic deformation of the vertebral structure during the image registration process, thereby effectively ensuring the accuracy of the target registered image.

[0065] Specifically, in some embodiments, the diffusion model module includes a U-Net model, Figure 1 Step S140 includes but is not limited to the following steps:

[0066] Step S210, using a preset first Markov chain to perform a preset number of Gaussian noise addition operations on the target 3D-C-arm image and the first mask to obtain a pure noise image, wherein one Gaussian noise addition operation corresponds to obtaining a first intermediate noise image;

[0067] Step S220, using a U-Net model to obtain noise distribution information of each first intermediate noise image;

[0068] Step S230, based on the preset second Markov chain and each noise distribution information, performing a preset number of noise signal subtraction operations on the pure noise image to obtain a second intermediate noise image;

[0069] Step S240 , performing pairing training on the second intermediate noise image and the target preoperative CT image to obtain a high-resolution CT image and a third mask.

[0070] It can be understood that the target 3D-C-arm image of this embodiment is a 3D-C-arm image in the intraoperative stage. By constructing a diffusion model module, the target 3D-C-arm image with a low signal-to-noise ratio in the source domain is converted into a high-resolution CT image in the target domain, providing an effective data basis for subsequent image registration operations.

[0071] It should be noted that the diffusion model module of this embodiment includes two steps: forward denoising and reverse denoising. Specifically, the forward denoising step is: adding Gaussian noise to the target 3D-C arm image and the first mask for a preset number of times (T steps) to obtain a pure noise image. Each time the Gaussian noise is added, a first intermediate noise image is obtained. The forward process encodes the original target 3D-C arm image into a feature space that obeys a Gaussian distribution. The process corresponds to a parameterized Markov chain, that is, the first Markov chain. The expression of the first Markov chain is as follows:

[0072]

[0073] Among them, a j is a hyperparameter for the variance of Gaussian noise, a 1:j ∈(0,1] T is a decreasing sequence.

[0074] The reverse denoising process of this embodiment is: using the U-Net model to obtain the noise distribution information of each first intermediate noise image, based on the preset second Markov chain and each noise distribution information, the pure noise image is subjected to a preset number of noise signal subtraction operations. Since the forward process noise (i.e., the noise distribution information of each first intermediate noise image) is known, the estimated noise distribution is supervised. By estimating the noise distribution, the noise signal is subtracted from the next step image to obtain the denoised previous step target image, i.e., the second intermediate noise image. This process corresponds to the second Markov chain, and the expression of the second Markov chain is as follows:

[0075]

[0076] p θ (S j-1 |S j )=N(t j-1 ;μ θ (t j ,j),∑ θ (t j ,j));

[0077] Among them, μ θ (t j ,j) and ∑ θ (t j ,j) respectively represent the mean and variance of the denoising model in the diffusion model module, and θ represents the parameters of the diffusion model module.

[0078] It can be understood that after obtaining the second intermediate noise image, the specific process of pairing the second intermediate noise image with the target preoperative CT image is as follows: the second intermediate noise image is paired with the target preoperative CT image to obtain an initial high-resolution CT image t i ′, calculate the initial high-resolution CT image t i ′ and the real CT image t i (i.e., the pixel-by-pixel loss value L between the target preoperative CT image and img , pixel-by-pixel loss value L img Calculated according to the following formula:

[0079]

[0080] At the same time, the noise distribution of each step is iteratively estimated (i.e., the noise distribution information of each first intermediate noise image), and the noise distribution error L between the predicted noise distribution and the actual noise distribution is calculated using KL divergence. noise ,,Noise distribution error L noise Calculated according to the following formula:

[0081]

[0082] Among them, the calculation optimization objective function L total =L noise +L img The objective function value L total Iterate the forward denoising and reverse denoising operations until the objective function value L total Less than the preset loss threshold, output high-resolution CT image t i ′ and the corresponding third mask, where λ is a hyperparameter used to balance the weight ratio.

[0083] In addition, this embodiment also provides a method for evaluating the image quality of a synthesized high-resolution CT image, which can be performed by separately evaluating the image quality of the high-resolution CT image t i ′ and the target paired preoperative CT image t i The mean absolute error (MAE), root mean square error (RMSE), peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used to evaluate the mean absolute error (MAE), which can measure the pixel-by-pixel distance between images. The expression of MAE is as follows:

[0084]

[0085] RMSE can calculate the mean absolute error between images. The expression of RMSE is as follows:

[0086]

[0087] PSNR can measure the contrast difference between images. The expression of PSNR is as follows:

[0088]

[0089] Among them, MAX is the maximum value of the image.

[0090] SSIM can evaluate the similarity between images. The expression of SSIM is as follows:

[0091]

[0093] Among them, μ x , μ y , σ x 2 , σ y 2 and σ xyRespectively represent the local mean, standard deviation and covariance, μ x and μ x Is a constant value.

[0094] In addition, in some embodiments, the multi-affine matrix estimation module includes a plurality of first convolution blocks, a flat layer, a preset number of dense layers, and a preset number of affine dense layers connected in sequence, each dense layer is connected to an affine dense layer, and each dense layer corresponds to a different vertebral structure and a different soft tissue structure in the lumbar sacral region, Figure 1 Step S150 includes but is not limited to the following steps:

[0095] Step S310, inputting the high-resolution CT image and the target 3D-SPACE MR image into a plurality of first convolution blocks to perform multi-scale image feature extraction to obtain a first intermediate feature image;

[0096] Step S320, inputting the first intermediate feature image into a flat layer and a preset number of dense layers to obtain a preset number of affine matrices;

[0097] Step S330, inputting each affine matrix into the corresponding affine dense layer to obtain each second intermediate feature image;

[0098] Step S340: Connect each second intermediate feature image with preset size information to obtain each multi-scale feature image.

[0099] It should be noted that, refer to Figure 3 The embodiment of the present application does not limit the specific number of the first convolution blocks 410, the flat layers 420, the dense layers and the affine dense layers 430 in the multi-affine matrix estimation module. Those skilled in the art can select them according to actual conditions. In this embodiment, the number of the first convolution blocks 410 is 4, the number of the flat layers 420 is 1, and the number of the dense layers and the affine dense layers 430 are both 11, which correspond to the L4 vertebra, L4 / L5 intervertebral disc, L5 vertebra, L4 nerve root, L5 nerve root, dura mater, ilium, iliac artery and branches, inferior vena cava and branches, psoas major muscle, skin and lesion targets, respectively.

[0100] It should be noted that each first convolution block 410 of the present embodiment is composed of a 3D convolution layer and a LeakyReLU activation function layer. The four first convolution blocks 410 are used as image feature extraction modules to extract multi-scale feature images of the target 3D-SPACE MR image to be registered and the synthetic high-resolution CT image to be registered; a flat layer 430 and multiple dense layers constitute a rigid transformation estimation module, which is used to calculate the initial deformation field of each scale; the flat layer 420 is used to project high-level image features to the corresponding affine matrices; the dense layer is used to predict the affine matrix of each structure, and the affine matrix of the overall structure is obtained by global optimization. The affine dense layer converts the affine matrix into a second intermediate feature image, and connects the second intermediate feature image with the preset size information u to obtain each multi-scale feature image, that is, the mixed feature Hu, for use by the subsequent affine elastic fusion module.

[0101] In addition, in some embodiments, the preprocessing result further includes a plurality of fourth masks, each of which is a two-dimensional mask corresponding to each vertebral structure and each soft tissue structure in the lumbar sacral region of the target 3D-SPACE MR image. The multimodal image-based registration method provided in the embodiment of the present application also includes but is not limited to the following steps:

[0102] Step S410, inputting each fourth mask into the corresponding affine dense layer, and outputting the first rigid transformation parameter corresponding to each vertebral structure;

[0103] Step S420, estimating second rigid transformation parameters of each third mask using a fully connected network;

[0104] Step S430, calculating a target rigid deformation field of each vertebral structure based on a preset cost function, each first rigid transformation parameter, and each second rigid transformation parameter;

[0105] Step S440, performing distance-weighted average calculation on all target rigid deformation fields in sequence to obtain an initial rigid deformation field corresponding to each vertebral structure;

[0106] Step S450: up-sample all initial rigid deformation fields to obtain initial deformation fields corresponding to each vertebral structure.

[0107] It can be understood that the segmentation mask of each structure of the target 3D-SPACE MR image to be registered, i.e., the fourth mask, and the segmentation mask of each structure of the synthetic high-resolution CT image to be registered, i.e., the third mask, will be used to weakly supervise the multi-affine matrix estimation module to obtain a more accurate affine matrix for each structure. The specific implementation process is as follows: input each fourth mask into the corresponding affine dense layer, and output the first rigid transformation parameter corresponding to each vertebral structure; use a fully connected network to estimate the second rigid transformation parameter of each third mask; calculate the target rigid deformation field of each vertebral structure based on a preset cost function, each first rigid transformation parameter, and each second rigid transformation parameter. The cost function of this embodiment uses the DICE cost of the segmented area corresponding to each rigid area of ​​the lumbar vertebral region; perform distance weighted average calculation on all target rigid deformation fields in turn to obtain the initial rigid deformation field corresponding to each vertebral structure; perform upsampling on all initial rigid deformation fields to obtain the initial deformation field corresponding to each vertebral structure, providing an effective data basis for the subsequent estimation of rigid deformation field and elastic deformation field.

[0108] In addition, in some embodiments, the affine elastic fusion module includes a plurality of combination units, a plurality of second convolution blocks and an integrated block connected in sequence, the combination unit includes a third convolution block and a first upsampling layer, the second convolution block includes a convolution layer and an activation function layer, the integrated block includes a vertical layer and a scaling layer, Figure 1 Step S160 includes but is not limited to the following steps:

[0109] Step S510, inputting the multi-scale feature image into a plurality of combination units for feature fusion to obtain a third intermediate feature image;

[0110] In step S520, the third intermediate feature image is input into a plurality of second convolution blocks and integrated blocks for multiple convolution and scaling transformation processes to obtain a first rigid deformation field and a first elastic deformation field, wherein the first rigid deformation field and the first elastic deformation field are respectively the rigid deformation field and the elastic deformation field corresponding to each bony structure in the lumbar sacral region of the target object.

[0111] It should be noted that this embodiment does not limit the specific number of the combination unit, the second convolution block and the integrated block, refer to Figure 3The affine elastic fusion module consists of four combination units 510, three second convolution blocks 520 and an integrated block 530, which are used to estimate the rigid deformation field and the elastic deformation field. The combination unit 510 is modified from the traditional decoding network in UNet. For each combination unit 510, the input multi-scale feature image corresponds to a coding feature Fu and a mixed feature Hu of scale u, and the output is a combined feature Ou-1 of scale u-1; three second convolution blocks 520 and one integrated block 530 are used to fuse multi-resolution rigid information and elastic information, and estimate the corresponding rigid deformation field and elastic deformation field. Specifically, the multi-scale feature image is input into multiple combination units 510 for feature fusion to obtain a third intermediate feature image, and the third intermediate feature image is input into multiple second convolution blocks 520 and integrated blocks 530 for multiple convolution and scaling conversion processing to obtain a first rigid deformation field and a first elastic deformation field. The first rigid deformation field and the first elastic deformation field are respectively the rigid deformation field and the elastic deformation field corresponding to each bony structure in the lumbar sacral region of the target object, providing an effective data basis for the subsequent output of the target registration image.

[0112] In addition, in some embodiments, the multimodal image-based registration method provided in the embodiments of the present application further includes but is not limited to the following steps:

[0113] Step S610, splicing the fourth mask with the first rigid deformation field and the first elastic deformation field to obtain a second rigid deformation field and a second elastic deformation field;

[0114] Step S620, calculating the mutual information loss cost between the high-resolution CT image and the second rigid deformation field and the second elastic deformation field;

[0115] Step S630, calculating elastic DICE loss values ​​between each initial deformation field and the second rigid deformation field and the second elastic deformation field, wherein the elastic DICE loss value is used to indicate the DICE cost of the segmented area corresponding to the rigid area of ​​each vertebral structure;

[0116] Step S640, calculating an initial registration image based on a preset space transformation network module, a second rigid deformation field, and a second elastic deformation field;

[0117] Step S650 , adjusting parameters of the affine elastic fusion module based on the initial registration image, the mutual information loss cost value and the elastic DICE loss value.

[0118] In addition, in some embodiments, the fourth mask of each structure corresponding to the 3D-SPACE MR image to be registered and the third mask corresponding to the high-resolution CT image to be registered can be used to weakly supervise the affine elastic fusion module to obtain the optimal rigid deformation field and elastic deformation field. Specifically, the fourth mask is spliced ​​with the first rigid deformation field and the first elastic deformation field to obtain the second rigid deformation field and the second elastic deformation field; the mutual information loss cost value between the high-resolution CT image and the second rigid deformation field and the second elastic deformation field is calculated; the elastic DICE loss value between each initial deformation field corresponding to the third mask and the second rigid deformation field and the second elastic deformation field is calculated, wherein the elastic DICE loss value is used to indicate the DICE cost of the segmented area corresponding to the rigid area of ​​each vertebral structure; the elastic DICE loss value between each initial deformation field and the second rigid deformation field and the second elastic deformation field is calculated, wherein the elastic DICE loss value is used to indicate the DICE cost of the segmented area corresponding to the rigid area of ​​each vertebral structure; the initial registration image is calculated based on the preset spatial transformation network module, the second rigid deformation field and the second elastic deformation field; the parameters of the affine elastic fusion module are adjusted based on the initial registration image, the mutual information loss cost value and the elastic DICE loss value to realize network supervision to obtain the optimal rigid deformation field and elastic deformation field.

[0119] Additionally, in some embodiments, Figure 1 Step S170 includes but is not limited to the following steps:

[0120] Step S710, using the third mask to perform least square regression on all the first rigid deformation fields and the first elastic deformation fields to obtain a third rigid deformation field;

[0121] Step S720, calculating the error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field;

[0122] Step S730, when the error value is greater than or equal to a preset error threshold, adjusting the network parameters of the image registration model, inputting the target 3D-C-arm image, the first mask and the target 3D-SPACE MR image into the adjusted image registration model, recalculating the first rigid deformation field and the first elastic deformation field, re-performing least squares regression on all the first rigid deformation fields and the first elastic deformation fields using the second mask to obtain a third rigid deformation field, and recalculating the error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field until the error value is less than the preset threshold;

[0123] Step S740: When the error value is less than a preset threshold, a target registration image is calculated based on a preset spatial transformation network module, a first rigid deformation field, and a first elastic deformation field.

[0124] It can be understood that the local rigid constraint module of this embodiment is used to ensure the reasonable rigid deformation of the bony structure of the lumbar sacral region corresponding to the image during the image registration process, so as to avoid unreasonable elastic deformation of the vertebral structure during the registration process, which affects the quality of the target configuration image. First, the third mask corresponding to the high-resolution CT image is used to perform least squares regression on all the first rigid deformation fields and the first elastic deformation fields to obtain the third rigid deformation field, that is, the rigid deformation field and the elastic deformation field of each vertebral structure rigid structure area are calculated, and the error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field is calculated to form a local rigid constraint cost function, and the error is fed back. When the error value is greater than or equal to the preset error threshold, it means that the current target registration image is not accurate enough, and the network parameters of the image registration model are adjusted to convert the target 3D-C-arm image, the first mask and the target 3D-SPACE The MR image is input into the adjusted image registration model, the first rigid deformation field and the first elastic deformation field are recalculated, the second mask is used to re-perform least squares regression on all the first rigid deformation fields and the first elastic deformation fields to obtain the third rigid deformation field, and the error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field is recalculated until the error value is less than the preset threshold; when the error value is less than the preset threshold, the target registration image is calculated based on the preset spatial transformation network module, the first rigid deformation field and the first elastic deformation field, thereby ensuring the quality of the target registration image and providing an effective data basis for visually displaying a more comprehensive and accurate 3D structure of the patient's lumbar intervertebral foramen during surgery.

[0125] In addition, this embodiment also provides a method for verifying the registration accuracy of the target 3D-C-arm image and the target 3D-SPACE MR image. Specifically, this embodiment adopts three traditional registration network models (affine matrix-based method, local affine matrix-based method and batch normalization symmetry-based method), three deep learning-based registration network models (deformable registration model based on VoxelMorph, deformable registration model based on VoxelMorph and mask, and deformable registration model based on VoxelMorph, mask and local rigid constraint) and Figure 3The image registration model shown in the figure aligns the target 3D-C-arm image and the target 3D-SPACE MR image of the target object respectively, and obtains 7 registered target registration images (including the registered 3D-C-arm image and the registered 3D-SPACE MR image). Each target registration image corresponds to a different registration method, and the registration time, Dice coefficient, and registration error of the anatomical point are used to evaluate the performance of the target registration images of the 7 method models. Among them, the anatomical points are selected by manually marking the anatomical points corresponding to the left and right pedicles of the two vertebrae L4 and L5 on the registered 3D-SPACE MR image and the registered 3D-C-arm image (the left and right anatomical points of the transverse diameter and the upper and lower anatomical points of the longitudinal diameter when the cross-sectional area of ​​the pedicle is the largest, and there are 16 anatomical points in total for the four pedicles of the two vertebrae) to obtain the manual marking results; the Euclidean distance between the anatomical points of the registered 3D-SPACE MR image and the anatomical points of the registered 3D-C-arm image in each target registration image is calculated, and the registration error is evaluated based on the Euclidean distance.

[0126] like Figure 2 As shown, Figure 2 2 is a structural diagram of a control device provided in an embodiment of the present application. The present invention also provides a control device 200, including:

[0127] The processor 210 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0128] The memory 220 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 220 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 220, and the processor 210 calls and executes the multimodal image-based registration method of the embodiment of the present application;

[0129] Input / output interface 230, used to implement information input and output;

[0130] Communication interface 240, used to realize communication interaction between the apparatus and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0131] bus 250 , which transmits information between the various components of the device (e.g., processor 210 , memory 220 , input / output interface 230 , and communication interface 240 );

[0132] The processor 210 , the memory 220 , the input / output interface 230 , and the communication interface 240 are connected to each other in communication within the device via the bus 250 .

[0133] In addition, an embodiment of the present application further provides an electronic device, including the control device 200 of the above embodiment.

[0134] In addition, an embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned multimodal image-based registration method is implemented.

[0135] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and are implemented to be located in one place, or may also be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0137] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A registration method based on multimodal images, characterized in that: include: Acquiring an initial 3D-C-arm image, an initial preoperative CT image, and an initial 3D-SPACE MR image, wherein the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image are images of the lumbar sacral region of the target object; Performing image preprocessing on the initial 3D-C-arm image, the initial preoperative CT image, and the initial 3D-SPACE MR image to obtain a preprocessing result; Acquire a preset image registration model, wherein the image registration model includes a diffusion model module, a multi-affine matrix estimation module, an affine elastic fusion module and a local rigid constraint module connected in sequence; A target 3D-C-arm image, a target preoperative CT image, a first mask, and a second mask are obtained from the preprocessing results, and the target 3D-C-arm image, the target preoperative CT image, the first mask, and the second mask are input into the diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, wherein the first mask is a two-dimensional mask corresponding to the target 3D-C-arm image for each vertebral structure and each soft tissue structure in the lumbar sacral region, the second mask is a two-dimensional mask corresponding to the target preoperative CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region, and the third mask is a two-dimensional mask corresponding to the high-resolution CT image for each vertebral structure and each soft tissue structure in the lumbar sacral region; Acquire a target 3D-SPACE MR image from the preprocessing result, and input the high-resolution CT image and the target 3D-SPACE MR image into the multi-affine matrix estimation module for feature extraction to obtain a multi-scale feature image; Inputting the multi-scale feature image into the affine elastic fusion module for image stitching processing to obtain a first rigid deformation field and a first elastic deformation field; The first rigid deformation field and the first elastic deformation field are input into the local rigid constraint module to obtain a target registration image.

2. The multimodal image registration method according to claim 1, characterized in that: The diffusion model module includes a U-Net model, and the target 3D-C arm image, the target preoperative CT image, the first mask, and the second mask are input into the diffusion model module for forward denoising and reverse denoising to obtain a high-resolution CT image and a third mask, including: Using a preset first Markov chain, a preset number of Gaussian noise addition operations are performed on the target 3D-C-arm image and the first mask to obtain a pure noise image, wherein each Gaussian noise addition operation corresponds to obtaining a first intermediate noise image; Using the U-Net model to obtain noise distribution information of each of the first intermediate noise images; Based on a preset second Markov chain and each of the noise distribution information, performing a preset number of noise signal subtraction operations on the pure noise image to obtain a second intermediate noise image; The high-resolution CT image and the third mask are obtained by pairing the second intermediate noise image with the target preoperative CT image for training.

3. The multimodal image registration method according to claim 1, characterized in that: The multi-affine matrix estimation module includes a plurality of first convolution blocks, a flat layer, a preset number of dense layers and the preset number of affine dense layers connected in sequence, each of the dense layers is connected to an affine dense layer, each of the dense layers corresponds to different vertebral structures and different soft tissue structures in the lumbar sacral region, the high-resolution CT image and the target 3D-SPACE MR image are input into the multi-affine matrix estimation module for feature extraction, and a multi-scale feature image is obtained, including: Inputting the high-resolution CT image and the target 3D-SPACE MR image into a plurality of the first convolution blocks to perform multi-scale image feature extraction to obtain a first intermediate feature image; Inputting the first intermediate feature image into the flat layer and the preset number of dense layers to obtain the preset number of affine matrices; Inputting each of the affine matrices into a corresponding affine dense layer to obtain each second intermediate feature image; Each of the second intermediate feature images is connected with preset size information to obtain each of the multi-scale feature images.

4. The multimodal image-based registration method according to claim 3, characterized in that: The preprocessing result further includes a plurality of fourth masks, each of which is a two-dimensional mask corresponding to each vertebral structure and each soft tissue structure in the lumbar sacral region in the target 3D-SPACE MR image. The method further includes: Inputting each of the fourth masks into a corresponding affine dense layer, and outputting a first rigid transformation parameter corresponding to each of the vertebral structures; Using a fully connected network to estimate the second rigid transformation parameters of each of the third masks; Calculating a target rigid deformation field of each of the vertebral structures based on a preset cost function, each of the first rigid transformation parameters, and each of the second rigid transformation parameters; Performing distance-weighted average calculation on all the target rigid deformation fields in sequence to obtain an initial rigid deformation field corresponding to each vertebral structure; All of the initial rigid deformation fields are upsampled to obtain initial deformation fields corresponding to each of the vertebral structures.

5. The multimodal image-based registration method according to claim 4, characterized in that: The affine elastic fusion module includes a plurality of combination units, a plurality of second convolution blocks and an integrated block connected in sequence, the combination unit includes a third convolution block and a first upsampling layer, the second convolution block includes a convolution layer and an activation function layer, the integrated block includes a vertical layer and a scaling layer, the multi-scale feature image is input into the affine elastic fusion module for image stitching processing, and a first rigid deformation field and a first elastic deformation field are obtained, including: Inputting the multi-scale feature image into a plurality of the combination units for feature fusion to obtain a third intermediate feature image; The third intermediate feature image is input into a plurality of the second convolution blocks and the integrated block for multiple convolution and scaling conversion processing to obtain the first rigid deformation field and the first elastic deformation field, wherein the first rigid deformation field and the first elastic deformation field are respectively the rigid deformation field corresponding to each of the bony structures and the elastic deformation field corresponding to each of the soft tissue structures in the lumbar sacral region of the target object.

6. The multimodal image-based registration method according to claim 5, characterized in that: The method further comprises: splicing the fourth mask with the first rigid deformation field and the first elastic deformation field to obtain a second rigid deformation field and a second elastic deformation field; Calculating a mutual information loss cost value between the high-resolution CT image and the second rigid deformation field and the second elastic deformation field; Calculating elastic DICE loss values ​​between each of the initial deformation field and the second rigid deformation field and the second elastic deformation field, wherein the elastic DICE loss value is used to indicate the DICE cost of the segmented area corresponding to the rigid area of ​​each vertebral structure; Calculating an initial registration image based on a preset spatial transformation network module, the second rigid deformation field, and the second elastic deformation field; The parameters of the affine elastic fusion module are adjusted based on the initial registration image, the mutual information loss cost value and the elastic DICE loss value.

7. The multimodal image-based registration method according to claim 1, characterized in that: Inputting the first rigid deformation field and the first elastic deformation field into the local rigid constraint module to obtain a target registration image includes: Using the third mask, performing least square regression on all of the first rigid deformation fields and the first elastic deformation fields to obtain a third rigid deformation field; Calculating an error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field; When the error value is greater than or equal to a preset error threshold, adjusting the network parameters of the image registration model, inputting the target 3D-C-arm image, the first mask and the target 3D-SPACE MR image into the adjusted image registration model, recalculating the first rigid deformation field and the first elastic deformation field, re-performing least squares regression on all of the first rigid deformation field and the first elastic deformation field using the third mask to obtain the third rigid deformation field, and recalculating the error value between the third rigid deformation field and the first rigid deformation field and the first elastic deformation field until the error value is less than the preset threshold; When the error value is less than the preset threshold, the target registration image is calculated based on a preset spatial transformation network module, the first rigid deformation field and the first elastic deformation field.

8. A control device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the multi-modal image-based registration method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: Comprising the control device as claimed in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the multi-modal image-based registration method as described in any one of claims 1 to 7.

Images