Spine medical image registration method and system based on semantic reconstruction

Abstract

This invention provides a method and system for registering spinal medical images based on semantic reconstruction. The method involves constructing 3D-CT image data of the spine and its corresponding 2D image data; providing a 2D-3D reconstruction model to reconstruct the 2D image data and obtain a 3D feature map b; providing a 3D-3D feature extraction model that takes the 3D-CT image data as input to obtain a feature map s with the same dimension as the 3D feature map; providing a parallel double-U-shaped network model based on cross-attention to fuse and register feature maps b and s; optimizing the model using a multi-weight loss function based on pixel and semantic information; training the model to obtain a registration model; and using the registration model to register spinal medical images. This invention avoids the information and accuracy loss caused by dimensionality reduction, improves the 2D / 3D information fusion capability, enhances the attention weight of key semantic regions, and can reduce the number of image captures while achieving accurate spinal image registration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image registration, in particular to a spine medical image registration method and system based on semantic reconstruction, and also relates to a corresponding computer terminal and computer readable storage medium. BACKGROUND

[0002] Currently, multi-dimensional image guidance is usually relied on in the surgical site for navigation in spine surgery. Before surgery, high-quality 3D-CT images are used for surgical planning and simulation. During surgery, X-rays generated by a C-arm are used for surgical guidance. An indispensable technique is to project the high-quality pre-scan 3D-CT images onto the intraoperative C-arm generated X-ray images for rigid registration. By guiding the physician to obtain the correct anatomical specific standard projection, the registered intraoperative imaging can further guide, monitor and evaluate the surgical results. However, the existing C-arm positioning is usually performed manually by the physician, which not only requires the physician to have rich experience, but also requires the physician to perform multiple fluoroscopy in most cases, which not only increases the operation time, but also increases the radiation received by the patient during surgery. In previous work, some use external tracking hardware to assist positioning, software assistance and optimization algorithms, but there are certain limitations, and there is still a certain distance from the clinic.

[0003] With the development of deep learning, there has been substantial progress in using convolutional neural networks for intelligent processing of medical images. The use of deep learning-based automatic registration has been widely studied thanks to the Digitally Reconstructed Radiograph (DRR) technology, which provides researchers with a large amount of simulated X-ray data, solving the problem of the need for large amounts of data for deep learning model training. Automatic registration based on deep learning is mainly divided into iterative registration and direct registration. Early work generally used reinforcement learning or iterative registration based on conditional judgment, while using image similarity as a measure of registration accuracy. Although the registration accuracy has been improved, the problem of more X-ray exposure of patients and registration time caused by multiple scans has not been effectively solved. In addition, due to the non-convex nature of similarity measurement, when the initial pose of the 3D model exceeds the capture range, such techniques may produce inaccurate registration results.

[0004] In contrast to iterative registration, direct registration theoretically only requires one registration, which greatly alleviates the problem of radiation dose and time. Li et al. inferred code vectors from input X-ray images and further generated DRR images, which could complete the registration of lateral head images under unsupervised conditions (Li, Peixin, et al. "Non-rigid 2D-3D registration using convolutional autoencoders." 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020.). Wu et al. constructed an architecture that combined deep learning with machine learning, achieving high-precision knee registration (Wu, Jing, Emam E. Abdel Fatah, and Mohamed R. Mahfouz. "Fully automatic initialization of two-dimensional-three-dimensional medical image registration using hybrid classifier." Journal of Medical Imaging 2.2 (2015): 024007-024007.). Further, work related to spinal registration has also been explored. Esfandiari et al. proposed a deep learning-based method to remove implants in the spine (Esfandiari, Hooman, et al. "Deep learning-based X-ray inpainting for improving spinal 2D-3D registration." The International Journal of Medical Robotics and Computer Assisted Surgery 17.2 (2021): e2228). Kausch et al., on the other hand, considered k-wire and screw implants in spinal registration, increasing the robustness of the network (Kausch, Lisa, et al. "C-arm positioning for standard projections during spinal implant placement." Medical Image Analysis 81 (2022): 102557.).

[0005] The above-mentioned technology generally uses 6DOF to measure the pros and cons of registration results, and registration failure may still occur in complex situations or situations with large initial errors. In contrast, feature-based technology has also been thoroughly researched, and features include segmentation maps, marker points, and other semantic information to assist registration. On the other hand, previous direct registration requires reducing the dimensionality of 3D CT images to 2D before registration with 2D X-rays. This technology inevitably loses information during dimensionality reduction, so it is also limited by a small capture range. With the improvement of hardware capabilities, techniques for upgrading 2D to 3D have also been tried. Shen et al. mapped the patient's projection radiograph to the corresponding 3D anatomical structure, and the trained network could generate the patient's 3D tomographic X-ray image from a single projection view (Shen, Liyue, Wei Zhao, and Lei Xing. "Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning." Nature biomedical engineering 3.11 (2019): 880-888.). Mi et al. proposed an SGReg method to reconstruct and predict segmentation maps from 2D and 3D images, so that 3D / 2D data have dimensional correspondence (Mi, Jia, et al. "SGReg: segmentation guided 3D / 2D rigid registration for orthogonal x-ray and CT images in spine surgery navigation." Physics in Medicine and Biology (2023).). Although extracting 3D features from 2D images avoids information loss, it also puts a test on the hardware requirements, and how to balance the two is still a problem to be solved in the field.

[0006] In summary, although there are currently medical image registration techniques based on different methods, the 3D / 2D registration of CT and X-ray images of the same patient still faces the problems of being unable to simultaneously satisfy a wide range and high precision, insufficient utilization of key semantic information, and the incompatibility of efficiency and accuracy. SUMMARY

[0007] The present application provides a spine medical image registration method and system based on semantic reconstruction to solve the above problems in the prior art, and also provides a corresponding computer terminal and computer readable storage medium.

[0008] According to an aspect of the present application, a spine medical image registration method based on semantic reconstruction is provided, comprising:

[0009] Constructing spine 3D-CT image data and its corresponding 2D image data as a training data set;

[0010] Providing a 2D-3D reconstruction model, using the 2D-3D reconstruction model to reconstruct the 2D image data to obtain the 3D feature map b corresponding to the 2D image data;

[0011] Providing a 3D-3D feature extraction model, taking the 3D-CT image data as the input of the 3D-3D feature extraction model to obtain the feature map s corresponding to the 3D-CT image data with the same dimension as the 3D feature map;

[0012] Providing a parallel double U-shaped network model based on cross attention, using the parallel double U-shaped network model to fuse and register the feature map b and the feature map s;

[0013] Using a multi-weight loss function based on pixel and semantic information to optimize the parallel double U-shaped network model;

[0014] Through the above steps, a registration model is trained, which is used to obtain the registration result of the spine medical image.

[0015] Preferably, the construction of the spine 3D-CT image data and its corresponding 2D image data comprises:

[0016] Obtaining spine CT data, unifying the data format of the spine CT data, and performing center cropping preprocessing to obtain spine CT preprocessing data;

[0017] Performing rigid body transformation on the spine CT preprocessing data to generate floating CT data; wherein the rigid body transformation represents the components of translation and rotation through six parameters, respectively, the translation displacement components (t x ,t y ,t z ) in three axial directions and the rotation components (r x ,r y ,r z ) around three axial directions; each parameter is randomly generated in a set range to form different combinations of 6 DOF parameters; then the DOF parameters are converted into a rigid body transformation matrix, and finally the corresponding floating CT data is generated according to the DOF parameters and the spine CT preprocessing data;

[0018] DRR projection is performed on the floating CT data, and image pixel value bit inversion and adaptive equalization processing are performed to obtain final simulated C-arm-X-ray data; finally, DRR projection is performed on the spine CT pre-processing data not subjected to rigid body transformation to generate standard registration reference 2D image data.

[0019] Preferably, the 2D-3D reconstruction model is provided, and the 2D image data is reconstructed by using the 2D-3D reconstruction model to obtain a 3D feature map b corresponding to the 2D image data, comprising:

[0020] The 2D-3D reconstruction model is constructed, the 2D image data and the artificially segmented spine mask are taken as inputs of the 2D-3D reconstruction model, and a 3D-CT volume image is reconstructed by using single-angle 2D image data projection through the 2D-3D reconstruction model, that is, a 3D feature map b is obtained.

[0021] Preferably, the 2D-3D reconstruction model comprises a representative network, a generation network and a conversion layer connected between the representative network and the generation network; wherein:

[0022] The representative network is used for extracting multi-scale features of the 2D image data layer by layer, converting high-dimensional data into embedded representation, and obtaining semantic information of hidden 3D structures in the input 2D image data;

[0023] The conversion layer is used for learning a manifold mapping function corresponding to the extracted multi-scale features, so that the extracted multi-scale features cross latitude;

[0024] The generation network is mainly composed of 3D deconvolution blocks, and is used for reconstructing high-dimensional images from feature information obtained from the conversion layer, that is, 2D image data projection corresponding 3D volume images.

[0025] Preferably, the 3D-3D feature extraction model is provided, and the 3D-CT image data is taken as an input of the 3D-3D feature extraction model to obtain a feature map s corresponding to the 3D-CT image data with the same dimension as the 3D feature map, comprising:

[0026] The 3D-3D feature extraction model is constructed by using a 3D Res-NET network;

[0027] The 3D-CT image data and the artificially segmented spine mask are taken as inputs of the 3D-3D feature extraction model, after downsampling of the 3D-CT image data, under the action of convolution and deconvolution, the key information of the spine in the 3D-CT image data is extracted and output to obtain a feature map s.

[0028] Preferably, a parallel double U-shaped network model based on cross attention is provided, and the feature map b and the feature map s are fused and registered by using the parallel double U-shaped network model, comprising:

[0029] The 3D feature map b and the feature map s are combined with the 2D image data and the 3D-CT image data respectively to form a key region feature map;

[0030] The key region feature map is window partitioned and window region partitioned;

[0031] A window-based multi-head cross attention mechanism is constructed for calculating new features between input key region feature maps with corresponding correlation degrees;

[0032] A parallel double U-shaped network is provided, the multi-head cross attention mechanism is used as a convolution layer of the parallel double U-shaped network, and a parallel double U-shaped network model is constructed, which is used to output a spine medical image registration result.

[0033] Preferably, the parallel double U-shaped network model is optimized by using a multi-weight loss function based on pixel and semantic information, comprising:

[0034] The loss function of the parallel double U-shaped network model is composed of an inter-pixel loss in the image domain and a perception loss, wherein the inter-pixel loss is used to measure the difference between the predicted image and the target image, and the perception loss is used to solve the problem of image over-smoothing caused by the inter-pixel loss, and then a multi-weight loss function L based on pixel and semantic information is constructed total :

[0035]

[0036] Wherein x is the original image, x * is the predicted image, Cov(·) is the covariance of two images, Var(·) is the variance of the image itself, μ is the sigmoid function, E(x,y) is whc, ρ is, Feature extraction.

[0037] Preferably, the method of the present application further comprises:

[0038] The registration model is used to process the spine medical image to be registered, and a registration result of the spine medical image is output.

[0039] According to another aspect of the present application, a spine medical image registration system based on semantic reconstruction is provided, comprising:

[0040] A data processing module is used to construct spine 3D-CT image data and its corresponding 2D image data as a training data set;

[0041] a 2D-3D reconstruction module configured to provide a 2D-3D reconstruction model, and reconstruct the 2D image data to obtain a 3D feature map b corresponding to the 2D image data by using the 2D-3D reconstruction model;

[0042] a 3D-3D feature extraction module configured to provide a 3D-3D feature extraction model, and obtain a feature map s corresponding to the 3D-CT image data by using the 3D-3D feature extraction model with the 3D-CT image data as an input of the 3D-3D feature extraction model, wherein the feature map s has the same dimension as the 3D feature map b;

[0043] a registration module configured to provide a parallel double-U network model based on cross-attention, and fuse and register the feature map b and the feature map s by using the parallel double-U network model; and a multi-weight loss function based on pixel and semantic information is used to optimize the parallel double-U network model, so as to output an optimal registration result of the spinal medical image.

[0044] According to a third aspect of the present application, a computer terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program to perform the method according to any one of the preceding aspects of the present application, or to run the system according to any one of the preceding aspects of the present application.

[0045] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to perform the method according to any one of the preceding aspects of the present application, or to run the system according to any one of the preceding aspects of the present application.

[0046] Thanks to the above technical solutions, the present application has at least one of the following beneficial effects compared with the prior art:

[0047] The spinal medical image registration method and system based on semantic reconstruction provided by the present application are different from the traditional technology of registering 3D CT with 2D X-ray after reducing the dimension of 3D CT to 2D, and can effectively avoid the problems of registration failure and large error caused by information and precision loss by extracting 3D feature maps from 2D information through a 2D-3D reconstruction network.

[0048] The spinal medical image registration method and system based on semantic reconstruction provided by the present application fully utilize the bone joint-semantic information extracted by the 2D-3D reconstruction network and the 3D-3D feature extraction network, reduce the size of the 3D image under the premise of avoiding loss of key information, and ensure the accuracy and precision of registration without increasing the required computing power of the network. BRIEF DESCRIPTION OF DRAWINGS

[0049] Other features, objects, and advantages of the application will become more apparent from the following detailed description when read in connection with the following drawings:

[0050] Figure 1 Workflow diagram of the spinal medical image registration method based on semantic reconstruction in an embodiment of the present application

[0051] Figure 2 Working principle diagram of the 2D-3D spinal medical image registration based on semantic reconstruction in a preferred embodiment of the present application.

[0052] Figure 3 Working diagram of the feature fusion based on cross-attention in a preferred embodiment of the present application.

[0053] Figure 4 Working diagram of the parallel double-U network model based on cross-attention in a preferred embodiment of the present application.

[0054] Figure 5 Spinal registration effect diagram in a specific application example of the present application; wherein (a) and (d) are fixed images respectively, (b) and (e) are floating images respectively, and (c) and (f) are registration prediction images respectively.

[0055] Figure 6 Composition module diagram of the spinal medical image registration system based on semantic reconstruction in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail as follows: The embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation modes and specific operation processes are given. It should be noted that, for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0057] An embodiment of the present application provides a spinal medical image registration method based on semantic reconstruction. The method uses a deep learning method to replace a traditional method to realize 2D-3D registration of 3D-CT medical images and 2D-X-ray images of a spine of the same patient, solves the problems that a large range and high precision cannot be satisfied at the same time, key semantic information utilization rate is insufficient, and efficiency and accuracy rate cannot be compatible, so as to assist doctors to obtain more accurate detection effect and improve work efficiency.

[0058] As shown in Figure 1 The spinal medical image registration method based on semantic reconstruction provided by the embodiment can include the following operations:

[0059] S1, construct the 3D-CT image data of the spine and its corresponding 2D image data as a training data set;

[0060] S2, provide a 2D-3D reconstruction model, and reconstruct the 2D image data by using the 2D-3D reconstruction model to obtain the 3D feature map b corresponding to the 2D image data;

[0061] S3, provide a 3D-3D feature extraction model, and take the 3D-CT image data as the input of the 3D-3D feature extraction model to obtain the feature map s corresponding to the 3D-CT image data with the same dimension as the 3D feature map;

[0062] S4, provide a parallel double U-shaped network model based on cross attention, and fuse and register the feature map b and the feature map s by using the parallel double U-shaped network model;

[0063] S5, use a multi-weight loss function based on pixel and semantic information to optimize the parallel double U-shaped network model;

[0064] S6, obtain a registration model by training S1-S5 above, and the registration model is used to obtain the registration result of the spine medical image.

[0065] In some preferred embodiments, S1 above, constructing the 3D-CT image data of the spine and its corresponding 2D image data, comprises:

[0066] S11, obtaining the CT data of the spine, performing data format unification and center cropping preprocessing on the CT data of the spine to obtain the CT preprocessing data of the spine;

[0067] S12, performing rigid body transformation on the CT preprocessing data of the spine to generate floating CT data; wherein the rigid body transformation is through six parameters, which are the components of translation and rotation, the six parameters of the rigid body transformation are the displacement components (t x ,t y ,t z ) in three axial directions and the rotation components (r x ,r y ,r z ) around three axial directions, and each parameter is randomly generated in a set range to form different combinations of 6 DOF parameters; the rigid body transformation matrix is a 4x4 matrix representing the rigid body transformation, which transforms a point from the initial position to the new position. In the present application, the rigid body transformation matrix is usually represented as T, and its form is as follows:

[0068]

[0069] wherein t is a 3x1 translation vector R is a 3x3 rotation matrix, which is obtained by multiplying three rotation matrices R x , y R z and R x , y R z , wherein

[0070]

[0071]

[0072]

[0073] Subsequently, the DOF parameters are converted into the rigid body transformation matrix according to the above formula, and finally the corresponding floating CT data is generated according to the DOF parameters and the spine CT pre-processing data;

[0074] S13, performing DRR projection on the floating CT data generated above, and performing bit-by-bit inversion and adaptive equalization processing on the image pixel values to obtain the final simulated C-arm-X-ray data. Finally, the spine CT pre-processing data not subjected to the rigid body transformation is subjected to DRR projection to generate standard registration reference 2D image data.

[0075] In some preferred embodiments, S2 above provides a 2D-3D reconstruction model, and the 2D image data is reconstructed by using the 2D-3D reconstruction model to obtain a 3D feature map b corresponding to the 2D image data, comprising:

[0076] S21, constructing a 2D-3D reconstruction model;

[0077] S22, taking the 2D image data and the artificially segmented spine mask as inputs of the 2D-3D reconstruction model, and projecting and reconstructing a 3D-CT volume image by using the 2D image data of a single angle through the 2D-3D reconstruction model, i.e., obtaining the 3D feature map b.

[0078] In some preferred embodiments, S21 above, the 2D-3D reconstruction model comprises a representative network, a generation network and a conversion layer connected between the representative network and the generation network; wherein:

[0079] The representative network is used for extracting multi-scale features of the 2D image data layer by layer, converting high-dimensional data into embedding representation, and obtaining semantic information of a hidden 3D structure in the input 2D image data;

[0080] The conversion layer is used for learning a manifold mapping function corresponding to the extracted multi-scale features, so that the extracted multi-scale features are across latitude;

[0081] The generation network is mainly composed of 3D deconvolution blocks, which are used to reconstruct high-dimensional image from the feature information obtained from the conversion layer, i.e. 2D image data projection corresponding to 3D volume image.

[0082] In some preferred embodiments, S3 provides a 3D-3D feature extraction model, which takes 3D-CT image data as input of the 3D-3D feature extraction model, and obtains a feature map s corresponding to the 3D-CT image data, which has the same dimension as the 3D feature map, including:

[0083] S31, a 3D Res-NET network is used to construct a 3D-3D feature extraction model;

[0084] S32, the 3D-CT image data and the manually segmented spine mask are taken as input of the 3D-3D feature extraction model, and after down-sampling of the 3D-CT image data, the key information of the spine in the 3D-CT image data is extracted under the action of convolution and deconvolution, and a feature map s is obtained.

[0085] In some preferred embodiments, S4 provides a parallel double-U network model based on cross attention, which is used to fuse and register the feature map b and the feature map s, including:

[0086] S41, the key region feature map b and s are window partitioned and window region partitioned;

[0087] S42, a window-based multi-head cross attention mechanism is constructed, which is used to calculate new features between the input key region feature maps with corresponding correlation degrees;

[0088] S43, a parallel double-U network is provided, which takes the multi-head cross attention mechanism as a convolution layer of the parallel double-U network, and a parallel double-U network model is constructed, which is used to output the spine medical image registration result.

[0089] In some preferred embodiments, S5 uses a multi-weight loss function based on pixel and semantic information to optimize the parallel double-U network model, including:

[0090] The loss function of the parallel double-U network model is composed of pixel loss and perception loss in the image domain, wherein: the pixel loss is used to measure the difference between the predicted image and the target image, and the perception loss is used to solve the problem of image over-smoothing caused by the pixel loss, and then a multi-weight loss function L based on pixel and semantic information is constructed. total is:

[0091]

[0092] Wherein: x is the original image, x *For prediction image, Cov(·) is the covariance of two images, Var(·) is the variance of the image itself, μ is a sigmoid function, E(x, y) is, whc is, ρ is, For feature extraction.

[0093] In some preferred embodiments, the method provided by the above embodiments of the present application can further include the following operations:

[0094] S7, using the registration model to process the spine medical image to be registered, and outputting a registration result of the spine medical image.

[0095] The technical solutions provided by the above embodiments of the present application will be described in detail in combination with a preferred embodiment.

[0096] As Figure 1 shown, the spine medical image registration method based on semantic reconstruction provided by the preferred embodiment includes the following steps:

[0097] Step 1, generation and preprocessing of spine 3D-CT image data and corresponding 2D image data;

[0098] Step 2, using a 2D-3D reconstruction model to reconstruct the 2D image data to obtain a 3D feature map;

[0099] Step 3, using a 3D-3D feature extraction model to process the 3D-CT image data to obtain a feature map of the same dimension;

[0100] Step 4, using a parallel double U-shaped network model based on cross-attention mechanism (cross-attention) to fuse and register two groups of feature maps;

[0101] Step 5, using a multi-weight loss function based on pixel and semantic information to further guide the parallel double U-shaped network model to obtain an optimal registration result;

[0102] Step 6, training a registration model through steps 1-5;

[0103] Step 7, using the registration model to process the spine medical image to be registered to obtain a registration result of the spine medical image.

[0104] The above step 1 includes the following steps:

[0105] Step 11, obtaining spine CT data, uniformly processing the data format of the spine CT data, saving it in NifTI format, and performing center clipping preprocessing.

[0106] Step 12, for the preprocessed spine CT data, making each data in t x,y,z ∈(-45mm,45mm), r x,y,z6 parameters in the range of (-45°, 45°) are randomly generated, wherein (t x ,t y ,t z ) are translation displacement components in three axial directions, and (r x ,r y ,r z ) are rotation components around three axial directions; then the DOF parameters are converted into a 4x4 rigid transformation matrix according to the formula, and finally the corresponding floating CT data is generated according to different parameters and the original spine CT data.

[0107] Step 13, using DeepDRR tool to generate DRR projection for the randomly transformed CT data to generate C-arm-X-ray data for simulation, and to generate standard registration reference 2D image data by DRR projection for the original spine CT data, and finally to perform bit inversion on the image pixel value of the C-arm-X-ray data and to perform adaptive equalization processing on the obtained image.

[0108] The above step 2 comprises the following steps:

[0109] A 3D-CT volume image is reconstructed by single-angle 2D image data projection, that is, a 3D feature map is obtained; wherein:

[0110] The 2D-3D reconstruction model comprises a representative network, a generation network, and a conversion layer connected between the representative network and the generation network, wherein the representative network extracts multi-scale features layer by layer, converts high-dimensional data into embedding representation, and the purpose is to obtain the semantic representation of the hidden 3D structure in the input projection, such as the size and position of the spine; the representative network and the generation network are connected through a conversion layer. The purpose of the conversion layer is to learn the manifold mapping function corresponding to the extracted features, so that the features can be across latitude. The generation network is composed of 3D deconvolution blocks, and finally the feature information obtained from the conversion layer is reconstructed into a high-dimensional image, that is, a 3D volume image corresponding to the projection. Through this architecture, 2D projection and 3D image share the same semantic feature representation in the feature domain, so that the model can learn how to generate 3D image from 2D projection; during the training phase, the artificial segmentation mask corresponding to the reconstructed volume is input at the same time, so as to obtain the significant region mask in the X-ray reconstructed image data at the same time.

[0111] The above step 3 comprises the following steps:

[0112] A 3D-3D feature extraction model is constructed by using a 3D Res-NET network. Wherein:

[0113] The input 3D-CT data is down-sampled, and under the action of convolution and deconvolution, the key information of the spine is extracted and the final key features are output. In the training process, the manually segmented spine mask is also input into the network to obtain the feature information and significant region mask of the three-dimensional image. Since the input of the feature extraction model is set to 3D information, the 3D image generated by CT does not need to be upscaled or down-scaled, and only needs to be processed by feature extraction to obtain a feature map with the same dimension as the 3D feature map obtained by the reconstruction model.

[0114] The above step 4 includes the following steps:

[0115] Step 4.1: Combine the 3D feature map corresponding to the 2D data and the feature map corresponding to the 3D-CT data with the 2D image data and the 3D-CT image data respectively, and perform center cropping to form key region feature maps b and s, which are used as inputs of the registration network.

[0116] Step 4.2: Perform window partitioning and window region partitioning.

[0117] Step 4.3: Construct a window-based multi-head cross-attention mechanism; the multi-head cross-attention mechanism aims to calculate a new feature with corresponding correlation between the input feature b and the feature s through attention mechanism; the feature b and the feature s are used for window-based attention calculation after generating basic windows and search windows in step 4.2; each basic window S ba is linearly projected and layer normalized to the query set, and each search window S se is linearly projected and layer normalized to the key set and the value set, and then the window-based multi-head cross-attention algorithm calculates the cross-attention between the two windows; finally, the new output set is sent to a two-layer MLP with Gelu non-linear mapping after passing through the LayerNorm (LN) layer, and the new feature with corresponding correlation between the feature b and the feature s after the multi-head cross-attention mechanism is output;

[0118] Step 4.4: Construct a parallel double U-shaped network model based on cross-attention mechanism as a registration network. The registration network uses a double U-shaped network based on a cross-attention feature fusion module to connect the floating data b and the standard data s in parallel, and outputs two features b' and s' of the same size after feature fusion. The two parallel U-shaped networks use the structure of U-NET in the encoding and decoding parts, down-sample the features in the encoder, up-sample the features in the decoder, and jump connect the encoder and the decoder. The window-based multi-head attention mechanism of step 4.3 is used to replace the convolution, so that the registration network can exchange cross-image information. Finally, a classification head is attached at the end of the network, the two features b' and s' are concatenated in the channel dimension and then input into the classification head after averaging, and finally 6 rigid transformation parameters are output after linear mapping and activation function.

[0119] The above step 5 includes the following steps:

[0120] The loss function of the registration network is composed of pixel and perceptual losses in the image domain. The pixel loss is used to measure the difference between the predicted image and the target image, and the perceptual loss alleviates the problem of image over-smoothing caused by the pixel loss, and further improves the structural similarity and perceptual similarity between the predicted image and the target image, and the formula is:

[0121]

[0122] In the formula, L total is a multi-weight loss function based on pixel and semantic information, x is the original image, x * is the predicted image, Cov(·) is the covariance of two images, Var(·) is the variance of the image itself, μ is a sigmoid function, E(x,y) is whc, ρ is, is feature extraction.

[0123] The technical solutions provided by the above embodiments of the application will be further described in detail in combination with a specific application example.

[0124] The semantic reconstruction-based spine medical image registration method used in the specific application example, as shown in Figure 1 , specifically includes the following contents:

[0125] Step S1: Generation and preprocessing of 3D-CT image and corresponding 2D data;

[0126] Step S2: Reconstruct 2D images to obtain 3D feature maps using a 2D-3D reconstruction model;

[0127] Step S3: Process 3D images to obtain feature maps of the same dimension using a 3D-3D feature extraction model;

[0128] Step S4: using a parallel double U-shaped network model based on cross attention mechanism to fuse and register the two sets of feature maps;

[0129] Step S5: using a multi-weight loss function based on pixel and semantic information to further guide the parallel double U-shaped network model to obtain an optimal registration result;

[0130] Step S6: training a registration model through steps S1-S5;

[0131] Step S7: using the registration model to process the spine medical image to be registered to obtain a registration result of the spine medical image

[0132] In order to realize the registration of 2D-3D spine registration, the application constructs a registration model based on a deep learning network architecture as shown in Figure 2 The following will describe the specific application example in detail.

[0133] Let 2D x-ray image M x-ray be an anterior-posterior (AP) view taken from a C-arm, and 3D image F CT be taken by a 3D CT before surgery, which respectively represent floating image and fixed image in spatial domain The application mainly aims at 2D-3D direct registration, so n = 2 for M x-ray and n = 3 for F CT 2D / 3D registration is to match 2D projection with 3D grayscale image, and for a rigid structure such as spine, rigid body transformation is usually used for registration, so first reconstruct a three-dimensional feature map M CT with n = 3 from M CT , and the goal is to learn the rigid body transformation matrix from M CT to F θ . Specifically, the registration model provided in the above embodiment of the application is used to parameterize the rigid body registration problem as a function f CT (M CT , F 3 ) = T, where θ is a set of rigid body transformation parameters, and T represents the predicted rigid body transformation matrix.

[0134] In step S1, the pre-processing of the spine image, and the preparation of the experimental data set specifically includes:

[0135] The largest publicly available annotated spine CT dataset, CTSpine1K, was chosen as the experimental dataset. CTSpine1K collects and annotates large-scale spine CT datasets from multiple domains and different manufacturers, totaling 1,005 CT volumes with different appearance variations (over 500,000 labeled slices and over 11,000 vertebrae). The CTSpine1K dataset is divided into a training dataset (610), a public test dataset (395), all saved in NIfTI format, with a slice size of 512*512, and the number of slices is center-cropped to 512, so the overall size of all data is 512 3 279 cases from CTSpine1K were selected as the training set, and 81 cases were selected as the test set.

[0136] For the existing CT data, each case of data has t x,y,z ∈(-45mm,45mm0,r x,y,z ∈(-45°,45°) range, then the DOF parameter is converted into a 4x4 rigid transformation matrix according to the formula, and finally the corresponding floating CT data is generated according to the different parameters and the original CT data according to the affine_transform method in the ndimage library of the scipy toolkit, a total of 1674 groups of data. Combined with the actual situation in the clinic, the parameter range in the test set is t x,y,z ∈(-20±15mm,20±15mm), r x,y,z ∈(-20±15°,20±15°), 4 different combinations of data are generated for each case, a total of 324 groups of data.

[0137] DeepDRR is a deep learning-based method for generating high-quality digital radiography (DRR). DRR is an image generated by simulating the passage of X-rays through human tissues at different projection angles, commonly used in computer-aided diagnosis (CAD) and medical image processing. Unlike traditional DRR methods based on physical models, DeepDRR uses deep convolutional neural networks (CNN) and other deep learning techniques to learn the features and structures of a large amount of medical image data, enabling the generation of more realistic and accurate DRR images.

[0138] The CT data after random transformation was projected using the DeepDRR tool to generate simulated C-arm-X-ray data, and the original CT data was projected to generate a standard registration reference image. The size of the simulated X-ray data is 128 2In order to make the C-arm X-ray data closer to the clinical reality, the data generated by DeepDRR is preprocessed in two steps. The preprocessing process includes first using the bitwise_not function of the OpenCV library to perform bit inversion of the image pixel value, and then using the createCLAHE function of the OpenCV library to perform adaptive equalization on the image.

[0139] In step S2, for feature extraction of the 2D X-ray image, different from the method of first projecting the 3D CT image to 2D and then registering with the 2D image in the traditional 3D / 2D registration, a single-angle 2D X-ray projection is used to reconstruct the 3D CT volume image. The volume reconstruction network can be specifically divided into a representative network and a generation network. The network input is a preprocessed simulated X-ray image with a size of 128 2 The representative network structure is a 2D 4*4 convolution layer, the first convolution increases the feature channel number to 64, and then the feature increases by 2 times per layer to 4096 under the action of convolution. The representative network extracts multi-scale features layer by layer, and converts high-dimensional data into embedding representation, which aims to obtain the semantic representation of the hidden 3D structure in the input projection, such as the size and position of the spine. The representative network and the generation network are connected through a conversion layer. The purpose of the conversion layer is to learn the manifold mapping function corresponding to the extracted features, so that the features can be across latitude. The conversion layer is specifically obtained by connecting a 2D convolution operation with a kernel size of 1*1 and a 3D deconvolution layer with a kernel size of 1*1*1. While keeping the feature size unchanged, the conversion layer connects the 2D and 3D feature spaces. The generation network is composed of 3D deconvolution blocks, and its structure is similar to that of the generation network. Under the action of deconvolution, it decreases the channel number by 2 times per layer to 64, and finally reconstructs the high-dimensional image with a size of 128 3 from the feature information obtained from the conversion layer, that is, the volume image corresponding to the projection. Through this architecture, 2D projection and 3D image share the same semantic feature representation in the feature domain, so that the network can learn how to generate 3D image from 2D projection. In the training stage, the artificial segmentation mask corresponding to the reconstructed volume is input at the same time, so as to obtain the significant region mask in the X-ray reconstruction image data at the same time.

[0140] In step S3, for feature extraction of the three-dimensional CT image, a 3D Res-NET network is used. The difference is that the input size of the 128 3 image data after downsampling in this specific application example is 512 3 Under the action of convolution and deconvolution, the key information of the spine is extracted and the final key features are output, which still have a size of 128 3During the training, the manually segmented spine mask is used as the input of the network to obtain the feature information and the significant region mask of the three-dimensional image. Since the input of the registration network is set to 3D information, the 3D image generated by CT does not need to be upscaled or downscaled, but only needs to be feature-extracted to obtain the feature map with the same dimension as the 2D reconstruction network.

[0141] In step S4, as shown in Figure 3 , the parallel double U-shaped network based on the cross-attention mechanism is used to fuse and register the two sets of feature maps.

[0142] Step 4.1: After the 2D X-ray and 3D CT images pass through the 2D reconstruction network and the 3D feature extraction network respectively, the intermediate feature maps with different dimensions are extracted and combined with the image data with a standard size of 512 3 × 512 3 × 512 ba , and then are center-cropped into the significant region b and s with a size of 128 ba × 128 ba × 128 ba for the next registration network. It should be noted that the feasibility of this lies in the fact that the single bone region accounts for a small proportion of the overall image. Finally, the semantic information containing only the significant region is input into the registration network to obtain the accurate registration result.

[0143] Step 4.2: window partitioning and window region partitioning

[0144] The multi-scale window partitioning includes two different methods of window partitioning and window region partitioning, which divide the input features b and s into windows of different sizes after step 4.1. The window partitioning directly divides the features into a basic window set S ba with a size of n × h × w × d, and the window region partitioning uses magnification factors a, b and g to expand the window size, where h = w = d = 2 and a = b = g = 3. Therefore, the calculation formula of the basic window and the search window size is:

[0145] h ba ,w ba ,d ba =h,w,d

[0146] h se ,w se ,d se =a h, b w, g d

[0147] where h ba , w ba , d ba are the sizes of the basic window, and h se , w se , d se are the sizes of the search window. In order to obtain the same number of two window sets, the window region partitioning uses a sliding window and sets the step size to be the size of the basic window, so that Sse with size n x a h x b w x g d.

[0148] Step 4.3: Window-based multi-head cross-attention mechanism

[0149] The multi-head cross-attention mechanism aims to calculate a new feature with a corresponding correlation degree between the input feature b and the feature s through the attention mechanism. After the basic window and the search window are generated in step 4.2, the features b and s are used for window-based attention calculation. Each basic window S ba is linearly projected and layer-normalized to the query set, and each search window S se is linearly projected and layer-normalized to the key and value sets, and then the window-based multi-head cross-attention algorithm is used to calculate the cross-attention degree between the two windows, as shown in Figure 4 The calculation formula is as follows:

[0150]

[0151] where Q ba , K se and V se represent the query, key and value matrices respectively.

[0152] Finally, the new feature output set is sent to a two-layer MLP with a Gelu nonlinear mapping after passing through a LayerNorm (LN) layer to enhance the learning ability.

[0153] Step 4.4: Parallel double U-shaped network based on cross-attention mechanism

[0154] As a registration network, the parallel double U-shaped network based on the cross-attention mechanism uses a double U-shaped network to extract the features of the moving image and the fixed image respectively, and connects them through a cross-attention feature fusion module, as shown in Figure 2 The network input size is 128 3The floating data b and the standard data s are input into the network, and two features b' and s' of the same size are output after feature fusion. The two parallel networks adopt the structure of U-NET in the encoding and decoding parts, down-sample the features in the encoder, up-sample the features in the decoder, and realize the purpose of constantly refining the features by jump connecting the encoder and the decoder. The window-based multi-head attention mechanism of step 4.3 is used to replace the convolution, so that the registration network can exchange cross-image information. Finally, a classification head is added at the end of the network, which is realized by two consecutive multi-layer perceptron (MLP) layers and uses the hyperbolic tangent (Tanh) activation function. The two features b' and s' are concatenated and averaged in the channel dimension, and then input into the classification head. After linear mapping and activation function, six rigid transformation parameters are finally output. Finally, the rigid transformation parameters and the floating CT data are subjected to rigid transformation to generate the corresponding standard CT data according to the affine_transform method in the ndimage library of the scipy toolkit.

[0155] In step S5, the loss function of the network is composed of the pixel and perceptual loss of the image domain.

[0156] The obtained estimated pose is combined with the original 3D CT to obtain a predicted 3D CT, and the error between the predicted 3D CT and the standard 3D CT is calculated. The error can be further divided into a pixel-level NCC loss and a structure-semantic-level perceptual loss. The pixel-level loss is used to measure the difference between the predicted image and the target image, and the optimization algorithm is optimized in such a way that the predicted image is closer to the target image:

[0157]

[0158] where x represents the original image, x * represents the predicted image, Cov(·) represents the covariance of the two images, and Var(·) represents the variance of the image itself.

[0159] The opposite of pixel-level loss is structure-semantic level perceptual loss. This loss function is often used in image denoising field, and its purpose is to alleviate the problem of image over-smoothing caused by pixel-level loss. Since the introduction of perceptual loss can further improve the structural similarity and perceptual similarity between the predicted image and the target image. Because it is more in line with human intuition, it has been used as an auxiliary or even a replacement for pixel-level loss in recent years. In the rigid spine registration scenario, the main concern of the physician will focus on the spine to be registered, so it is also an ideal application scenario for perceptual loss. Traditional perceptual loss uses vgg16 as the backbone network for feature extraction. However, vgg16 is trained using regular images, and its feature extraction weights are more suitable for general fields, but when used in the medical field, it will produce a certain deviation. In contrast, the encoder in the 3D feature extraction network is set as the feature extraction architecture, and the hyperparameters that have been trained are directly migrated. Since this architecture is originally designed for 2D spine image 3D reconstruction, it can better extract the features of the spine, and a result much better than vgg16 is observed in a specific application experiment. When calculating the architecture-perceptual loss, the predicted image and the target image are respectively sent into the frozen encoder for forward propagation. Specifically, the L2 loss is used to calculate the perceptual similarity of the feature maps obtained from the two images, and the specific formula is as follows:

[0160] Perceptual loss (formula)

[0161]

[0162] Representative feature extraction.

[0163] According to the different focuses of the two different LOSS and the experiment process, mutual information loss is usually used to measure the information correlation degree between images, while perceptual loss is usually used to measure the similarity of image quality or content. Perceptual loss is usually related to high-level feature representation of neural networks, which is more sensitive to the content and structure of the image. Mutual information loss can help the network better handle deformation and other situations, because it focuses on the alignment of information. Combining them can consider the image alignment or the information retention and visual quality of the generated image, which helps to complete more accurate registration. By combining the two losses, the robustness of the algorithm can be improved, and it can perform better in various situations. In summary, the specific application example proposes an adaptive multi-dimensional loss function, and the specific formula is as follows:

[0164] L total =L NCC +μL perceptual

[0165] Wherein mu is a sigmoid function, which will adaptively allocate the weight of NCC Loss and perceptual Loss according to the set total number of network training rounds, so that the network initially emphasizes alignment error and then emphasizes image visual quality.

[0166] In summary, the above method, the present application input mobile position 2D-C type arm X-ray, output from the mobile position to the standard position of rigid transformation matrix, further can obtain standard position 3D-CT, again through projection can obtain standard position 2D-C type arm X-ray, thereby realizing two-dimensional C type arm X-ray and three-dimensional CT image data direct rigid registration, such as Figure 5 As shown in (a)-(f), effectively assist doctors to obtain more accurate detection effect, improve work efficiency.

[0167] An embodiment of the present application provides a spine medical image registration system based on semantic reconstruction, as shown in the figure, the system can include the following modules: Figure 6

[0168] The data processing module is used for constructing spine 3D-CT image data and its corresponding 2D image data as a training data set.

[0169] The 2D-3D reconstruction module is used for providing a 2D-3D reconstruction model, and reconstructing the 2D image data by using the 2D-3D reconstruction model to obtain the 3D feature map b corresponding to the 2D image data.

[0170] The 3D-3D feature extraction module is used for providing a 3D-3D feature extraction model, and taking the 3D-CT image data as the input of the 3D-3D feature extraction model to obtain the feature map s corresponding to the 3D-CT image data with the same dimension as the 3D feature map.

[0171] The registration module is used for providing a parallel double U-shaped network model based on cross attention, and fusing and registering the feature map b and the feature map s by using the parallel double U-shaped network model; a multi-weight loss function based on pixel and semantic information is used to optimize the parallel double U-shaped network model, and the optimized parallel double U-shaped network model is used to output the optimal registration result of the spine medical image.

[0172] In some preferred embodiments:

[0173] The data processing module is used for generating and preprocessing the 3D-CT image and the corresponding 2D data.

[0174] The 2D-3D reconstruction module is used for reconstructing the 2D image to obtain the 3D feature map by using the 2D-3D reconstruction network.

[0175] ​3D-3D feature extraction module: using 3D-3D feature extraction network to process 3D image to get the same dimension feature map;

[0176] Registration module: using parallel double U-shaped network based on cross attention mechanism to fuse two sets of feature maps for registration; using multi-weight loss function based on pixel and semantic information to further guide the network to get the optimal registration result.

[0177] The data processing module specifically includes the following contents: selecting the spine CT data set to be uniformly saved in NifTI format, center cropping, generating corresponding floating CT data according to different parameters and original CT data, and using DeepDRR tool to perform DRR projection on the randomly transformed CT data to generate simulated C-arm-X-ray data, and performing DRR projection on the original CT data to generate a standard registration reference image, and finally performing bit inversion of the image pixel value and adaptive equalization of the image for two-step preprocessing operations.

[0178] The 2D-3D reconstruction module specifically includes the following contents: for feature extraction of 2D X-ray images, a single-angle 2D X-ray projection is used to reconstruct a 3D CT volume image. The volume reconstruction network can be specifically divided into a representative network and a generation network. The representative network extracts multi-scale features layer by layer, and converts high-dimensional data into embedded representations, while the generation network is composed of 3D deconvolution blocks, which reconstruct high-dimensional images from the feature information obtained from the conversion layer connecting the two network parts, i.e. the projection corresponding volume image. This network can learn how to generate 3D images from 2D projections. In the training stage, the artificial segmentation mask corresponding to the reconstructed volume is input at the same time to obtain the significant region mask in the X-ray reconstructed image data.

[0179] The 3D-3D feature extraction module specifically includes the following details: for feature extraction of three-dimensional CT images, a 3DRes-NET network is used. In the training process, the artificial segmentation spine mask is simultaneously input as the network input to obtain the feature information and the significant region mask of the three-dimensional image. Since the input of the registration network is set to 3D information, the 3D image generated by the CT data does not need to be upscaled or downscaled, but only needs to be processed by feature extraction to obtain the same dimension feature map as the 2D reconstruction network.

[0180] The registration module is specifically divided into the following submodules:

[0181] Feature map extraction submodule: this submodule combines the intermediate same dimension feature maps of 2D X-ray and 3D CT images through 2D reconstruction network and 3D feature extraction network respectively, together with standard size 512 3 image data, and center cropping, and placing in the following registration network;

[0182] Window partitioning submodule: the submodule is used for window partitioning and window region partitioning on a feature map;

[0183] Cross-attention mechanism construction submodule: the submodule is based on a multi-head cross-attention mechanism of a window;

[0184] Registration network submodule: the submodule is based on a parallel double U-shaped network of a cross-attention mechanism;

[0185] Optimization submodule: the submodule is used for optimizing the parallel double U-shaped network; wherein, a loss function for measuring the network is composed of pixel loss in an image domain and perceptual loss. The pixel loss is used as a measure of the difference between a predicted image and a target image, and the perceptual loss alleviates the problem of image oversmoothing caused by the pixel loss, and can further improve the structural similarity and perceptual similarity between the predicted image and the target image.

[0186] It should be noted that the steps in the method provided by the present application can be realized by corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solutions of the method to realize the composition of the system, i.e., the embodiments in the method can be understood as preferred examples of constructing the system, which will not be described here.

[0187] An embodiment of the present application provides a computer terminal, including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to execute the method of any one of the above-mentioned embodiments of the present application, or run the system of any one of the above-mentioned embodiments of the present application.

[0188] Optionally, the memory is used for storing programs; the memory can include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviation: RAM), such as static random access memory (English: static random-access memory, abbreviation: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviation: DDR SDRAM) and the like; the memory can also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used for storing computer programs (such as application programs, functional modules and the like for realizing the above-mentioned method), computer instructions and the like, and the above-mentioned computer programs, computer instructions and the like can be stored in one or more memories. And the above-mentioned computer programs, computer instructions, data and the like can be called by the processor.

[0189] The computer program, computer instruction, and the like described above can be stored in one or more memories in a partitioned manner. And the computer program, computer instruction, data, and the like described above can be called by the processor.

[0190] The processor is used to execute the computer program stored in the memory to realize each step in the method or each module of the system involved in the above-mentioned embodiments. For details, please refer to the related description in the above method and system embodiments.

[0191] The processor and the memory can be an independent structure, or an integrated structure. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0192] An embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to execute the method in any one of the above-mentioned embodiments of the present application or run the system in any one of the above-mentioned embodiments of the present application.

[0193] The spinal column registration method, system, terminal and medium based on semantic reconstruction provided by the above-mentioned embodiments of the present application comprehensively utilize deep learning network technology, avoid information and precision loss caused by dimension reduction by reconstructing 2D projection into 3D feature map, and improve the network's 2D / 3D information fusion capability by adopting a double U-shaped network based on cross attention mechanism, and further improve the attention weight of the key semantic region in the network by using a multi-weight loss function. The spinal column registration method, system, terminal and medium based on semantic reconstruction provided by the above-mentioned embodiments of the present application can reduce the number of image shooting times while realizing accurate spinal column image registration.

[0194] Those skilled in the art know that, in addition to implementing the system and each device thereof provided by the present application in the form of pure computer readable program code, the same function can also be realized by logically programming the method steps to make the system and each device thereof provided by the present application in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like. Therefore, the system and each device thereof provided by the present application can be considered as a hardware component, and the devices included therein for realizing various functions can also be considered as structures in the hardware component; the devices for realizing various functions can also be considered as both software modules for realizing methods and structures in the hardware component.

[0195] The details not described in the above-mentioned embodiments of the present application are all known in the art.

[0196] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which do not affect the essence of the present application.

Claims

1. A method for registering spinal medical images based on semantic reconstruction, the method comprising: The method comprises the following steps: constructing 3D-CT image data of a spine and corresponding 2D image data thereof as a training data set; providing a 2D-3D reconstruction model, reconstructing the 2D image data by using the 2D-3D reconstruction model to obtain a 3D feature map b corresponding to the 2D image data; providing a 3D-3D feature extraction model, taking the 3D-CT image data as an input of the 3D-3D feature extraction model to obtain a feature map s corresponding to the 3D-CT image data with the same dimension as the 3D feature map; providing a parallel double-U-shaped network model based on multi-head cross attention, and fusing and registering the feature map b and the feature map s by using the parallel double-U-shaped network model; adopting an inter-pixel loss to measure a difference between a predicted image and a target image, adopting a perception loss to improve structural similarity and perceptual similarity of the predicted image and the target image, and constructing a multi-weight loss function based on pixel and semantic information to optimize the parallel double-U-shaped network model; and obtaining a registration model through the above steps.

2. The method of claim 1, wherein, The method of constructing 3D-CT image data of a spine and corresponding 2D image data thereof comprises the following steps: obtaining spine CT data, unifying data formats of the spine CT data, and performing center cropping preprocessing to obtain spine CT preprocessing data; rigid body transformation is performed on the spine CT pre-processing data to generate floating CT data; wherein the rigid body transformation is represented by six parameters, the components of translation and rotation are respectively translation displacement components (t x ,t y ,t z ) in three axial directions and rotation components (r x ,r y ,r z ) around three axial directions; each parameter is randomly generated in a set range to form different combinations of 6 DOF parameters; then the DOF parameters are converted into a rigid body transformation matrix, and finally the corresponding floating CT data is generated according to the DOF parameters and the spine CT pre-processing data; performing DRR projection on the floating CT data, and performing image pixel value bit inversion and adaptive equalization processing to obtain final simulated C-arm-X-ray data; finally, performing DRR projection on the spine CT preprocessing data not subjected to rigid body transformation to generate standard registration reference 2D image data.

3. The method of claim 1, wherein the method further comprises: The method of providing a 2D-3D reconstruction model and reconstructing the 2D image data by using the 2D-3D reconstruction model to obtain a 3D feature map b corresponding to the 2D image data comprises the following steps: constructing a 2D-3D reconstruction model, taking the 2D image data and an artificially segmented spine mask as inputs of the 2D-3D reconstruction model, and projecting and reconstructing a 3D-CT volume image by using a single-angle 2D image data through the 2D-3D reconstruction model to obtain the 3D feature map b; wherein: the 2D-3D reconstruction model comprises a representative network, a generation network, and a conversion layer connected between the representative network and the generation network; wherein: the representative network is used for layer by layer extraction of multi-scale features of the 2D image data, conversion of high-dimensional data into embedded representations, and obtaining semantic information of a hidden 3-dimensional structure in the input 2D image data; the conversion layer is used for learning a manifold mapping function corresponding to the extracted multi-scale features, so that the extracted multi-scale features are across latitude; the generation network mainly comprises a 3D deconvolution block, and is used for reconstructing a high-dimensional image from feature information obtained from the conversion layer, i.e., a 3D volume image corresponding to the 2D image data projection.

4. The method of claim 1, wherein, The 3D-3D feature extraction model is provided, the 3D-CT image data is taken as the input of the 3D-3D feature extraction model, and a feature map s corresponding to the 3D-CT image data is obtained, including: The 3D Res-NET network is adopted to construct the 3D-3D feature extraction model; The 3D-CT image data and the manually segmented spine mask are taken as the input of the 3D-3D feature extraction model, the 3D-CT image data is down-sampled, and then, under the action of convolution and deconvolution, the key information of the spine in the 3D-CT image data is extracted and the feature map s is output.

5. The method of claim 1, wherein, The multi-head cross attention mechanism is taken as the convolution layer of the parallel double U-shaped network, a parallel double U-shaped network model based on the multi-head cross attention is provided, and the feature map b and the feature map s are fused and registered by using the parallel double U-shaped network model, including: The 3D feature map b and the feature map s are combined with the 2D image data and the 3D-CT image data respectively, and a key region feature map is formed; The key region feature map is divided into a window partition and a window region partition; A multi-head cross attention mechanism based on a window is constructed, which is used to calculate new features between the input key region feature maps with corresponding correlation degrees; A parallel double U-shaped network is provided, the multi-head cross attention mechanism is taken as the convolution layer of the parallel double U-shaped network, and a parallel double U-shaped network model is constructed, which is used to output the registration result of the spine medical image.

6. The method of claim 1-5, wherein, Further comprising: The registration model is used to process the to-be-registered spine medical image, and the registration result of the spine medical image is output.

7. A system for registering spinal medical images based on semantic reconstruction, the system comprising: Including: A data processing module is used to construct the spine 3D-CT image data and the corresponding 2D image data as the training data set; A 2D-3D reconstruction module is used to provide a 2D-3D reconstruction model, and the 2D image data is reconstructed by using the 2D-3D reconstruction model to obtain the 3D feature map b corresponding to the 2D image data; A 3D-3D feature extraction module is used to provide a 3D-3D feature extraction model, and the 3D-CT image data is taken as the input of the 3D-3D feature extraction model to obtain the feature map s corresponding to the 3D-CT image data with the same dimension as the 3D feature map; A registration module is used to take the multi-head cross attention mechanism as the convolution layer of the parallel double U-shaped network, provide a parallel double U-shaped network model based on the multi-head cross attention, and fuse and register the feature map b and the feature map s by using the parallel double U-shaped network model; the inter-pixel loss is used to measure the difference between the predicted image and the target image, the perception loss is used to improve the structural similarity and perception similarity between the predicted image and the target image, a multi-weight loss function based on the pixel and semantic information is constructed to optimize the parallel double U-shaped network model, and the optimal registration result of the spine medical image is output.

8. A computer terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program and can be used to execute the method in any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program can be executed by a processor to perform the method of any one of claims 1-6.

Images