A method and system for real-time registration of CT and X-ray based on ViT

Abstract

The present invention discloses a ViT-based real-time CT and X-ray registration method and system, which relates to the technical field of medical image processing. Obtain a patient's preoperative CT image; apply random perturbations to the standard camera pose matrix used for shooting X-rays and project to generate a DRR image; convert the DRR image into an X-ray style, and use a pre-trained ViT encoder to extract image features, calculate the cross-attention vector with the embedded features of the standard camera pose matrix, and predict the camera pose matrix; use the prediction results to reproject the CT image and then convert it into an X-ray style; fine-tune the pre-trained ViT encoder by combining the camera pose matrix loss and the X-ray style DRR image loss; use the fine-tuned pre-trained ViT encoder to obtain the initial camera pose matrix for registering intraoperative X-rays and preoperative CT. The present invention performs self-supervised learning on CT images obtained before surgery for specific patients, and can quickly initialize and register X-ray images generated during surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a ViT-based CT and X-ray real-time registration method and system. Background Art

[0002] In image-guided puncture surgery, the C-arm X-ray machine can illuminate the patient's lesion in real time, guide the doctor to locate the lesion and perform the surgery, but the X-ray image is 2D, and the imaging information it provides is fuzzy. In addition, the overlapping of internal tissues and organs in the human body makes it difficult for doctors to directly locate the lesion on the X-ray with the naked eye, and it is also difficult to avoid key tissues and organs during the puncture surgery, which increases the difficulty of the surgery. A common practice is to take a CT image of the patient before surgery. The CT image is a 3D image reconstructed after a tomographic scan, and its tissue structure is clear. The doctor can plan the needle insertion path based on the CT image before the surgery. During the surgery, the X-ray taken by the intraoperative C-arm is registered with the preoperative CT, and the intraoperative X-ray image is enhanced to help the doctor perform the surgery.

[0003] Traditional registration methods are usually based on optimization, which relies on the selection of initial points. Poor initial values ​​of the camera pose matrix used for registration will cause the registration time to be too long and may fall into a local optimum. With the development of computers, some deep learning-based methods have been proposed. Some methods use an end-to-end method for training. This method does not require intraoperative optimization and is very fast, but has low accuracy. Another method is to use deep learning to obtain the initial values ​​of the registration, and then use traditional methods to optimize during surgery. This method has relatively high accuracy, but the initial values ​​obtained by deep learning are currently difficult to achieve sufficient accuracy, and most of them require a large number of annotated data sets, which are difficult to obtain, and the results of generalized training are difficult to apply to specific patients. Large initialization errors may occur, resulting in registration failure. Summary of the invention

[0004] In view of the above problems, the present invention proposes a ViT-based CT and X-ray real-time registration method and system for achieving more accurate initial registration. The present invention constructs a data set for a specific patient and performs training, thereby improving the accuracy and robustness of obtaining the initial registration value.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides a ViT-based CT and X-ray real-time registration method, comprising:

[0007] Obtain the patient's preoperative CT images;

[0008] A random perturbation matrix is ​​applied to the standard camera pose matrix used for X-ray shooting to obtain the real camera pose matrix at different angles, and the CT image is projected according to the real camera pose matrix at different angles to generate a DRR image; the generated DRR image is converted into an X-ray style using a pre-trained style transfer network to obtain a real DRR image in the X-ray style;

[0009] The pre-trained ViT encoder is used to extract image features of a real DRR image in an X-ray style, a cross-attention vector is calculated by using the extracted image features and the embedded features of the standard camera pose matrix, a perturbation matrix is ​​predicted according to the cross-attention vector, and a predicted camera pose matrix is ​​obtained according to the predicted perturbation matrix and the standard camera pose matrix;

[0010] The CT image is reprojected using the predicted camera pose matrix to obtain a predicted DRR image, and then the predicted DRR image is converted into an X-ray style to obtain an X-ray style predicted DRR image;

[0011] The loss between the X-ray style predicted DRR image and the X-ray style real DRR image, as well as the loss between the real camera pose matrix and the predicted camera pose matrix are jointly used to fine-tune the pre-trained ViT encoder; the initial camera pose matrix for aligning intraoperative X-rays and preoperative CT is obtained using the fine-tuned pre-trained ViT encoder.

[0012] Furthermore, the pre-trained ViT encoder is fine-tuned using the LoRA approach.

[0013] Furthermore, the standard camera pose matrix is ​​a camera pose matrix that takes the center point of the patient as the world coordinate system and faces the front of the patient's chest.

[0014] Furthermore, the real camera pose matrices at different angles are obtained by right-multiplying different random perturbation matrices by the standard camera pose matrix.

[0015] Furthermore, the pre-trained style transfer network adopts a pre-trained CUT style transfer network.

[0016] Furthermore, when calculating the cross-attention vector between the extracted image features and the embedded features of the standard camera pose matrix, the embedded features of the standard camera pose matrix are used as the query vector, and the image features of the real DRR image in the X-ray style extracted by the pre-trained ViT encoder are used as the key vector.

[0017] Furthermore, the cross attention vector is used to predict the perturbation matrix using a multi-layer perceptron.

[0018] Furthermore, the predicted camera pose matrix is ​​obtained by right-multiplying the predicted perturbation matrix by the standard camera pose matrix.

[0019] Furthermore, the method of using the fine-tuned pre-trained ViT encoder to obtain an initial camera pose matrix for registering intraoperative X-rays and preoperative CT scans includes:

[0020] The intraoperative X-ray images are directly input into the fine-tuned pre-trained ViT encoder to extract image features;

[0021] The embedded features of the standard camera pose matrix are used as the query vector, and the image features of the X-ray image are used as the key vector to calculate the cross attention and obtain the cross attention vector;

[0022] Predict the perturbation matrix based on the obtained cross-attention vector;

[0023] The predicted perturbation matrix is ​​multiplied by the standard camera pose matrix to obtain the initial camera pose matrix for registration of intraoperative X-ray and preoperative CT.

[0024] In a second aspect, the present invention provides a ViT-based CT and X-ray real-time registration system for implementing the above-mentioned ViT-based CT and X-ray real-time registration method.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1) The present invention constructs a data set by randomly sampling the perturbation matrix based on the CT images taken for a specific patient, which is more in line with the actual situation during surgery. It also uses an unsupervised training method, which greatly reduces the dependence on other data sets. The patient-specific design enables a better registration effect to be achieved for the patient during surgery.

[0027] 2) The present invention uses a pre-trained style transfer network to transfer the style of DRR images to obtain DRR images with X-ray style, making them closer to the real X-ray images during the operation, so as to achieve closer effects in the training and reasoning stages.

[0028] 3) The present invention uses a pre-trained ViT encoder and performs fine-tuning, which can greatly reduce the number of parameters required for training and effectively utilize the pre-training information; by calculating the cross-attention vector by combining the extracted image features with the embedded features of the standard camera pose matrix, the information of the camera pose matrix can be better predicted.

[0029] 4) When fine-tuning the pre-trained ViT encoder, the present invention combines the loss between the X-ray style predicted DRR image and the X-ray style real DRR image, as well as the loss between the real camera pose matrix and the predicted camera pose matrix, which can better utilize the global information and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall structure diagram of a ViT-based CT and X-ray real-time registration method shown in an embodiment of the present invention;

[0031] Figure 2 This is a structural diagram of a CUT style transfer network shown in an embodiment of the present invention;

[0032] Figure 3 A structural diagram of a classic visual Transformer (ViT) block shown in an embodiment of the present invention;

[0033] Figure 4 A traditional registration framework is shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention.

[0035] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0037] In view of the problems existing in existing image-guided puncture surgery, the present invention discloses a real-time CT and X-ray registration method based on ViT, which is used to obtain the initial value of global registration. The present invention first pre-trains a network for digitally reconstructed radiograph (DRR) to X-ray style transfer, and then targets the preoperative CT image of a specific patient by using the standard camera pose matrix Apply perturbation, and the perturbation result is used as the real camera pose matrix , project the CT image using the real camera pose matrix to obtain DRR images at different angles, and convert the obtained DRR image into an X-ray style using the pre-trained style transfer network to obtain an X-ray style DRR image, thereby constructing a data-label pair for training the ViT model. Use the pre-trained ViT model to extract the image features of the X-ray style DRR image and compare it with the standard camera pose matrix The generated embedded features calculate cross attention, use multi-layer perceptron (MLP) to predict the perturbation matrix, and further calculate the predicted camera pose matrix , based on the predicted camera pose matrix, the CT image is reprojected and X-ray style conversion is performed. The camera pose matrix loss and X-ray style DRR image loss are combined, and the pre-trained ViT model is fine-tuned using the LoRA method. The camera pose matrix used for real-time alignment of CT and X-ray is predicted based on the fine-tuned ViT model to achieve more accurate initialization alignment.

[0038] The present invention proposes a ViT-based real-time CT and X-ray registration method, which mainly includes six steps: CT image acquisition, DRR dataset generation, DRR image style transfer, model construction, model training and model reasoning. Figure 1 As shown, specifically:

[0039] S1. Obtain the patient's preoperative CT image;

[0040] S2. Apply a random perturbation matrix to the standard camera pose matrix used for shooting X-rays to obtain real camera pose matrices at different angles, project the CT images according to the real camera pose matrices at different angles to generate DRR images, and construct a DRR dataset;

[0041] S3, using the pre-trained style transfer network, convert the DRR image generated in S2 into an X-ray style to obtain a real DRR image in the X-ray style;

[0042] S4. Use the pre-trained ViT encoder to extract the image features of the real DRR image in the X-ray style, calculate the cross attention between the extracted image features and the embedded features of the standard camera pose matrix, predict the perturbation matrix based on the cross attention results, and obtain the predicted camera pose matrix based on the predicted perturbation matrix and the standard camera pose matrix. ;

[0043] S5. Reproject the CT image using the predicted camera pose matrix to obtain a predicted DRR image; then convert the predicted DRR image into an X-ray style to obtain an X-ray style predicted DRR image;

[0044] S6, the loss between the predicted DRR image in X-ray style and the real DRR image in X-ray style, as well as the loss between the real camera pose matrix and the predicted camera pose matrix, are combined to fine-tune the pre-trained ViT encoder using the LoRA method;

[0045] S7. Use the fine-tuned pre-trained ViT encoder to obtain the initial camera pose matrix for aligning intraoperative X-rays and preoperative CT.

[0046] The key steps are described in detail below.

[0047] In the above step S2, a DRR dataset for training is generated by taking CT projections of a specific patient before surgery, without any labeling information, which greatly reduces the difficulty of obtaining the dataset.

[0048] In a specific implementation of the present invention, the optional implementation process is as follows:

[0049] S21. For a CT image of a patient before surgery, the camera pose matrix facing the front of the chest with the patient's center point as the world coordinate system is called the standard camera pose matrix , apply a random perturbation matrix to the standard camera pose matrix , the perturbation matrix Right multiply the standard camera pose matrix to get the real camera pose matrix at different angles , the perturbed real camera pose matrix is ​​used to simulate the real camera pose for shooting X-rays;

[0050] S22, according to the real camera pose matrix at different angles The preoperative CT images are projected according to the real camera pose matrix at different angles using the Ray-Cast method accelerated by the graphics card to generate DRR images, which are used to simulate the X-ray images generated in real time during the operation.

[0051] S23, use the DRR image obtained in S22 as sample data, and the real camera pose matrix after perturbation obtained in S21 As labels, the DRR dataset is constructed.

[0052] In the above step S3, since there is usually a large style difference between the DRR image generated by CT image projection and the actual X-ray image, in order to enable the ViT encoder to be directly applied to the X-ray images in the actual surgical process and reduce the impact of this gap on the reasoning stage, the present invention does not use the DRR image generated in S2 directly as training data, but uses a pre-trained style transfer network to perform style conversion on the generated DRR image.

[0053] In a specific implementation of the present invention, the optional implementation process is as follows:

[0054] For the DRR images generated in S2, the pre-trained CUT style transfer network is used to convert the simulated DRR images into X-ray style;

[0055] The structural diagram of the pre-trained CUT style transfer network is as follows: Figure 2As shown in Figure 2, the network does not require paired data and labels, only some DRR images and some real X-ray images. The network can perform unsupervised contrastive learning through the generator and the discriminator, where the generator The DRR image generated by projection is used as input, and the input image is encoded and decoded to obtain an X-ray style DRR image. Different image blocks are taken from the original DRR image and the generated X-ray style DRR image, and the contrast loss between the multi-layer image features is calculated through contrast learning. The parameters of the generator network are trained based on this. The discriminator is used to judge the real X-ray image and the generated X-ray style DRR image to obtain a classification to determine whether it is real or not.

[0056] This gives us the training data (X-ray style DRR images) and labels (real camera pose matrices). )right.

[0057] In the above step S4, the ViT model performs well in the image field. The pre-trained ViT model is selected as the encoder for the X-ray style DRR image. The input of the ViT encoder is the X-ray style DRR image, and the output is the image features of the X-ray style DRR image. Here, the cross-attention mechanism is used to calculate the cross-attention of the image features of the X-ray style DRR image and the embedded features of the standard camera pose matrix and predict the perturbation matrix. Then, the predicted camera pose matrix is ​​obtained based on the predicted perturbation matrix and the standard camera pose matrix. .

[0058] In a specific implementation of the present invention, the optional implementation process is as follows:

[0059] S41. For X-ray style DRR images, use the pre-trained ViT encoder as the backbone network to extract image features;

[0060] S42, using the embedded features of the standard camera pose matrix after Fourier position encoding as the query vector, and the image features obtained in S41 as the key vector, calculating the cross attention, and obtaining the cross attention vector;

[0061] S43, pass the cross attention vector obtained in S42 through the multi-layer perceptron MLP to predict the disturbance matrix , multiply the predicted perturbation matrix by the standard camera pose matrix to get the predicted camera pose matrix ,Right now .

[0062] In the above step S6, since the ViT encoder contains a large number of parameters, full fine-tuning requires a lot of resources and time. Therefore, the present invention uses the LoRA method to fine-tune the pre-trained ViT encoder. LoRA is based on the premise of limited information, fixes the pre-trained ViT model parameters, and uses two low-rank matrices to obtain a matrix with the same structure as the ViT parameters, thereby greatly reducing the parameters of model training. The network structure of a classic ViT block is as follows Figure 3 As shown in the figure, the network input is the encoded image feature vector. After layer normalization and multi-head self-attention, the internal features of the image can be fully extracted and fused. Here, a residual connection is made with the input feature vector, and then layer normalization is performed again. After passing through a multi-layer perceptron, a residual connection is made again, which further enhances the expressive power of the model.

[0063] In a specific implementation of the present invention, the pre-trained ViT encoder is fine-tuned in an end-to-end manner, and the overall expression of the loss function is:

[0064]

[0065] in, represents the error between the predicted camera pose matrix and the actual camera pose matrix, which is calculated as , The function is a logarithmic mapping from Lie group to Lie algebra. This equation is used to describe the camera pose matrix arrive The difference The normalized cross-correlation between the losses between the predicted DRR image of X-ray style and the real DRR image of X-ray style is represented. This embodiment uses multi-scale normalized cross-correlation to obtain a globally robust result.

[0066] Update the pre-trained ViT encoder and multi-layer perceptron MLP parameters according to the calculated loss, and optimize the network parameters until the network converges.

[0067] In the above step S7, the fine-tuned pre-trained ViT encoder is used to obtain an initial camera pose matrix for registering intraoperative X-rays and preoperative CT scans. The initial camera pose matrix can be further corrected using an optimization method.

[0068] In a specific implementation of the present invention, the optional implementation process is as follows:

[0069] S71, for a specific patient's intraoperative X-ray image, directly input it into the fine-tuned pre-trained ViT encoder to extract image features;

[0070] S72. Use the embedded features of the standard camera pose matrix after Fourier position encoding as the query vector, and the image features obtained in S71 as the key vector, calculate the cross attention, and obtain the cross attention vector.

[0071] S73, pass the cross attention vector obtained in S72 through the multi-layer perceptron MLP to predict the disturbance matrix , the predicted perturbation matrix is ​​multiplied by the standard camera pose matrix to obtain the initial camera pose matrix for registration of intraoperative X-ray and preoperative CT.

[0072] S74: Using the initial camera pose matrix obtained in S73 as the initial value of the traditional optimization algorithm, further intraoperative optimization is performed to obtain the final camera pose matrix .

[0073] like Figure 4 The figure shows a traditional optimization algorithm. The method proposed in the present invention can be used to calculate an initial value (initial parameter) and input it into the traditional registration framework. The preoperative CT image is projected using the initial value to obtain a DRR image, and the similarity is calculated with the intraoperative X-ray image. The error of the loss function is reduced through continuous optimization during the operation, thereby obtaining an accurate registration result.

[0074] In order to verify the technical effect of the present invention, the public dataset DeepFluoro is used for training and testing in this embodiment. The dataset contains CT images of 6 patients and contains 24 to 111 real X-ray projections, camera poses and annotation points.

[0075] Before the experiment, build Figure 1 The network model shown is deployed on the server. For each patient, only the preoperative CT image is used as the training data set, which is input into the constructed network model for self-supervised training. In this embodiment, PyTorch is used to build the network model framework, and the Adam optimizer is used for training. The learning rate is set to 0.001. The image resolution of the CUT style transfer network is uniformly adjusted to 224*224, and the rank of the LoRA fine-tuning low-rank matrix is ​​set to 16.

[0076] In order to verify the superiority of the present invention, a real data set was used for registration. The method proposed in the present invention and the method of randomly taking values ​​near the true value point were used to obtain the initial value of the registration. Then, the traditional method was used for registration. The experimental results can be seen in Table 1. In Table 1, the initial value of the registration is obtained by the method of the present invention before the “ / ”, and the initial value of the registration is obtained by randomly taking points near the true value after the “ / ”. Since the present invention uses the pre-trained ViT as the encoder, it can more effectively extract the features of the X-ray image, and use the LoRA method to fine-tune ViT, which greatly reduces the resources and time required for model training, and the features of the X-ray image and the standard camera pose matrix are adjusted. Cross-attention calculation can more effectively capture the information of the camera pose matrix. It can also be seen from the comparison results that the present invention is significantly superior to the comparison method.

[0077] Table 1 Experimental results

[0078]

[0079] The present invention also provides a ViT-based CT and X-ray real-time registration system, which is used to implement the above-mentioned embodiments. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0080] This embodiment provides a ViT-based CT and X-ray real-time registration system, including:

[0081] A patient data acquisition module, which is used to acquire the patient's preoperative CT image;

[0082] An image reprojection module, which applies a random perturbation matrix to a standard camera pose matrix for shooting X-rays to obtain real camera pose matrices at different angles, and projects the CT image according to the real camera pose matrix at different angles to generate a DRR image; and, is used to reproject the CT image using the predicted camera pose matrix to obtain a predicted DRR image;

[0083] An image style conversion module, which is used to convert the generated DRR image into an X-ray style using a pre-trained style transfer network to obtain a real DRR image in an X-ray style; and, used to convert the predicted DRR image into an X-ray style to obtain a predicted DRR image in an X-ray style;

[0084] A camera pose matrix prediction module, which is used to extract image features of real DRR images of X-ray style using a pre-trained ViT encoder in a training phase, calculate a cross-attention vector using the extracted image features and the embedded features of the standard camera pose matrix, predict a perturbation matrix based on the cross-attention vector, and obtain a predicted camera pose matrix based on the predicted perturbation matrix and the standard camera pose matrix; and obtain an initial camera pose matrix for registering intraoperative X-rays and preoperative CT using a fine-tuned pre-trained ViT encoder in a real-time registration phase;

[0085] A training module is used to fine-tune the pre-trained ViT encoder by jointly using the loss between the predicted X-ray-style DRR image and the real X-ray-style DRR image, as well as the loss between the real camera pose matrix and the predicted camera pose matrix.

[0086] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation method of the specific module will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0087] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0088] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A ViT-based CT and X-ray real-time registration method, characterized in that: include: Obtain the patient's preoperative CT images; A random perturbation matrix is ​​applied to the standard camera pose matrix used for shooting X-rays to obtain the real camera pose matrix at different angles, and the CT image is projected according to the real camera pose matrix at different angles to generate the DRR image; Use the pre-trained style transfer network to convert the generated DRR image into an X-ray style to obtain a real DRR image in X-ray style; The pre-trained ViT encoder is used to extract image features of a real DRR image in an X-ray style, a cross-attention vector is calculated by using the extracted image features and the embedded features of the standard camera pose matrix, a perturbation matrix is ​​predicted according to the cross-attention vector, and a predicted camera pose matrix is ​​obtained according to the predicted perturbation matrix and the standard camera pose matrix; The CT image is reprojected using the predicted camera pose matrix to obtain a predicted DRR image, and then the predicted DRR image is converted into an X-ray style to obtain an X-ray style predicted DRR image; Fine-tune the pre-trained ViT encoder by combining the loss between the predicted X-ray-style DRR image and the real X-ray-style DRR image, as well as the loss between the real camera pose matrix and the predicted camera pose matrix; The fine-tuned pre-trained ViT encoder is used to obtain the initial camera pose matrix for registration of intraoperative X-rays and preoperative CT scans, including: The intraoperative X-ray images are directly input into the fine-tuned pre-trained ViT encoder to extract image features; The embedded features of the standard camera pose matrix are used as the query vector, and the image features of the X-ray image are used as the key vector to calculate the cross attention and obtain the cross attention vector; Predict the perturbation matrix based on the obtained cross-attention vector; The predicted perturbation matrix is ​​multiplied by the standard camera pose matrix to obtain the initial camera pose matrix for registration of intraoperative X-ray and preoperative CT.

2. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: The pre-trained ViT encoder is fine-tuned using the LoRA approach.

3. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: The standard camera pose matrix is ​​a camera pose matrix that takes the center point of the patient as the world coordinate system and faces the front of the patient's chest.

4. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: The real camera pose matrices at different angles are obtained by right-multiplying different random perturbation matrices by the standard camera pose matrix.

5. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: The pre-trained style transfer network adopts a pre-trained CUT style transfer network.

6. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: When calculating the cross-attention vector between the extracted image features and the embedded features of the standard camera pose matrix, the embedded features of the standard camera pose matrix are used as the query vector, and the image features of the real DRR image in the X-ray style extracted by the pre-trained ViT encoder are used as the key vector.

7. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: The cross attention vector is used to predict the perturbation matrix using a multi-layer perceptron.

8. The ViT-based CT and X-ray real-time registration method according to claim 1, characterized in that: The predicted camera pose matrix is ​​obtained by right-multiplying the predicted perturbation matrix by the standard camera pose matrix.

9. A ViT-based CT and X-ray real-time registration system, characterized in that: include: A patient data acquisition module, which is used to acquire the patient's preoperative CT image; An image reprojection module applies a random perturbation matrix to the standard camera pose matrix used for shooting X-rays to obtain real camera pose matrices at different angles, and projects the CT image according to the real camera pose matrices at different angles to generate a DRR image; and, for reprojecting the CT image using the predicted camera pose matrix to obtain a predicted DRR image; An image style conversion module, which is used to convert the generated DRR image into an X-ray style using a pre-trained style transfer network to obtain a real DRR image in an X-ray style; and, used to convert the predicted DRR image into an X-ray style to obtain a predicted DRR image in an X-ray style; A camera pose matrix prediction module is used to extract image features of a real DRR image in an X-ray style using a pre-trained ViT encoder during a training phase, calculate a cross-attention vector using the extracted image features and the embedded features of the standard camera pose matrix, predict a perturbation matrix based on the cross-attention vector, and obtain a predicted camera pose matrix based on the predicted perturbation matrix and the standard camera pose matrix; And in the real-time registration stage, the fine-tuned pre-trained ViT encoder is used to obtain an initial camera pose matrix for registering the intraoperative X-ray and the preoperative CT, including: directly inputting the intraoperative X-ray image into the fine-tuned pre-trained ViT encoder to extract image features; using the embedded features of the standard camera pose matrix as a query vector and the image features of the X-ray image as a key vector, calculating the cross attention, and obtaining the cross attention vector; predicting the perturbation matrix according to the obtained cross attention vector; multiplying the predicted perturbation matrix by the standard camera pose matrix to obtain the initial camera pose matrix for registering the intraoperative X-ray and the preoperative CT; A training module is used to fine-tune the pre-trained ViT encoder by jointly using the loss between the predicted X-ray-style DRR image and the real X-ray-style DRR image, as well as the loss between the real camera pose matrix and the predicted camera pose matrix.

Images