Spine registration method, device, equipment, storage medium and program product

Through the cross-attention fusion mechanism, the digital reconstruction of radiographic images and X-ray images are fused, which solves the problem of low spinal registration efficiency in the existing technology, and achieves more efficient and high-precision spinal registration, thereby improving surgical navigation efficiency and accuracy.

CN119027467APending Publication Date: 2024-11-26MAESTRO SURGICAL ROBOTICS LTD CO
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411107862.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, the spinal registration efficiency of two-dimensional X-ray images and three-dimensional CT images is low, the iteration process takes a long time, and the single-vertebral image feature information is less, and the model is low robustness.

Method used

The digitally reconstructed radiographic image features and X-ray image features are fused by the cross-attention fusion mechanism. The spatial transformation parameters from three-dimensional CT images to two-dimensional X-ray images are obtained through the pre-trained registration model, which improves registration accuracy and efficiency .

Benefits of technology

It significantly improves the efficiency and accuracy of spinal registration and improves the efficiency and accuracy of surgical navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027467A_ABST
    Figure CN119027467A_ABST
Patent Text Reader

Abstract

The invention provides a spine registration method, device and equipment, a storage medium and a program product, and relates to the technical field of medical image processing, and the method comprises the steps: obtaining a digital reconstruction radiation image and a two-dimensional X-ray image containing a target vertebral body in a spine, and the digital reconstruction radiation image is obtained according to a three-dimensional CT image; and inputting the digital reconstruction radiation image and the two-dimensional X-ray image into a registration model, obtaining spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image based on the fusion features, and completing registration of the target vertebral body. The fusion feature is obtained by performing feature fusion on the extracted digital reconstruction radiation image feature and the X-ray image feature by adopting a cross attention fusion mechanism. According to the method, the spine registration efficiency can be greatly improved, the spine registration precision can be effectively improved, and thus the efficiency and accuracy of surgical navigation can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a spinal alignment method, device, equipment, storage medium and program product. Background Art

[0002] Surgical navigation technology, also known as image-guided surgery, is an advanced medical technology. In surgical navigation, the preoperative 3D computed tomography (CT) image is registered with the intraoperative 2D X-ray image based on image registration technology to obtain the spatial transformation parameters from the 3D CT image to the 2D X-ray image. Based on the spatial transformation parameters, the preoperative plan can be converted to the actual position of the patient during the operation, thereby guiding the surgical robot or assistant doctor to perform high-precision surgical operations.

[0003] At present, for the spinal registration of two-dimensional X-ray images and three-dimensional CT images, multiple optimization iterations are usually used to obtain spatial transformation parameters. Specifically, a three-dimensional CT image and a two-dimensional X-ray image containing the same vertebra are obtained, and according to the current projection posture of the three-dimensional CT image, a digitally reconstructed radiograph (DRR) method is used to project and generate a DRR image in the corresponding posture of the three-dimensional CT image; the similarity between the DRR image and the two-dimensional X-ray image is obtained, and based on the similarity, the next projection posture of the three-dimensional CT image is determined, and the above process is repeated until the similarity reaches the maximum value, and the spinal registration is completed. However, the spinal registration performed in the above manner has the problem of low registration efficiency. Summary of the invention

[0004] The present application provides a spine registration method, device, equipment, storage medium and program product to solve the problem of low registration efficiency in the current spine registration method.

[0005] In a first aspect, the present application provides a spine registration method, comprising:

[0006] acquiring a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional computed tomography (CT) image;

[0007] The digitally reconstructed radiographic image and the two-dimensional X-ray image are input into the registration model, and the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained based on the fusion features to complete the registration of the target vertebra. The fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and X-ray image features using a cross-attention fusion mechanism.

[0008] Optionally, the registration model includes a first visual converter VIT model, a second VIT model, a fusion module and a multi-layer perceptron. The digitally reconstructed radiographic image and the two-dimensional X-ray image are input into the registration model, and spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained based on the fusion features, including: inputting the digitally reconstructed radiographic image into the first VIT model for feature extraction to obtain digitally reconstructed radiographic image features; inputting the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features; performing feature fusion of the digitally reconstructed radiographic image features and the X-ray image features using a cross-attention fusion mechanism through the fusion module to obtain fused features; and performing regression processing on the fused features through the multi-layer perceptron to obtain spatial transformation parameters.

[0009] Optionally, the first VIT model includes a first image block division module and a first transformer encoder, and the digitally reconstructed radiological image is input into the first VIT model for feature extraction to obtain features of the digitally reconstructed radiological image, including: dividing the digitally reconstructed radiological image into image blocks of a preset size through the first image block division module to obtain a first image block vector sequence corresponding to the digitally reconstructed radiological image, the first image block vector sequence including an image block vector and an image block position encoding embedding vector corresponding to each first image block; inputting the first image block vector sequence into the first transformer encoder for feature extraction to obtain features of the digitally reconstructed radiological image.

[0010] Optionally, the second VIT model includes a second image block division module and a second Transformer encoder, and the two-dimensional X-ray image is input into the second VIT model for feature extraction to obtain X-ray image features, including: dividing the two-dimensional X-ray image into image blocks of a preset size by the second image block division module to obtain a second image block vector sequence corresponding to the two-dimensional X-ray image, the second image block vector sequence including an image block vector and an image block position encoding embedding vector corresponding to each second image block; inputting the second image block vector sequence into the second Transformer encoder for feature extraction to obtain X-ray image features.

[0011] Optionally, a cross-attention fusion mechanism is used to fuse the digitally reconstructed radiographic image features and the X-ray image features through a fusion module to obtain fused features, including: for a first sub-feature corresponding to each first image block in the digitally reconstructed radiographic image features, a query vector corresponding to the first sub-feature is generated through a first convolutional layer and a first weight; for a second sub-feature corresponding to each second image block in the X-ray image features, a key vector corresponding to the second sub-feature is generated through a second convolutional layer and a second weight, and a value vector corresponding to the second sub-feature is generated through a third convolutional layer and a third weight; based on the query vector, the key vector and the value vector, a target attention weight is obtained; a weighted summation process is performed on multiple target attention weights and the value vectors corresponding to the target attention weights to obtain a weighted sum feature; and the weighted sum feature and the digitally reconstructed radiographic image feature are feature-fused to obtain a fused feature.

[0012] Optionally, the registration model is obtained by training in the following manner: obtaining multiple training samples, the training samples including digitally reconstructed radiographic image samples including sample vertebrae in the sample spine, two-dimensional X-ray image samples and labels, the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are obtained based on the three-dimensional CT image samples, and the labels are used to represent the reference space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples; inputting the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples into the registration model, and obtaining the training space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples based on the fused feature samples, and the fused feature samples are obtained by feature fusion of the extracted digitally reconstructed radiographic image feature samples and the X-ray image feature samples using a cross-attention fusion mechanism; obtaining the loss function value based on the training space transformation parameters and the labels; and adjusting the model parameters of the registration model based on the loss function value to obtain a trained registration model.

[0013] In a second aspect, the present application provides a spinal alignment device, comprising:

[0014] an acquisition module, used for acquiring a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in a spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional computed tomography (CT) image;

[0015] The registration module is used to input the digitally reconstructed radiological image and the two-dimensional X-ray image into the registration model, obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image based on the fusion features, and complete the registration of the target vertebra. The fusion features are obtained by fusing the extracted digitally reconstructed radiological image features and X-ray image features using a cross-attention fusion mechanism.

[0016] Optionally, the registration model includes a first visual converter VIT model, a second VIT model, a fusion module and a multi-layer perceptron, and the registration module is specifically used to: input the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain digitally reconstructed radiological image features; input the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features; use the fusion module to perform feature fusion on the digitally reconstructed radiological image features and the X-ray image features using a cross-attention fusion mechanism to obtain fused features; and perform regression processing on the fused features through a multi-layer perceptron to obtain spatial transformation parameters.

[0017] Optionally, the first VIT model includes a first image block division module and a first transformer encoder. When the registration module is used to input the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain the features of the digitally reconstructed radiological image, it is specifically used to: divide the digitally reconstructed radiological image into image blocks of a preset size through the first image block division module to obtain a first image block vector sequence corresponding to the digitally reconstructed radiological image, the first image block vector sequence including an image block vector and an image block position encoding embedding vector corresponding to each first image block; input the first image block vector sequence into the first transformer encoder for feature extraction to obtain the features of the digitally reconstructed radiological image.

[0018] Optionally, the second VIT model includes a second image block division module and a second Transformer encoder, and when the registration module is used to input the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features, it is specifically used to: divide the two-dimensional X-ray image into image blocks of a preset size through the second image block division module to obtain a second image block vector sequence corresponding to the two-dimensional X-ray image, the second image block vector sequence including an image block vector and an image block position code embedding vector corresponding to each second image block; input the second image block vector sequence into the second Transformer encoder for feature extraction to obtain X-ray image features.

[0019] Optionally, when the registration module is used to perform feature fusion on the digitally reconstructed radiographic image features and the X-ray image features through the fusion module using a cross-attention fusion mechanism to obtain a fused feature, it is specifically used to: for a first sub-feature corresponding to each first image block in the digitally reconstructed radiographic image feature, generate a query vector corresponding to the first sub-feature through a first convolution layer and a first weight; for a second sub-feature corresponding to each second image block in the X-ray image feature, generate a key vector corresponding to the second sub-feature through a second convolution layer and a second weight, and generate a value vector corresponding to the second sub-feature through a third convolution layer and a third weight; based on the query vector, the key vector and the value vector, obtain a target attention weight; perform weighted sum processing on multiple target attention weights and the value vectors corresponding to the target attention weights to obtain a weighted sum feature; perform feature fusion on the weighted sum feature and the digitally reconstructed radiographic image feature to obtain a fused feature.

[0020] Optionally, the spinal registration device also includes a training module for training to obtain a registration model in the following manner: obtaining multiple training samples, the training samples including digitally reconstructed radiographic image samples, two-dimensional X-ray image samples and labels containing sample vertebrae in the sample spine, the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are obtained based on the three-dimensional CT image samples, and the labels are used to represent the reference space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples; inputting the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples into the registration model, and obtaining the training space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples based on the fused feature samples, and the fused feature samples are obtained by feature fusion of the extracted digitally reconstructed radiographic image feature samples and the X-ray image feature samples using a cross-attention fusion mechanism; obtaining the loss function value based on the training space transformation parameters and the labels; and adjusting the model parameters of the registration model based on the loss function value to obtain a trained registration model.

[0021] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0022] Memory stores computer-executable instructions;

[0023] The processor executes the computer-executable instructions stored in the memory to implement the spinal column registration method as described in the first aspect of the present application.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed, the spinal alignment method as described in the first aspect of the present application is implemented.

[0025] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the spinal alignment method as described in the first aspect of the present application when the computer program is executed.

[0026] The spinal registration method, apparatus, device, storage medium and program product provided in the present application obtain a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional CT image; the digitally reconstructed radiographic image and the two-dimensional X-ray image are input into a registration model, and the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained based on fusion features to complete the registration of the target vertebra, wherein the fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and the X-ray image features using a cross-attention fusion mechanism. In the present application, the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained by a registration model to complete the registration of the target vertebra, which can greatly improve the efficiency of spinal registration; at the same time, the fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and the X-ray image features using a cross-attention fusion mechanism, which can effectively improve the accuracy of spinal registration, thereby helping to improve the efficiency and accuracy of surgical navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 A flowchart of a spine registration method provided in one embodiment of the present application;

[0029] Figure 2 A flowchart of a spine registration method provided in another embodiment of the present application;

[0030] Figure 3 A schematic diagram of dividing a two-dimensional X-ray image into image blocks provided by an embodiment of the present application;

[0031] Figure 4 A schematic diagram of the structure of a Transformer encoder provided in one embodiment of the present application;

[0032] Figure 5 A flowchart of a method for training a registration model provided in one embodiment of the present application;

[0033] Figure 6 A schematic diagram of the structure of a spinal alignment device provided in one embodiment of the present application;

[0034] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0037] First, some technical terms involved in this application are explained:

[0038] Image registration is the process of aligning an image with another image in terms of image structure (such as organs, tissues, or lesions) through spatial transformation, such as rotation or translation. The spatial transformation parameters of image registration include rotation parameters and translation parameters. The rotation parameters include r x 、r y and r z , the translation parameters include t x ,t y and t z .

[0039] Digitally reconstructed radiography (DRR) is a method of converting a three-dimensional image (three-dimensional volume data) into a simulated radiographic image on a two-dimensional image plane by perspective projection. Through computer algorithms, the tissue structure and anatomical information in the three-dimensional image is projected onto a two-dimensional image to simulate the image produced by X-rays or other radiological imaging techniques. This technology is widely used in the field of medical imaging and can be used for surgical planning, imaging diagnosis, and treatment simulation, helping doctors better understand the patient's anatomical structure and make relevant medical decisions.

[0040] The Vision Transformer (VIT) model is a model that applies the Transformer to the field of Computer Vision (CV). The VIT model first divides the image into several image patches, flattens the patches, linearly maps the flattened patches to a lower-dimensional space, and adds position embedding encoding information to obtain an image patch vector sequence, which is then input into a standard Transformer encoder for feature extraction. Finally, the class token is regressed and classified through the classification head. At present, the research on Transformer in images is becoming more and more perfect, so it has great application prospects in medical image registration.

[0041] Cross-Attention is a mechanism for combining feature maps from different sources. Its principle is to enable the model to selectively focus on important information between different features by calculating and applying attention weights. It is used to fuse the information of two different feature maps together so that the output feature map can better capture the relationship and complementary information between the input feature maps.

[0042] Surgical navigation technology, also known as Image Guided Surgery, is an advanced medical technology that combines computer image processing technology with medical imaging technology to achieve preoperative organ or tissue segmentation of patient images, and to perform surgical planning and simulation. In surgical navigation, the preoperative three-dimensional CT image is registered with the intraoperative two-dimensional X-ray image based on image registration technology to obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image. Based on the spatial transformation parameters, the preoperative plan can be converted to the actual position of the patient during the operation, thereby guiding the surgical robot or assisting the doctor to perform high-precision surgical operations; doctors can use real-time image-guided surgery to better locate the lesion and protect the surrounding healthy tissues, and ensure the success and safety of the operation.

[0043] At present, for the registration of the spine (such as the lumbar spine, thoracic spine and cervical spine) with a two-dimensional X-ray image and a three-dimensional CT image, multiple optimization iterations are usually used to obtain spatial transformation parameters. Specifically, a three-dimensional CT image and a two-dimensional X-ray image containing the same vertebra are obtained, and according to the current projection posture of the three-dimensional CT image, a DRR method is used to project and generate a DRR image in the corresponding posture of the three-dimensional CT image; the similarity between the DRR image and the two-dimensional X-ray image is obtained, and based on the similarity, the next projection posture of the three-dimensional CT image is determined, and the above process is repeated until the similarity reaches the maximum value (that is, there is no trend of further improvement), and the spine registration is completed. However, when performing spine registration in the above manner, the iterative process requires multiple projections to generate DRR images, which consumes a lot of time. Therefore, there is a problem of low registration efficiency.

[0044] In addition, in the related art, the spatial transformation parameters from three-dimensional CT images to two-dimensional X-ray images can also be directly predicted through a trained model. However, they are all for a single view (i.e., the frontal view), which leads to a large error in the spatial transformation parameters of a viewing angle, and the image features formed by the parameters of the front-to-back translation of the corresponding view change little, making it difficult for the model to extract the features of the parameters. At the same time, although the patient's spine is regarded as a rigid object, considering the physiological structure of the spine, there is still a relative position deviation between each vertebra as the patient's posture changes. The three-dimensional CT image registration of the complete spine cannot provide the final registration result required during surgery. Therefore, the registration for practical applications still needs to be oriented to single vertebrae research. However, existing research lacks exploration in this area, and there is less image feature information for single vertebrae. Existing models have low robustness for such data.

[0045] Based on the above problems, the present application provides a spinal registration method, apparatus, equipment, storage medium and program product, which input the acquired digitally reconstructed radiographic image and two-dimensional X-ray image containing the target vertebra in the spine into a pre-trained registration model, wherein the digitally reconstructed radiographic image is obtained based on the three-dimensional CT image; the registration model uses a cross-attention fusion mechanism to perform feature fusion on the extracted digitally reconstructed radiographic image features and X-ray image features to obtain fused features, and then based on the fused features, obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image, which can greatly improve the efficiency of spinal registration and can effectively improve the accuracy of spinal registration, thereby helping to improve the efficiency and accuracy of surgical navigation.

[0046] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0047] Figure 1This is a flow chart of a spine registration method provided in an embodiment of the present application. The method of the embodiment of the present application can be applied to an electronic device, which can be a server or a server cluster. Figure 1 As shown, the method of the embodiment of the present application includes:

[0048] S101. Acquire a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in a spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional CT image.

[0049] In an embodiment of the present application, a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine may be input by a user to an electronic device executing an embodiment of the present method, or may be sent by other devices to an electronic device executing an embodiment of the present method. Exemplarily, a three-dimensional CT image is obtained by CT scanning the spine (such as the lumbar spine) before surgery on a patient, and the three-dimensional CT image, for example, contains a partial spinal region from the first lumbar vertebra (L1) to the fifth lumbar vertebra (L5). The three-dimensional CT image may be segmented to obtain a three-dimensional CT image containing a target vertebra in the spine, so that a digitally reconstructed radiographic image containing a target vertebra in the spine may be generated based on the three-dimensional CT image by using a DRR method for projection. The two-dimensional X-ray image is obtained by scanning the spine by a C-arm X-ray machine during surgery on the patient. Both the digitally reconstructed radiographic image and the two-dimensional X-ray image contain a single target vertebra in the spine.

[0050] S102. Input the digitally reconstructed radiographic image and the two-dimensional X-ray image into the registration model, obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image based on the fusion features, and complete the registration of the target vertebra. The fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and X-ray image features using a cross-attention fusion mechanism.

[0051] Exemplarily, after obtaining a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine, the digitally reconstructed radiographic image and the two-dimensional X-ray image can be input into a registration model, and the digitally reconstructed radiographic image features corresponding to the digitally reconstructed radiographic image and the X-ray image features corresponding to the two-dimensional X-ray image can be extracted by the registration model, and the extracted digitally reconstructed radiographic image features and the X-ray image features can be feature fused using a cross-attention fusion mechanism to obtain fused features, so that the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image can be obtained based on the fused features to complete the registration of the target vertebra. Among them, when extracting the digitally reconstructed radiographic image features and the X-ray image features, the registration model pays more attention to the relative positional relationship of the target vertebra in the digitally reconstructed radiographic image and the two-dimensional X-ray image; considering that the ultimate goal of the registration model is to obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image, therefore, after extracting the digitally reconstructed radiographic image features and the X-ray image features, it is necessary to fuse the digitally reconstructed radiographic image features and the X-ray image features, and obtain the transformation features between the target vertebra corresponding to the same anatomical position points from the digitally reconstructed radiographic image features and the X-ray image features. Therefore, the embodiment of the present application introduces a cross-attention fusion mechanism. Through the cross-attention fusion mechanism, the registration model can strengthen its attention to the corresponding positions in the digitally reconstructed radiological image and the two-dimensional X-ray image, thereby effectively improving the registration accuracy.

[0052] For details on how to obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image based on the fusion features, please refer to the subsequent embodiments, which will not be described in detail here.

[0053] The spinal registration method provided in the embodiment of the present application is achieved by obtaining a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional CT image; the digitally reconstructed radiographic image and the two-dimensional X-ray image are input into a registration model, and the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained based on fusion features to complete the registration of the target vertebra, wherein the fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and the X-ray image features using a cross-attention fusion mechanism. In the embodiment of the present application, the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained by using a registration model to complete the registration of the target vertebra, which can greatly improve the efficiency of spinal registration; at the same time, the fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and the X-ray image features using a cross-attention fusion mechanism, which can effectively improve the accuracy of spinal registration, thereby helping to improve the efficiency and accuracy of surgical navigation.

[0054] Figure 2 This is a flowchart of a spine registration method provided by another embodiment of the present application. Based on the above embodiment, the present application further describes the spine registration method. Figure 2 As shown, the method of the embodiment of the present application may include:

[0055] S201. Acquire a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in a spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional CT image.

[0056] The detailed description of this step can be found in Figure 1 The relevant description of S101 in the illustrated embodiment will not be repeated here.

[0057] In the embodiment of the present application, the registration model may include a first VIT model, a second VIT model, a fusion module and a multi-layer perceptron. Therefore, Figure 1 The step S102 may further include the following four steps S202 to S205:

[0058] S202 , inputting the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain features of the digitally reconstructed radiological image.

[0059] In this step, after the digitally reconstructed radiographic image including the target vertebra in the spine is obtained, the digitally reconstructed radiographic image may be input into the first VIT model for feature extraction to obtain features of the digitally reconstructed radiographic image.

[0060] Further, optionally, the first VIT model includes a first image block division module and a first Transformer encoder, and inputting the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain features of the digitally reconstructed radiological image can include: dividing the digitally reconstructed radiological image into image blocks of a preset size through the first image block division module to obtain a first image block vector sequence corresponding to the digitally reconstructed radiological image, the first image block vector sequence including an image block vector and an image block position encoding embedded vector corresponding to each first image block; inputting the first image block vector sequence into the first Transformer encoder for feature extraction to obtain features of the digitally reconstructed radiological image.

[0061] Exemplarily, the preset size can be set as needed, and the embodiments of the present application do not limit this. For example, considering that an image block (patch) that is too large may lose detail information, and a patch that is too small may result in excessive consumption of computing resources, assuming that the image size of the digitally reconstructed radiographic image is 512×512 pixels, the digitally reconstructed radiographic image can be divided into 8×8 patches (i.e., the first image block) through the first image block division module, and the size of each patch is 64×64 pixels. Each patch is flattened into a vector, and a 4096-dimensional image block vector is obtained after flattening. In order to reduce the computational burden and storage requirements, optionally, the embodiments of the present application use principal component analysis to perform feature dimensionality reduction on the flattened image block vector, such as projecting a 4096-dimensional image block vector to a 1024-dimensional image block vector.

[0062] For each first image block, a position index can be generated to indicate the position of the first image block in the entire digitally reconstructed radiological image. Position indexes are usually assigned in row-major order, and each position index is mapped to a vector of fixed dimension. Since the first Transformer encoder itself does not have the ability to perceive position information, and at the same time, in the embodiment of the present application, the relative position of the target vertebra in the digitally reconstructed radiological image mainly depends on features, it is necessary to add position encoding to each patch. Position encoding can help the first Transformer encoder understand the relative position of the patch and retain spatial structure information. For example, the patch can be positionally encoded by sine and cosine functions to obtain an image block position encoding embedding vector, which is similar to the original position encoding in the first Transformer encoder. Assuming that the position index is represented by pos and the encoding dimension is represented by d, the position encoding (PositionalEncoding, PE) of the i-th dimension can be expressed as:

[0063]

[0064] It can be understood that the above position encoding is static, that is, it remains unchanged during the training and reasoning process of the registration model. For the absolute position information of the vertebra in the image, the above position encoding method can avoid the loss of position information to the greatest extent. During the entire registration model training process, the absolute position of the vertebra can be constrained at all times.

[0065] After obtaining the image block vector and the image block position code embedding vector corresponding to each first image block through the first image block division module, the image block vector and the corresponding image block position code can be spliced ​​to obtain a first image block vector sequence corresponding to the digitally reconstructed radiological image.

[0066] The first image block vector sequence is input into the first Transformer encoder for feature extraction, and the features of the digitally reconstructed radiographic image can be obtained. Among them, the first Transformer encoder is used to extract rich high-dimensional features from the input digitally reconstructed radiographic image, thereby converting the digitally reconstructed radiographic image into a potential feature representation, and effectively summarizing the visual information required for position feature extraction. It should be noted that the first Transformer encoder used for feature extraction in the embodiment of the present application does not include the additional class token added in the current VIT model. This modification shifts the focus of the first Transformer encoder from image classification to providing dense feature representation. The first Transformer encoder includes a multi-head self-attention mechanism. Based on the multi-head self-attention mechanism, the input can be processed through multiple parallel self-attention layers, self-attention calculations can be performed in different subspaces, and then the results can be spliced ​​together to capture richer information. The specific structure of the first Transformer encoder can refer to the subsequent embodiments and will not be repeated here.

[0067] S203, inputting the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features.

[0068] In this step, after obtaining a two-dimensional X-ray image containing a target vertebra in the spine, the two-dimensional X-ray image can be input into a second VIT model for feature extraction to obtain X-ray image features.

[0069] Further, optionally, the second VIT model includes a second image block division module and a second Transformer encoder, and inputting the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features can include: dividing the two-dimensional X-ray image into image blocks of a preset size by the second image block division module to obtain a second image block vector sequence corresponding to the two-dimensional X-ray image, the second image block vector sequence including an image block vector and an image block position encoding embedding vector corresponding to each second image block; inputting the second image block vector sequence into the second Transformer encoder for feature extraction to obtain X-ray image features.

[0070] In this embodiment, the second image block division module has a function similar to that of the first image block division module, and the function and structure of the second Transformer encoder are similar to those of the first Transformer encoder. Exemplarily, referring to the example of step S202, the image block vector corresponding to each second image block can be obtained by dividing the two-dimensional X-ray image into image blocks of a preset size through the second image block division module. Figure 3 A schematic diagram of dividing a two-dimensional X-ray image into image blocks provided by an embodiment of the present application is shown in FIG. Figure 3As shown, the image size of the two-dimensional X-ray image is, for example, 512×512 pixels, and the two-dimensional X-ray image can be divided into 8×8 patches (i.e., second image blocks), each of which has a size of 64×64 pixels. For each second image block, a position index can be generated to indicate the position of the second image block in the entire two-dimensional X-ray image, so that the image block position code embedding vector corresponding to each second image block can be obtained according to the position index; the image block vector corresponding to the second image block and the image block position code are spliced ​​to obtain a second image block vector sequence corresponding to the two-dimensional X-ray image.

[0071] The second image block vector sequence is input into the second Transformer encoder for feature extraction to obtain X-ray image features. It should be noted that the second Transformer encoder used for feature extraction in the embodiment of the present application is similar to the first Transformer encoder and does not include the class token additionally added in the current VIT model. The specific structure of the second Transformer encoder can be referred to in subsequent embodiments and will not be repeated here.

[0072] It should be noted that the embodiment of the present application does not limit the order in which steps S202 and S203 are executed.

[0073] S204, using a cross-attention fusion mechanism through a fusion module to fuse the digitally reconstructed radiological image features and the X-ray image features to obtain fused features.

[0074] In this step, after the digitally reconstructed radiographic image features and the X-ray image features are obtained, the digitally reconstructed radiographic image features and the X-ray image features can be fused using a cross-attention fusion mechanism through a fusion module to obtain fused features.

[0075] Further, optionally, the digitally reconstructed radiographic image features and the X-ray image features are fused by a fusion module using a cross-attention fusion mechanism to obtain fused features, which may include: for a first sub-feature corresponding to each first image block in the digitally reconstructed radiographic image features, generating a query vector corresponding to the first sub-feature through a first convolutional layer and a first weight; for a second sub-feature corresponding to each second image block in the X-ray image features, generating a key vector corresponding to the second sub-feature through a second convolutional layer and a second weight, and generating a value vector corresponding to the second sub-feature through a third convolutional layer and a third weight; obtaining a target attention weight based on the query vector, the key vector and the value vector; performing weighted summation processing on multiple target attention weights and the value vectors corresponding to the target attention weights to obtain a weighted sum feature; and performing feature fusion processing on the weighted sum feature and the digitally reconstructed radiographic image features to obtain a fused feature.

[0076] It can be understood that the embodiment of the present application obtains the transformation features between the target vertebrae corresponding to the same anatomical position points from the digitally reconstructed radiographic image features and the X-ray image features through the cross-attention fusion mechanism. DRR Denotes the first sub-feature corresponding to the first image block, and F X-ray represents the second sub-feature corresponding to the second image block; it can be obtained by the first 1×1 convolution layer and the first weight (for example, using W q (W) generates the query vector (Q) corresponding to the first sub-feature: Q = W q F DRR ; can be achieved through a 1×1 second convolutional layer and a second weight (for example, using W k (K) generates the key vector (K) corresponding to the second sub-feature: K = W k F X-ray ; can be achieved through a 1×1 third convolution layer and a third weight (for example, using W v (W) generates the value vector (V) corresponding to the second sub-feature: V = W v F X-ray .

[0077] The similarity between the query vector Q and the key vector K is obtained through the dot product to obtain the target attention score, which can then be converted into a target attention weight (such as AttentionWeighs) through an activation function (such as the softmax function): Where T represents transpose; d k Represents the dimension of the key vector, which is used for normalization. Perform weighted summation on multiple target attention weights and the value vectors corresponding to the target attention weights to obtain the weighted summed features (for example, represented by Output): Where n represents the total number of patches, refer to Figure 3 , n is 64. In the above manner, it is possible to focus on highly relevant information in the digitally reconstructed radiographic image and the two-dimensional X-ray image.

[0078] After obtaining the weighted summed features, the weighted summed features and the digitally reconstructed radiographic image features may be fused to obtain fused features. For example, the weighted summed features and the digitally reconstructed radiographic image features may be added to obtain fused features, or the product of the weighted summed features and a learning parameter (such as γ) may be obtained, and the product of the weighted summed features and the digitally reconstructed radiographic image features (such as DRR) may be added to obtain fused features. F FusedOutput = γ × Output + DRR F .

[0079] With reference to the above-mentioned embodiment, through the cross-attention fusion mechanism, the feature information from the digitally reconstructed radiological image features and the X-ray image features can be effectively fused, so that the registration model can pay more attention to important related features.

[0080] S205, performing regression processing on the fusion features through a multi-layer perceptron to obtain spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image, and completing the registration of the target vertebra.

[0081] Exemplarily, the multilayer perceptron includes, for example, four fully connected layers, and the fused features can be regressed through the four fully connected layers to convert the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image, thereby completing the registration of the target vertebra.

[0082] The spine registration method provided in the embodiment of the present application comprises a registration model including a first VIT model, a second VIT model, a fusion module and a multi-layer perceptron; a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine are obtained, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional CT image; the digitally reconstructed radiographic image is input into the first VIT model for feature extraction to obtain digitally reconstructed radiographic image features, and the two-dimensional X-ray image is input into the second VIT model for feature extraction to obtain X-ray image features; the digitally reconstructed radiographic image features and the X-ray image features are feature-fused with a cross-attention fusion mechanism through the fusion module to obtain fused features; the fused features are regressed through a multi-layer perceptron, and the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are used to complete the registration of the target vertebra. In the embodiment of the present application, the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained through the first VIT model, the second VIT model, the fusion module and the multi-layer perceptron included in the registration model to complete the registration of the target vertebra, wherein the first VIT model and the second VIT model can extract rich high-dimensional features in the image, convert the image into a potential feature representation, and effectively summarize the visual information required for position feature extraction. Through the cross-attention fusion mechanism, the registration model can strengthen its attention to the corresponding positions in the digitally reconstructed radiographic image and the two-dimensional X-ray image, thereby improving the registration accuracy. Therefore, the embodiment of the present application can greatly improve the efficiency of spinal registration, and at the same time improve the accuracy of spinal registration, thereby helping to improve the efficiency and accuracy of surgical navigation.

[0083] Based on the above embodiments, Figure 4 A schematic diagram of the structure of a Transformer encoder provided in an embodiment of the present application, such as Figure 4 As shown, both the first Transformer encoder and the second Transformer encoder can adopt Figure 4 The structure shown. Figure 4, the Transformer encoder includes a first layer normalization (Layer Norm) module, a multi-head attention (Multi-Head Attention) mechanism module, a first random dropout module, a second layer normalization module, a multi-layer perceptron (MLP) and a second random dropout module. Among them, the first layer normalization module is used to perform layer normalization on the input data, that is, to standardize the distribution of the input data, thereby speeding up the training speed and improving stability. The multi-head attention mechanism module is used to obtain the correlation between different positions in the input data, allowing the Transformer encoder to process information in parallel in different subspaces, thereby capturing rich feature representations in the input data. The first random dropout module is used to randomly set the output of some neurons to zero to prevent overfitting. Optionally, a first residual connection module (not shown in the figure) can also be added to the output of the first random dropout module to facilitate gradient flow, alleviate the gradient vanishing problem, and help train deep networks. The second layer normalization module is used after the first random dropout module to further standardize the distribution of data. The multilayer perceptron includes two fully connected layers, and there is usually an activation function between the fully connected layers, such as the Gaussian Error Linear Units (GELU) activation function. The multilayer perceptron is used to further process and transform the feature representation after the self-attention mechanism. The second random zeroing module is used to further randomly set the output of some neurons to zero after the multilayer perceptron to reduce overfitting. A second residual connection module (not shown in the figure) can also be added to the output of the second random zeroing module to facilitate gradient flow and further alleviate the gradient vanishing problem. Usually, the structure of the above Transformer encoding can be stacked with multiple layers (such as 12 layers) to deeply process the input data. Reference Figure 4 The first Transformer encoder of the embodiment of the present application can extract rich high-dimensional features from the digitally reconstructed radiological image, and the second Transformer encoder can extract rich high-dimensional features from the two-dimensional X-ray image, thereby effectively summarizing the visual information required for position feature extraction.

[0084] Figure 5 The flowchart of the training method of the registration model provided in one embodiment of the present application. The method of the embodiment of the present application can be applied to an electronic device, which can be a server or a server cluster. Figure 5 As shown, the method of the embodiment of the present application includes:

[0085] S501. Acquire multiple training samples, where the training samples include digitally reconstructed radiographic image samples of sample vertebrae in a sample spine, two-dimensional X-ray image samples, and labels, where the digitally reconstructed radiographic image samples and two-dimensional X-ray image samples are obtained based on three-dimensional CT image samples, and the labels are used to represent reference space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples.

[0086] In the embodiment of the present application, the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are obtained based on the three-dimensional CT image samples, that is, the simulated two-dimensional X-ray image samples are synthesized from the three-dimensional CT image samples for model training. For example, firstly, a spatial transformation parameter is randomly generated in a uniformly distributed manner within the range of [-5°, 5°], [-5mm, 5mm] as the initial position parameter (for example, using T init denoted by ), and based on the initial position parameter, the 3D CT image sample is spatially transformed, and then the digitally reconstructed radiographic image sample at the initial angle is projected. Then, the spatial transformation parameters are randomly generated in a uniformly distributed manner in the range of, for example, [-10°, 10°], [-5mm, 5mm] (for example, using T detal ) as labels, and according to T detal The three-dimensional CT image samples at the initial angle are further spatially transformed and projected to obtain target digitally reconstructed radiographic image samples. The style of the target digitally reconstructed radiographic image samples is converted through the style transfer model to obtain simulated two-dimensional X-ray image samples.

[0087] In an embodiment of the present application, for example, 140 three-dimensional CT images containing lumbar vertebrae can be collected and screened, wherein each three-dimensional CT image at least contains a portion of the spinal region from the first lumbar vertebra (L1) to the fifth lumbar vertebra (L5). A total of 5 single vertebrae from L1 to L5 can be segmented from each three-dimensional CT image, and finally 700 single vertebrae can be obtained. Among them, 500 of the 700 single vertebrae can be used as a training set containing multiple training samples (for training the model), 100 as a validation set (for adjusting parameters and selecting the best model), and 100 as a test set (for evaluating the performance of the final model).

[0088] S502. Input the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples into the registration model, and obtain the training space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples based on the fused feature samples. The fused feature samples are obtained by fusing the extracted digitally reconstructed radiographic image feature samples and the X-ray image feature samples using a cross-attention fusion mechanism.

[0089] In this step, after obtaining the training samples, the training samples can be input into the registration model in batches (the batch size is, for example, 32) for training to obtain the training space transformation parameters output by the registration model. Figure 2 In the illustrated embodiment, the registration model may include a first VIT model, a second VIT model, a fusion module and a multi-layer perceptron; the digitally reconstructed radiological image samples may be input into the first VIT model for feature extraction to obtain digitally reconstructed radiological image feature samples; the two-dimensional X-ray image samples may be input into the second VIT model for feature extraction to obtain X-ray image feature samples; the feature fusion of the digitally reconstructed radiological image feature samples and the X-ray image feature samples may be performed through the fusion module using a cross-attention fusion mechanism to obtain fused feature samples; the fused feature samples may be regressed through the multi-layer perceptron to obtain training space transformation parameters.

[0090] S503: Obtain a loss function value based on the training space transformation parameters and labels.

[0091] In this step, the loss function is, for example, a mean squared error (MSE) loss function, and the loss function value can be obtained based on the training space transformation parameters and labels.

[0092] S504: Adjust the model parameters of the registration model based on the loss function value to obtain a trained registration model.

[0093] For example, an Adaptive Moment Estimation (Adam) optimizer may be used, the number of training rounds (epochs) may be set to 100, the initial learning rate may be 0.0002, and after the 50th epoch, the learning rate may be linearly decayed to 0. During the training process, the model parameters of the registration model may be adjusted based on the loss function value, and finally a trained registration model may be obtained.

[0094] The registration model of the embodiment of the present application can be implemented using Python (an object-oriented interpreted computer programming language), using PyTorch (an open source deep learning framework for machine learning and deep learning) as a registration model building framework.

[0095] The training method of the registration model provided in the embodiment of the present application obtains multiple training samples, the training samples include digitally reconstructed radiographic image samples including sample vertebrae in the sample spine, two-dimensional X-ray image samples and labels, the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are obtained based on the three-dimensional CT image samples, and the labels are used to represent the reference space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples; the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are input into the registration model, and the training space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples are obtained based on the fused feature samples, and the fused feature samples are obtained by feature fusion of the extracted digitally reconstructed radiographic image feature samples and the X-ray image feature samples using a cross-attention fusion mechanism; based on the training space transformation parameters and the labels, a loss function value is obtained, and based on the loss function value, the model parameters of the registration model are adjusted to obtain a trained registration model. In an embodiment of the present application, digitally reconstructed radiographic image samples and two-dimensional X-ray image samples are automatically generated based on three-dimensional CT image samples, thereby solving the problem of sample shortage; the registration model can extract rich features from the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples and perform feature fusion. Therefore, the registration model can quickly and accurately predict the spatial transformation parameters from the digitally reconstructed radiographic image samples to the two-dimensional X-ray image samples, thereby enabling the spinal registration to be completed quickly and accurately when the trained registration model is applied to spinal registration.

[0096] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0097] Figure 6 This is a schematic diagram of the structure of a spinal alignment device provided in one embodiment of the present application. Figure 6 As shown, the spinal column registration device 600 of the embodiment of the present application includes: an acquisition module 601 and a registration module 602. Among them:

[0098] The acquisition module 601 is used to acquire a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine. The digitally reconstructed radiographic image is obtained based on the three-dimensional CT image.

[0099] The registration module 602 is used to input the digitally reconstructed radiological image and the two-dimensional X-ray image into the registration model, obtain the spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image based on the fusion features, and complete the registration of the target vertebra. The fusion features are obtained by fusing the extracted digitally reconstructed radiological image features and X-ray image features using a cross-attention fusion mechanism.

[0100] Optionally, the registration model includes a first VIT model, a second VIT model, a fusion module and a multi-layer perceptron. The registration module 602 can be specifically used to: input the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain digitally reconstructed radiological image features; input the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features; perform feature fusion of the digitally reconstructed radiological image features and the X-ray image features using a cross-attention fusion mechanism through the fusion module to obtain fused features; and perform regression processing on the fused features through the multi-layer perceptron to obtain spatial transformation parameters.

[0101] Optionally, the first VIT model includes a first image block division module and a first Transformer encoder. When the registration module 602 is used to input the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain the features of the digitally reconstructed radiological image, it can be specifically used to: divide the digitally reconstructed radiological image into image blocks of a preset size through the first image block division module to obtain a first image block vector sequence corresponding to the digitally reconstructed radiological image, the first image block vector sequence including an image block vector and an image block position encoding embedded vector corresponding to each first image block; input the first image block vector sequence into the first Transformer encoder for feature extraction to obtain the features of the digitally reconstructed radiological image.

[0102] Optionally, the second VIT model includes a second image block division module and a second Transformer encoder. When the registration module 602 is used to input the two-dimensional X-ray image into the second VIT model for feature extraction to obtain X-ray image features, it can be specifically used to: divide the two-dimensional X-ray image into image blocks of a preset size through the second image block division module to obtain a second image block vector sequence corresponding to the two-dimensional X-ray image, the second image block vector sequence including an image block vector and an image block position encoding embedding vector corresponding to each second image block; input the second image block vector sequence into the second Transformer encoder for feature extraction to obtain X-ray image features.

[0103] Optionally, when the registration module 602 is used to perform feature fusion on the digitally reconstructed radiographic image features and the X-ray image features through the fusion module using a cross-attention fusion mechanism to obtain a fused feature, it can be specifically used to: for a first sub-feature corresponding to each first image block in the digitally reconstructed radiographic image feature, generate a query vector corresponding to the first sub-feature through a first convolution layer and a first weight; for a second sub-feature corresponding to each second image block in the X-ray image feature, generate a key vector corresponding to the second sub-feature through a second convolution layer and a second weight, and generate a value vector corresponding to the second sub-feature through a third convolution layer and a third weight; based on the query vector, the key vector and the value vector, obtain a target attention weight; perform weighted sum processing on multiple target attention weights and the value vectors corresponding to the target attention weights to obtain a weighted sum feature; perform feature fusion on the weighted sum feature and the digitally reconstructed radiographic image feature to obtain a fused feature.

[0104] Optionally, the spinal registration device 600 may also include a training module (not shown in the figure) for training to obtain a registration model in the following manner: obtaining multiple training samples, the training samples including digitally reconstructed radiographic image samples, two-dimensional X-ray image samples and labels containing sample vertebrae in the sample spine, the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are obtained based on the three-dimensional CT image samples, and the labels are used to represent the reference space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples; inputting the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples into the registration model, and obtaining the training space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples based on the fused feature samples, and the fused feature samples are obtained by feature fusion of the extracted digitally reconstructed radiographic image feature samples and the X-ray image feature samples using a cross-attention fusion mechanism; obtaining the loss function value based on the training space transformation parameters and the labels; and adjusting the model parameters of the registration model based on the loss function value to obtain a trained registration model.

[0105] The device of the embodiment of the present application can be used to execute the technical solution of any of the above-mentioned method embodiments, and its implementation principle and technical effect are similar, which will not be repeated here.

[0106] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 700 may include: at least one processor 701 and a memory 702 .

[0107] The memory 702 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0108] The memory 702 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0109] The processor 701 is used to execute the computer-executable instructions stored in the memory 702 to implement the spinal alignment method described in the aforementioned method embodiment. The processor 701 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. Specifically, when implementing the spinal alignment method described in the aforementioned method embodiment, the electronic device may be, for example, an electronic device with processing capabilities such as a server.

[0110] Optionally, the electronic device 700 may further include a communication interface 703. In a specific implementation, if the communication interface 703, the memory 702 and the processor 701 are implemented independently, the communication interface 703, the memory 702 and the processor 701 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0111] Optionally, in a specific implementation, if the communication interface 703, the memory 702 and the processor 701 are integrated on a chip, the communication interface 703, the memory 702 and the processor 701 can communicate through an internal interface.

[0112] The present application also provides a computer-readable storage medium, in which computer program instructions are stored. When a processor executes the computer program instructions, the above-mentioned spine alignment method is implemented.

[0113] The present application also provides a computer program product, including a computer program, which implements the above spine registration method when executed by a processor.

[0114] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0115] An exemplary readable storage medium is coupled to the processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit. Of course, the processor and the readable storage medium can also be present in the spinal registration device as discrete components.

[0116] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A spine registration method, characterized in that: include: acquiring a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in a spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional computed tomography (CT) image; The digitally reconstructed radiographic image and the two-dimensional X-ray image are input into a registration model, and spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained based on fusion features to complete the registration of the target vertebra. The fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and X-ray image features using a cross-attention fusion mechanism.

2. The spine registration method according to claim 1, characterized in that: The registration model includes a first visual converter VIT model, a second VIT model, a fusion module and a multi-layer perceptron. The digitally reconstructed radiological image and the two-dimensional X-ray image are input into the registration model, and spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image are obtained based on fusion features, including: Inputting the digitally reconstructed radiological image into a first VIT model for feature extraction to obtain features of the digitally reconstructed radiological image; Inputting the two-dimensional X-ray image into the second VIT model to perform feature extraction to obtain the X-ray image features; The fusion module adopts a cross attention fusion mechanism to fuse the digitally reconstructed radiological image features and the X-ray image features to obtain the fused features; The fusion features are regressed by the multi-layer perceptron to obtain the spatial transformation parameters.

3. The spine registration method according to claim 2, characterized in that: The first VIT model includes a first image block division module and a first transformer encoder, and the step of inputting the digitally reconstructed radiological image into the first VIT model for feature extraction to obtain the features of the digitally reconstructed radiological image includes: The digitally reconstructed radiographic image is divided into image blocks of a preset size by the first image block division module to obtain a first image block vector sequence corresponding to the digitally reconstructed radiographic image, wherein the first image block vector sequence includes an image block vector corresponding to each first image block and an image block position code embedding vector; The first image block vector sequence is input into a first Transformer encoder for feature extraction to obtain the digitally reconstructed radiological image features.

4. The spine registration method according to claim 3, characterized in that: The second VIT model includes a second image block division module and a second Transformer encoder, and the two-dimensional X-ray image is input into the second VIT model for feature extraction to obtain the X-ray image features, including: The second image block division module divides the two-dimensional X-ray image into image blocks of the preset size to obtain a second image block vector sequence corresponding to the two-dimensional X-ray image, wherein the second image block vector sequence includes an image block vector corresponding to each second image block and an image block position code embedding vector; The second image block vector sequence is input into the second Transformer encoder for feature extraction to obtain the X-ray image features.

5. The spine registration method according to claim 4, characterized in that: The step of fusing the digitally reconstructed radiological image features and the X-ray image features using a cross-attention fusion mechanism through the fusion module to obtain the fused features includes: For each first sub-feature corresponding to the first image block in the digitally reconstructed radiological image feature, generating a query vector corresponding to the first sub-feature through a first convolutional layer and a first weight; For each second sub-feature corresponding to the second image block in the X-ray image feature, generate a key vector corresponding to the second sub-feature through a second convolution layer and a second weight, and generate a value vector corresponding to the second sub-feature through a third convolution layer and a third weight; Obtaining a target attention weight based on the query vector, the key vector, and the value vector; Performing weighted sum processing on the multiple target attention weights and the value vectors corresponding to the target attention weights to obtain weighted summed features; The weighted summed feature and the digitally reconstructed radiological image feature are fused to obtain the fused feature.

6. The spine registration method according to any one of claims 1 to 5, characterized in that: The registration model is trained in the following way: Acquire a plurality of training samples, wherein the training samples include digitally reconstructed radiographic image samples, two-dimensional X-ray image samples, and labels including sample vertebrae in a sample spine, wherein the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples are obtained based on three-dimensional CT image samples, and the labels are used to represent reference space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples; Inputting the digitally reconstructed radiographic image samples and the two-dimensional X-ray image samples into a registration model, and obtaining training space transformation parameters from the three-dimensional CT image samples to the two-dimensional X-ray image samples based on fused feature samples, wherein the fused feature samples are obtained by performing feature fusion on the extracted digitally reconstructed radiographic image feature samples and the X-ray image feature samples using a cross-attention fusion mechanism; Based on the training space transformation parameter and the label, obtaining a loss function value; The model parameters of the registration model are adjusted based on the loss function value to obtain a trained registration model.

7. A spinal alignment device, characterized in that: include: an acquisition module, for acquiring a digitally reconstructed radiographic image and a two-dimensional X-ray image containing a target vertebra in the spine, wherein the digitally reconstructed radiographic image is obtained based on a three-dimensional computed tomography (CT) image; A registration module is used to input the digitally reconstructed radiographic image and the two-dimensional X-ray image into a registration model, obtain spatial transformation parameters from the three-dimensional CT image to the two-dimensional X-ray image based on fusion features, and complete the registration of the target vertebra. The fusion features are obtained by fusing the extracted digitally reconstructed radiographic image features and X-ray image features using a cross-attention fusion mechanism.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the spinal registration method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the spinal registration method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the spine registration method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Characteristic and intensity-based X-ray and three-dimensional scanning model registration method and system

    CN121437256A