Biplane image spinal key point three-dimensional positioning method and system

Abstract

The application relates to the medical field and discloses a double-plane image spine key point three-dimensional positioning method and system, which can improve the accuracy and robustness of three-dimensional positioning of spine key points under double-plane spine X-ray images. The method obtains double-plane spine X-ray images containing front and rear views and side views, and receives a point guide mark specified by a user on the same level reference vertebra. By extracting initial two-dimensional visual features of each view, a guide mask is generated according to the position of the point guide mark, first attention calculation is performed on the guide mask and the initial features of the reference vertebra, and the two-dimensional visual features corresponding to other vertebrae are enhanced. By using a learnable vertebra representation and enhanced features, the semantic context of the vertebra is extracted through a dynamic sparse attention mechanism, and two-dimensional vertebra heat maps of the front, rear and side views are obtained. The heat maps are fused with corresponding features to construct a pseudo-three-dimensional feature volume, and the accurate three-dimensional coordinates of the vertebrae other than the reference vertebra are obtained by processing the volume.

Description

Technical Field

[0001] This application relates to the medical field, and in particular to a three-dimensional spinal localization technology based on medical imaging. Background Technology

[0002] This section is intended to provide background or context for understanding the implementation of this application and is for reference only. It should not be construed as an admission by the applicant that this section pertains to prior art that was disclosed before the filing date of this application.

[0003] In orthopedics, accurately determining the three-dimensional spatial position of the vertebral bodies is crucial. X-ray fluoroscopy is commonly used due to its advantages such as operational flexibility, relatively low radiation dose, and cost-effectiveness. Among these techniques, biplanar imaging, which acquires X-ray images of the patient from two different angles (typically the standard anteroposterior (AP) and lateral laminar (LAT) views), is a frequently used method for obtaining three-dimensional information about the target.

[0004] However, X-ray images are projections of three-dimensional anatomical structures onto a two-dimensional plane. This inherent characteristic presents numerous challenges to accurately reconstructing or locating three-dimensional structures from two-dimensional images, especially vertebrae with similar and closely arranged morphologies. Existing methods for three-dimensional vertebral localization based on biplane imaging can be broadly classified into two categories.

[0005] The first category is based on two-dimensional mapping. These methods typically begin by detecting or labeling key points of the vertebra (such as the center or corners of the vertebra) in various two-dimensional views, and then use triangulation principles combined with optimization algorithms (such as least squares or statistical shape model-based methods) to reconstruct the three-dimensional coordinates of these key points.

[0006] The second category is based on direct volume reconstruction. These methods attempt to construct a pseudo-3D feature volume from the features of multi-view 2D images, and then directly regress the position of each keypoint in 3D voxel space or its likelihood map through operations such as 3D convolution. When constructing these methods or evaluating their performance, digitally reconstructed radiographs (DRRs) generated from computed tomography (CT) data are sometimes utilized. DRRs are X-ray images simulated from CT volume data and can be used to generate training or testing data with precise 3D annotations.

[0007] Despite the progress made in existing technologies, the above methods still have some insurmountable limitations in practical applications:

[0008] 1. Depth information uncertainty: Due to the projection characteristics of X-ray images, the overlap of anatomical structures, and the low contrast between the vertebral body and the surrounding soft tissue, it is difficult to accurately infer depth information from a single two-dimensional view. In particular, when there is lesion in the vertebral body or obstruction (such as being obscured by other bones or metal implants), the accuracy of three-dimensional localization will be severely affected.

[0009] 2. Difficulty in vertebral body differentiation: The spine is a chain-like structure composed of multiple vertebral units with similar morphology, especially in the thoracic and lumbar segments. Adjacent vertebrae are highly similar in morphology and imaging appearance, making it very difficult to accurately distinguish and correspond the same vertebra in different views on two-dimensional images. In the presence of spinal deformities or internal metal fixation devices, the vertebral body morphology may be altered or obscured, further exacerbating the difficulty in differentiation.

[0010] 3. Difficulty in ensuring viewpoint consistency: The projected shape and relative position of the same vertebra will change in X-ray images taken from different viewpoints (such as AP and LAT views). When processing each view separately, how to effectively establish and ensure the correct one-to-one correspondence between the key points of the vertebra detected in different views is a critical and complex problem.

[0011] 4. The trade-off between computational resource consumption and accuracy: For volumetric reconstruction-based methods, the positioning accuracy largely depends on the spatial resolution of the reconstructed volume. To increase the resolution for more accurate positioning, computational costs (including memory consumption and computation time) often increase dramatically, even quadratically or cubically, which limits its application in intraoperative scenarios requiring real-time feedback. Summary of the Invention

[0012] The purpose of this application is to provide a method and system for three-dimensional localization of key points of the spine in dual-plane imaging, which can improve the accuracy and robustness of three-dimensional localization of key points of the spine under dual-plane spinal X-ray imaging, especially in the case of complex clinical situations such as obstruction and deformity.

[0013] This application discloses a three-dimensional localization method for key points of the spine using dual-plane imaging, including:

[0014] Obtain biplane spinal radiographs of the object to be processed, including at least anteroposterior and lateral views;

[0015] Receive user-specified point guide markers on the dual-plane spinal radiographs, corresponding to the same level of reference vertebrae in the anterior-posterior and lateral views, respectively;

[0016] Based on the biplane spinal ray images, the initial two-dimensional visual features of each view are extracted;

[0017] Based on the position of the point guidance marker, a guidance mask for the reference vertebra is generated; using the guidance mask and the initial two-dimensional visual features of the reference vertebra, the initial two-dimensional visual features corresponding to other vertebrae in the dual-plane image are enhanced through a first attention calculation to obtain enhanced features;

[0018] Using the learnable vertebral representation and the enhanced features, the semantic context of the vertebral body is extracted through a dynamic sparse attention mechanism to obtain two-dimensional vertebral heatmaps of the anterior and posterior views and the lateral views.

[0019] The two-dimensional vertebral heatmaps of the anterior and posterior and lateral views are fused with their respective enhanced features or initial two-dimensional visual features, and a pseudo-three-dimensional feature volume is constructed by triangulation or feature projection methods.

[0020] The pseudo-3D feature volume is processed to obtain the 3D coordinates of the other vertebrae in the dual-plane image, excluding the reference vertebra.

[0021] In a preferred embodiment, the dynamic sparse attention mechanism includes: downsampling the enhanced features to obtain low-resolution features; identifying high-attention region indices through a second attention calculation based on the low-resolution features; mapping the high-attention region indices back to the un-downsampled enhanced features to obtain a sparse high-resolution foreground feature subset; and generating context-rich cone feature representations through a third attention calculation based on the sparse high-resolution foreground feature subset, and converting them into two-dimensional cone heatmaps for each view.

[0022] In a preferred embodiment, in the dynamic sparse attention mechanism, the learnable cone representation serves as the query, the low-resolution features serve as the key for the second attention computation, and the sparse high-resolution foreground feature subset serves as the key for the third attention computation.

[0023] In a preferred embodiment, during the first attention calculation, the initial two-dimensional visual features corresponding to the other vertebrae are used as queries, and the guiding features extracted from the initial two-dimensional visual features of the reference vertebrae using the guiding mask are used as keys and values.

[0024] In a preferred embodiment, the bootstrap mask includes at least a binary bootstrap mask and a one-way distance mask;

[0025] The one-way distance mask contains vertical distance information from each pixel to the point guide marker, and is combined with the enhanced features before the dynamic sparse attention mechanism extracts the cone semantic context.

[0026] In a preferred embodiment, the step of extracting the initial two-dimensional visual features of each view employs a two-dimensional U-shaped network structure to extract multi-scale visual features.

[0027] In a preferred embodiment, the step of processing the pseudo-3D feature volume includes: processing the aggregated multi-view feature volume using 3D convolution to obtain a 3D cone heatmap; and processing the 3D cone heatmap using a soft weighted summation function to obtain the 3D coordinates.

[0028] In a preferred embodiment, the method further includes: defining an overall loss function that includes two-dimensional context loss and three-dimensional localization loss, and using an integrated training method to simultaneously optimize the generation of the two-dimensional cone heatmap and the localization of the three-dimensional coordinates, without the need for a pre-trained two-dimensional feature extraction backbone network.

[0029] This application also discloses a dual-plane imaging three-dimensional localization system for key points of the spine, including:

[0030] Memory, used to store computer-executable instructions; and,

[0031] A processor, coupled to the memory, is configured to implement the steps of the method as described above when executing the computer-executable instructions.

[0032] This application also discloses a computer program product including computer-executable instructions that, when executed by a processor, implement the steps in the method described above.

[0033] In the embodiments of this application, a method is employed that receives point guidance markers corresponding to the same level of reference vertebrae specified by the user on a dual-plane image, generates a guidance mask based on these markers, and enhances the initial two-dimensional visual features of other vertebrae by combining first attention calculation. Then, using learnable vertebrae representations and enhanced features, a dynamic sparse attention mechanism is used to extract the semantic context of the vertebrae to obtain a two-dimensional vertebrae heatmap. Finally, the two-dimensional heatmap and features are fused to construct a pseudo-three-dimensional feature volume, and the three-dimensional coordinates of other vertebrae are obtained. This overall method effectively integrates user prior knowledge (point guidance) with the feature extraction capabilities of deep learning networks. Through layer-by-layer guidance and feature enhancement, and extraction of contextual information, a mapping from sparse user input to precise three-dimensional localization is achieved. This method overcomes the problems of depth uncertainty, difficulty in vertebrae differentiation, difficulty in ensuring viewpoint consistency, and high computational resource consumption in traditional methods. Specifically, point guidance provides a clear starting point and reference, reducing the blindness of the network's search throughout the spine; feature enhancement based on reference vertebrae helps distinguish vertebrae with similar morphology; semantic context extraction can utilize the anatomical relationships between vertebrae; and subsequent multi-view fusion and 3D reconstruction integrate information from different planes, thereby improving the accuracy, robustness, and computational efficiency of 3D localization even in complex situations (such as occlusion, deformity, and metal implants).

[0034] Furthermore, by employing a specific dual attention processing flow in the dynamic sparse attention mechanism—which involves downsampling enhanced features, performing a second attention calculation based on low-resolution features to identify high-interest region indices, mapping the indices back to un-downsampled enhanced features to obtain a sparse, high-resolution foreground feature subset, and finally performing a third attention calculation based on this subset to generate context-rich vertebral feature representations—this approach can significantly reduce computational load while still focusing on the most important foreground feature regions for vertebral localization and performing refined context extraction. This results in a significant improvement in the model's computational efficiency while maintaining high localization accuracy.

[0035] Furthermore, by assigning specific roles to the learnable cone representation as the query, the downsampled low-resolution features as the key for the second attention computation, and the sparse high-resolution foreground feature subset as the key for the third attention computation in the dynamic sparse attention mechanism, the attention mechanism can efficiently scan global information using low-resolution features to identify potential foreground regions in the first stage. In the second stage, it can combine the learnable cone anatomical prior (as the query) and the selected high-resolution foreground features (as the key) to extract more accurate and targeted fine-grained contextual information, thereby optimizing the effectiveness and targeting of the dual attention computation.

[0036] Furthermore, by using the initial two-dimensional visual features corresponding to other vertebrae as queries in the first attention calculation of the point-guided feature enhancement unit, and using the guiding features extracted from the initial two-dimensional visual features of the reference vertebrae using the guiding mask as both key and value settings, the network can effectively utilize the user-specified reference vertebrae information (guiding features) to enhance and associate the feature expressions of other vertebrae to be detected. This strengthens the guiding role of the reference point in the learning of the entire spinal feature and improves the discriminability and relevance of the features.

[0037] Furthermore, by including not only a binary guiding mask but also a one-way distance mask in the guiding mask, and combining this one-way distance mask with the enhanced features before the dynamic sparse attention mechanism extracts the semantic context, explicit vertical spatial distance prior information about the reference cone's position can be incorporated into the features. This not only facilitates feature extraction within the reference region defined by the binary mask but also provides the network with the relative positional relationships between different pixels and the reference point. In particular, it helps establish more accurate semantic correspondences and spatial alignments across different views, enhancing the network's ability to perceive and utilize the relative position of the cone during subsequent semantic context extraction.

[0038] Furthermore, by employing a two-dimensional U-shaped network structure in the step of extracting the initial two-dimensional visual features of each view, its encoder-decoder structure and skip connection characteristics can be utilized to effectively capture and integrate multi-scale visual cues in the image, thereby providing richer and more robust initial feature maps for subsequent point-guided feature enhancement and semantic context extraction, which helps to improve the overall localization accuracy.

[0039] Furthermore, by first using 3D convolution to process the aggregated multi-view feature volume to obtain a 3D cone heatmap when processing pseudo-3D feature volume, and then processing the heatmap with a soft-weighted summation function to obtain 3D coordinates, the network can learn the probability distribution (heatmap) in 3D space from the fused multi-view information. The probability distribution can then be transformed into accurate, sub-pixel-level 3D coordinates through a differentiable soft-weighted summation method, thereby improving the accuracy and continuity of the final 3D localization result.

[0040] Furthermore, by defining an overall loss function that includes two-dimensional context loss and three-dimensional localization loss, and by adopting an integrated training approach to simultaneously optimize two-dimensional cone heatmap generation and three-dimensional coordinate localization, and without the need for a pre-trained two-dimensional feature extraction backbone network, end-to-end training can be achieved. This simplifies the model training process, avoids suboptimal solutions that may result from staged training, and promotes more thorough collaborative learning among network modules, thereby improving the overall model performance and convergence efficiency.

[0041] The various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which should be considered as having been recorded in this specification), unless such a combination of technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; it is impossible to use both simultaneously. Feature E can be technically combined with feature C. Therefore, the solution A+B+C+D should not be considered as having been recorded because it is technically infeasible, while the solution A+B+C+E should be considered as having been recorded. Attached Figure Description

[0042] Figure 1 This is a schematic flowchart of a two-plane imaging method for three-dimensional localization of key points of the spine according to one embodiment of this application. Detailed Implementation

[0043] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] One embodiment of this application relates to a three-dimensional localization method for key spinal points using dual-plane imaging, the process of which is as follows: Figure 1 As shown, the method includes the following steps:

[0046] In step 101, a biplane spinal radiograph of the object to be processed, including at least anteroposterior and lateral views, is acquired.

[0047] Then proceed to step 102, where user-specified point guide markers on biplane spinal radiographs, corresponding to the same level of reference vertebrae in the anteroposterior and lateral views respectively, are received.

[0048] Then proceed to step 103, where initial two-dimensional visual features are extracted from each view based on the biplane spinal ray images. Preferably, a two-dimensional U-shaped network structure can be used to extract multi-scale visual features.

[0049] Then, proceeding to step 104, a guiding mask for the reference vertebra is generated based on the position of the guiding marker. Using the guiding mask and the initial two-dimensional visual features of the reference vertebra, the initial two-dimensional visual features corresponding to other vertebrae in the dual-plane image are enhanced through a first attention calculation, resulting in enhanced features. Preferably, in the first attention calculation, the initial two-dimensional visual features corresponding to other vertebrae are used as queries, and the guiding features extracted from the initial two-dimensional visual features of the reference vertebra using the guiding mask are used as keys and values. Preferably, the guiding mask includes at least a binary guiding mask and a one-way distance mask. The one-way distance mask contains the vertical distance information from each pixel to the guiding marker and is combined with the enhanced features before the dynamic sparse attention mechanism extracts the semantic context of the vertebrae.

[0050] The process then proceeds to step 105, where the learnable vertebral representation and enhancement features are used to extract the semantic context of the vertebrae through a dynamic sparse attention mechanism, resulting in two-dimensional vertebral heatmaps for anteroposterior and lateral views. Preferably, the dynamic sparse attention mechanism includes: downsampling the enhancement features to obtain low-resolution features; identifying high-attention region indices through a second attention calculation based on the low-resolution features; mapping the high-attention region indices back to the undownsampled enhancement features to obtain a sparse high-resolution foreground feature subset; and generating a context-rich vertebral feature representation through a third attention calculation based on the sparse high-resolution foreground feature subset, which is then converted into two-dimensional vertebral heatmaps for each view. In the dynamic sparse attention mechanism, the learnable vertebral representation serves as the query, the low-resolution features as the key for the second attention calculation, and the sparse high-resolution foreground feature subset as the key for the third attention calculation.

[0051] Then proceed to step 106, where the two-dimensional vertebral heatmaps of the anterior and posterior views and the lateral views are fused with their respective enhanced features or initial two-dimensional visual features, and a pseudo-three-dimensional feature volume is constructed by triangulation or feature projection methods.

[0052] Then, proceed to step 107 to process the pseudo-3D feature volume and obtain the 3D coordinates of the vertebrae other than the reference vertebra in the dual-plane image. Preferably, 3D convolution can be used to process the aggregated multi-view feature volume to obtain a 3D vertebral heatmap, and a soft-weighted summation function can be used to process the 3D vertebral heatmap to obtain the 3D coordinates.

[0053] Optionally, in one embodiment, an overall loss function including two-dimensional context loss and three-dimensional localization loss can be defined, and an integrated training method can be used to simultaneously optimize the generation of two-dimensional cone heatmaps and the localization of three-dimensional coordinates, without the need for a pre-trained two-dimensional feature extraction backbone network.

[0054] To better understand the technical solution of this application, a specific embodiment will be used for illustration below. The details listed in this example are mainly for ease of understanding and are not intended to limit the scope of protection of this application.

[0055] This embodiment enhances the features of other vertebrae by utilizing the location information and visual features (indicated by user-marked points) of a reference vertebra. Subsequently, a set of two-dimensional vertebrae contexts is captured through a series of learnable vertebrae representations, generating a set of two-dimensional vertebrae heatmaps. These two-dimensional heatmaps are integrated with features from high-resolution multi-view images to construct pseudo-3D volumes for estimating the 3D coordinates of the vertebrae. Unlike previous vertebrae localization methods that require pre-training of a two-dimensional feature extraction backbone network, this embodiment utilizes simultaneous learning of two-dimensional vertebrae context and 3D vertebrae location to achieve integrated training.

[0056] Perform two-dimensional visual feature extraction first. This is shared on the anteroposterior and lateral spine radiographs. For a given pair of input images, the set can be represented as the anteroposterior and lateral views, each with a dimension of 512×512 pixels and 1 channel. To capture multi-scale visual cues, a two-dimensional U-shaped structure can be used, which contains four levels and a spatial pyramid pooling module for feature integration. Subsequently, the two-dimensional visual feature extraction unit generates a pair of feature maps corresponding to the respective lateral and anteroposterior views, with 64 channels. For long images with a height exceeding 512 pixels, a sliding window technique can be used to sample consecutive image patches.

[0057] Then perform point-guided feature enhancement. For a given pair of input images, the point-guided feature enhancement unit allows the user to specify dot-like markers representing the center of each vertebra in each image at the corresponding spine level. These marker points are used as references to guide the network to predict the positions of the centers of the vertebrae at the remaining levels. For long images divided into a series of consecutive image patches, the bottom vertebra in the previous patch is used as the guiding point for the subsequent patch.

[0058] The guiding points can be represented as coordinates (x0, y0), where x0 and y0 correspond to the horizontal and vertical coordinates of the point marker respectively. The input guiding points can be preprocessed first to create two types of masks: a binary guiding mask and a one-way distance mask. Specifically, the binary guiding mask is constructed as follows:

[0059] Mask = {

[0060] 1, if (x0 - r ≤ x < x0 + r) and (y0 - r ≤ y < y0 + r) [[ID=--]] [[ID=--]]

[0061] 0, otherwise

[0062] }

[0063] where (x, y) represents pixel coordinates, and r = 40 is an empirical value used to define the range that can cover the area of the reference vertebra.

[0064] Assuming the feature representation comes from the features of the two-dimensional visual feature extraction unit, the guiding mask can be used to obtain the guiding features, that is, the non-zero feature values within the masked area.

[0065] At the same time, the one-way distance mask is calculated as follows:

[0066] Distance mask = the vertical distance of each pixel point to the y coordinate of the guiding point

[0067] This distance mask is crucial for establishing cross-view correspondences.

[0068] After the preprocessing step, both the features and the guiding features are downsampled by a ratio of 8 and then fed into a 4-layer attention mechanism block with 8 heads and 64 hidden dimensions. In the attention block, the guiding features serve as both keys and values, enhancing the repetitive cone features present in the original features. The output of the attention layer can be represented as follows:

[0069] Output = Attention Weight Calculation(Query, Key, Value)

[0070] The query comes from the original feature, while the key and value come from the guiding feature.

[0071] Finally, the point-guided feature enhancement unit generates a set of enhanced features, which are used by the semantic context extraction unit to extract important contextual information of the remaining vertebrae.

[0072] Semantic context extraction is then performed. Spatial context information can be aggregated using learnable cone representations. These cone representations are structured into a series of spatial feature maps with a spatial dimension of 16×16, 64 channels, and a maximum number of n predicted cones.

[0073] In traditional attention mechanisms, computational costs increase quadratically with the spatial dimensions of the key and query matrices. To alleviate this problem, this application provides a novel architectural variant called the Dynamic Sparse Attention Mechanism. This mechanism reduces computational costs by introducing dual attention computation. Specifically, in the first attention layer, it automatically distinguishes foreground regions from the background using low-resolution feature maps; while in the second attention layer, it optimizes the computation process by processing only selected high-resolution but sparse foreground features to extract the cone context.

[0074] Mathematically, the enhanced features and distance mask can be combined as follows:

[0075] Mask feature = Enhanced feature × Binary distance mask + Distance mask

[0076] Binary distance mask (x, y) = {

[0077] 0, if distance mask (x,y) = 0

[0078] 1. If the distance mask (x, y) > 0

[0079] }

[0080] Here, the mask features integrate the vertical distance of each pixel to the reference cone indicated by the guide point, thereby facilitating the spatial alignment of two-dimensional features extracted from different views.

[0081] Next, the masked features are subjected to max pooling with a ratio of α to obtain low-resolution features, which are then fed into the first attention layer of the dynamic sparse attention mechanism. Specifically, in the first attention layer, the attention matrix is ​​calculated as follows:

[0082] Attention Matrix 1 = Attention Weight Calculation (Cone represents query, low-resolution feature key)

[0083] After attention computation, the elements of the attention matrix can be sorted in descending order according to their attention scores. This sorting operation allows the indices of the top k elements with the highest scores to be identified, generating a set of high-attention indices. These indices can then be mapped to corresponding positions on high-resolution mask features using a mapping relationship, resulting in a new set of indices, which will be used to extract a subset of foreground features. Subsequently, these foreground features are fed into the second attention layer of the dynamic sparse attention mechanism for fine-grained attention computation, where the attention matrix is ​​computed as follows:

[0084] Attention Matrix 2 = Attention Weight Calculation (Cone represents query, foreground feature key)

[0085] Finally, the output of the dynamic sparse attention mechanism is a set of context-rich cone feature representations, which are transformed into predicted two-dimensional cone heatmaps through linear layers.

[0086] In summary, the computational complexity of attention in a standard attention mechanism with L attention layers is O(LNMd), where N represents the spatial dimension of the mask features, d is the hidden dimension, and M represents the product of the spatial dimension and the total number of cone representations. In contrast, the proposed dynamic sparse attention mechanism significantly reduces the computational complexity to L / 2[O1(NMd / α]] while using the same number of attention layers. 2 )+O2(kαMd)]. Here, the first term corresponds to attention computation on low-resolution features, used to identify the foreground cone region. The second term represents fine attention computation on high-resolution but sparse foreground features, aiming to improve localization accuracy while maintaining low computational cost.

[0087] Next, 3D multi-view feature fusion is performed. The predicted 2D vertebral heatmaps from the lateral and anteroposterior views are rescaled to match the input size. These heatmaps are combined with their respective feature maps and forwarded to the 3D feature fusion unit. In this unit, features are projected onto a fixed-size volume using triangulation, and the volumes from the multiple views are then aggregated and processed through 3D convolution to output heatmaps representing the 3D vertebral positions. These 3D vertebral heatmaps are then processed through a soft-weighted summation function, which converts the heatmaps into precise vertebral coordinates. Finally, the 3D feature fusion unit generates a set of predicted vertebral 3D coordinates.

[0088] To predict the location of the two-dimensional vertebral body, assuming the actual two-dimensional vertebral body heatmap size on each ray image is n×16×16, the mean squared error loss and similarity loss can be calculated as follows:

[0089] Mean squared error loss = (1 / (n×16×16)) × sum of squares of the differences between the predicted and true values ​​of all pixels

[0090] as well as

[0091] Similarity loss = 1 - (2 × dot product of predicted and true values) / (sum of squares of predicted values ​​+ sum of squares of true values ​​+ smoothing term)

[0092] The predicted value and the true value represent the true and predicted probabilities of the pixel at position (j,k) of the i-th vertebra.

[0093] Therefore, a two-dimensional context loss can be obtained, as shown below:

[0094] Two-dimensional context loss = mean squared error loss + similarity loss

[0095] To predict the 3D coordinates of the vertebrae, we assume the true vertebrae positions are a set of 3D coordinates, where m is the number of vertebrae present in the current input. The mean squared error loss can be calculated for each vertebra, as shown below:

[0096] 3D localization loss = (1 / m) × the sum of the squares of the differences between the predicted and actual coordinates of all vertebrae. Therefore, the overall detection loss is:

[0097] Overall loss = 2D context loss + 3D localization loss

[0098] This embodiment uses a synthetic biplane spinal digital reconstruction radiograph dataset. This dataset contains 337 pairs of biplane spinal radiographs, each pair corresponding to one tomographic volume. The number of vertebrae in each radiograph ranges from 3 to 24. 95% of the dataset is used as the training and testing set (8:2 ratio), and 5% is used for validation.

[0099] The evaluation metrics used in this embodiment are as follows:

[0100] 1. Percentage of Correctly Labeled Positions: Defined as the percentage of predicted cone positions within a given Euclidean distance threshold, compared to the corresponding true positions. The calculation formula is:

[0101] Percentage of correctly labeled tags = (1 / N) × Number of tags that meet the threshold condition

[0102] The threshold condition is as follows: if the Euclidean distance between the predicted position and the actual position is less than the preset threshold τ, the value is 1; otherwise, it is 0. N represents the total number of labeled vertebral positions.

[0103] 2. Average position error: The average Euclidean distance in millimeters during three-dimensional measurement. The calculation formula is:

[0104] Average position error = (1 / N) × the sum of the Euclidean distances between the predicted and actual positions of all vertebrae.

[0105] 3. Area under the curve: A measure of the region enclosed by a properly labeled percentage curve between the upper and lower limits on the horizontal axis.

[0106] For three-dimensional vertebral body localization assessment, the threshold τ is set to 10 mm and 20 mm, respectively. Similarly, for the estimation of the area under the curve in three dimensions, the upper limit can be set to 10 mm and the lower limit to 50 mm.

[0107] The technical solution of this embodiment has been developed using a deep learning framework in a high-level programming language and trained on a workstation equipped with a high-performance graphics processor. The input image size is 512×512. The network is trained uniformly from scratch using an adaptive optimizer, with a weight decay of 0.05, a batch size of 2, and approximately 100 training epochs.

[0108] For parameter tuning, the maximum number of predicted cones can be set to n=10, and the dynamic sparse attention mechanism parameters can be α=32 and k=8 for the synthetic dataset. In contrast, for real-world testing, n=5, α=8, and k=4 can be used. These parameter choices are guided by an approximate estimate of the foreground-to-background ratio present in each dataset.

[0109] During the training phase, random cropping can be incorporated as a data augmentation technique, and the centroid of the top-level vertebrae in each input lateral and anteroposterior view can be selected as a guide point. During inference, the user can input two point-like matching markers, corresponding to the same-level vertebrae in the lateral and anteroposterior views of the ray image, respectively. The network simultaneously generates n heatmaps and n 3D vertebrae coordinates for each ray view. For each predicted heatmap, if the maximum probability is higher than a threshold of 0.5, the vertebra is considered to exist. Therefore, the maximum probability value can be used as a criterion for evaluating the effectiveness of the 3D coordinate predictions.

[0110] In other embodiments, the spatial dimension of the vertebral feature representation can also be set to 4×4, 8×8, or 16×16, corresponding to the resolution of the predicted two-dimensional vertebral heatmap. Higher resolution provides more contextual information but also increases computational cost.

[0111] In other embodiments, the pooling ratio α can also be set to 1, 2, or 4. A larger α corresponds to a larger selected area for fine-grained attention calculations, allowing more local context to be aggregated, thereby improving localization accuracy.

[0112] In other embodiments, the first k value can also be set to 2, 4, or 8. A larger k indicates that more features are involved in the fine attention calculation.

[0113] In other embodiments, the two-dimensional visual feature extraction unit may also use different backbone network architectures, such as residual networks, densely connected networks, or efficient networks, to extract multi-scale visual features.

[0114] In other embodiments, in addition to single-point guidance, this method can be extended to support multi-point guidance, allowing users to mark multiple reference cones on each view, further enhancing feature extraction and contextual understanding.

[0115] Compared with the prior art, the technical solution of this application has the following advantages:

[0116] 1. High Detection Accuracy: This method demonstrates superior vertebral body localization accuracy on large synthetic datasets, achieving a significantly higher percentage of correct labels than existing methods under both low and high threshold evaluation conditions. The system exhibits a small mean position error, and the area under the curve (AUC) is close to the ideal value, indicating highly accurate and stable localization results. Testing on real-world clinical datasets also validates the superior performance of this method, confirming its practical value and potential for widespread application in real-world scenarios.

[0117] 2. Effective handling of occlusion and deformation: By integrating point-guided reference vertebrae and vertebral semantic context into the detection framework, this application can effectively handle challenging situations such as vertebral occlusion, spinal deformities, and metal implants.

[0118] 3. Computational efficiency: By employing dual attention computation in the dynamic sparse attention mechanism, this application significantly reduces the complexity of attention computation while maintaining high detection accuracy.

[0119] 4. Integrated Training: Unlike existing methods that require pre-training of a two-dimensional feature extraction backbone network, this application supports integrated training, which simplifies the training process and improves performance.

[0120] 5. Cross-view semantic consistency: Through unidirectional distance masking and point-guided feature enhancement, this application can effectively solve the problem of viewpoint differences between multi-view ray images and establish a more accurate cross-view semantic correspondence.

[0121] The technical solution of this application can be used for real-time spinal navigation, spinal morphology assessment, education and training, postoperative assessment (by comparing the vertebral positions before and after surgery to assess the surgical effect and the degree of spinal correction), etc.

[0122] Accordingly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the various method embodiments of this application. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0123] Furthermore, embodiments of this application also provide a dual-plane imaging three-dimensional localization system for spinal key points, including a memory for storing computer-executable instructions and a processor; the processor is used to implement the steps in the above-described method embodiments when executing the computer-executable instructions in the memory. The processor may be a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Microcontroller Unit (MCU), Neural Processing Unit (NPU), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), or other programmable logic devices. The aforementioned memory may be read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or solid-state drive, etc. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0124] Furthermore, embodiments of this application also provide a computer program product, including computer-executable instructions that, when executed by a processor, implement the steps in the above-described method embodiments.

[0125] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this application, if it refers to performing an action according to an element, it means performing the action at least according to that element, including two cases: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0126] The numbering used in describing the steps of a method does not inherently limit the order of these steps. For example, a step with a higher number does not necessarily have to be executed after a step with a lower number; it can be executed first and then second, or even in parallel, as long as this execution order is reasonable to someone skilled in the art. Similarly, multiple steps with consecutively numbered sequences (e.g., step 101, step 102, step 103, etc.) do not restrict other steps from being executed between them; for example, there can be other steps between step 101 and step 102.

[0127] This specification includes combinations of various embodiments described herein. Individual references to embodiments are made (e.g., "one embodiment," "some embodiments," or "preferred embodiments"); however, these embodiments are not mutually exclusive unless indicated to be mutually exclusive or are readily apparent to those skilled in the art. It should be noted that the word "or" is used in a non-exclusive sense throughout this specification unless the context explicitly indicates or requires it.

[0128] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for three-dimensional localization of key points of the spine using dual-plane imaging, characterized in that, include: Obtain biplane spinal radiographs of the object to be processed, including at least anteroposterior and lateral views; Receive user-specified point guide markers on the dual-plane spinal radiographs, corresponding to the same level of reference vertebrae in the anterior-posterior and lateral views, respectively; Based on the biplane spinal ray images, the initial two-dimensional visual features of each view are extracted; Based on the position of the point guidance marker, a guidance mask for the reference vertebra is generated; using the guidance mask and the initial two-dimensional visual features of the reference vertebra, the initial two-dimensional visual features corresponding to other vertebrae in the dual-plane image are enhanced through a first attention calculation to obtain enhanced features; Using the learnable vertebral representation and the enhanced features, the semantic context of the vertebral body is extracted through a dynamic sparse attention mechanism to obtain two-dimensional vertebral heatmaps of the anterior and posterior views and the lateral views. The two-dimensional vertebral heatmaps of the anterior and posterior and lateral views are fused with their respective enhanced features or initial two-dimensional visual features, and a pseudo-three-dimensional feature volume is constructed by triangulation or feature projection methods. The pseudo-3D feature volume is processed to obtain the 3D coordinates of the other vertebrae in the dual-plane image, excluding the reference vertebra.

2. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 1, characterized in that, The dynamic sparse attention mechanism includes: downsampling the enhanced features to obtain low-resolution features; identifying high-attention region indices through a second attention calculation based on the low-resolution features; mapping the high-attention region indices back to the un-downsampled enhanced features to obtain a sparse high-resolution foreground feature subset; and generating context-rich cone feature representations through a third attention calculation based on the sparse high-resolution foreground feature subset, and converting them into two-dimensional cone heatmaps for each view.

3. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 2, characterized in that, In the dynamic sparse attention mechanism, the learnable cone representation serves as the query, the low-resolution features serve as the key for the second attention computation, and the sparse high-resolution foreground feature subset serves as the key for the third attention computation.

4. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 1, characterized in that, In the first attention calculation, the initial two-dimensional visual features corresponding to the other vertebrae are used as queries, and the guiding features extracted from the initial two-dimensional visual features of the reference vertebrae using the guiding mask are used as keys and values.

5. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 1, characterized in that, The boot mask includes at least a binary boot mask and a one-way distance mask; The one-way distance mask contains vertical distance information from each pixel to the point guide marker, and is combined with the enhanced features before the dynamic sparse attention mechanism extracts the cone semantic context.

6. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 1, characterized in that, The step of extracting the initial two-dimensional visual features of each view uses a two-dimensional U-shaped network structure to extract multi-scale visual features.

7. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 1, characterized in that, The steps for processing the pseudo-3D feature volume include: using 3D convolution to process the aggregated multi-view feature volume to obtain a 3D cone heatmap; and processing the 3D cone heatmap using a soft weighted summation function to obtain the 3D coordinates.

8. The method for three-dimensional localization of key spinal points using dual-plane imaging as described in claim 1, characterized in that, The method further includes: defining an overall loss function that includes two-dimensional context loss and three-dimensional localization loss, and using an integrated training method to simultaneously optimize the generation of the two-dimensional cone heat map and the localization of the three-dimensional coordinates, without the need for a pre-trained two-dimensional feature extraction backbone network.

9. A three-dimensional positioning system for key points of the spine using dual-plane imaging, characterized in that, include: Memory is used to store executable instructions for a computer; as well as, A processor, coupled to the memory, is configured to implement the steps of the method as described in any one of claims 1 to 8 when executing the computer-executable instructions.

10. A computer program product comprising computer-executable instructions, characterized in that, When executed by a processor, the computer-executable instructions implement the steps of the method according to any one of claims 1 to 8.

Images