Double-plane image spine key point three-dimensional positioning method and system

Abstract

The invention relates to the field of medical treatment, discloses a biplane image spine key point three-dimensional positioning method and system, and can improve the accuracy and robustness of spine key point three-dimensional positioning under a biplane spine ray image. According to the method, a biplanar spine ray image containing anterior-posterior position and lateral position views is obtained, and point guide marks specified by a user on a reference vertebral body of the same level are received. The initial two-dimensional visual features of each view are extracted, a guide mask is generated according to the position of a point guide mark, first attention calculation is performed by using the guide mask and the initial features of a reference vertebral body, and the two-dimensional visual features corresponding to other vertebral bodies are enhanced. And extracting a centrum semantic context through a dynamic sparse attention mechanism by adopting learnable centrum representation and enhanced features, and obtaining a two-dimensional centrum heat map of anterior-posterior view and lateral view. And fusing the heat map with the corresponding features, constructing a pseudo three-dimensional feature volume, and processing the volume to obtain accurate three-dimensional coordinates of other vertebral bodies except the reference vertebral body.

Description

Technical Field

[0001] The present application relates to the medical field, and in particular to a three-dimensional spinal positioning technology based on medical imaging. Background Art

[0002] This section is intended to provide a background or context for understanding the embodiments of the present application and is for reference only. The applicant should not be considered to admit that this section belongs to the prior art that has been disclosed before the filing date of this application.

[0003] In orthopedics, accurately understanding the three-dimensional spatial position of vertebrae is crucial. X-ray fluoroscopy is commonly used due to its flexibility, relatively low radiation dose, and cost-effectiveness. Biplanar imaging, which acquires X-ray images of the patient from two different angles (usually the standard anteroposterior (AP) and lateral (LAT) views), is a common method for obtaining target three-dimensional information.

[0004] However, X-ray images are projections of 3D anatomical structures onto a 2D plane. This inherent property makes accurately reconstructing or localizing 3D structures from 2D images, especially for morphologically similar and closely packed vertebrae, challenging. Existing 3D vertebral localization methods based on biplane imaging can be roughly divided into two categories.

[0005] The first category is 2D mapping-based methods. These methods typically first detect or mark key points of the vertebra (such as the vertebral center or corners) in each 2D view. They then use triangulation principles combined with optimization algorithms (such as the least squares method or methods based on statistical shape models) to reconstruct the 3D coordinates of these key points.

[0006] The second category is direct volume reconstruction-based methods. These methods attempt to construct a pseudo-three-dimensional feature volume from the two-dimensional image features of multiple views, and then directly regress the position of each key point in the three-dimensional voxel space or its likelihood map through operations such as three-dimensional convolution. When constructing these methods or evaluating their performance, digitally reconstructed radiographs (DRRs) generated by computed tomography (CT) data are sometimes used. DRRs are X-ray images simulated from CT volume data and can be used to generate training or test data with accurate three-dimensional annotations.

[0007] Although existing technologies have made certain progress, the above methods still have some limitations that are difficult to overcome in practical applications:

[0008] 1. Depth information uncertainty: Due to the projection characteristics of X-ray images, the overlapping of anatomical structures, and the low contrast between the vertebral body and the surrounding soft tissue, it is difficult to accurately infer depth information from a single 2D view. This is especially true when the vertebral body is diseased or obscured (such as by other bones or metal implants), which can severely affect the accuracy of 3D positioning.

[0009] 2. Difficulty distinguishing vertebrae: The spine is a chain-like structure composed of multiple morphologically similar vertebral units connected in series. Particularly in the thoracic and lumbar spine, adjacent vertebrae are highly similar in morphology and imaging appearance. This makes it extremely difficult to accurately distinguish and map the same vertebrae in different views on 2D images. In the presence of spinal deformities or metal internal fixation devices, vertebral morphology may be altered or obscured, further exacerbating the difficulty of differentiation.

[0010] 3. Difficulty in ensuring view consistency: X-ray images taken from different viewpoints (e.g., AP and LAT) will show changes in the projection morphology and relative position of the same vertebra. When processing each view separately, effectively establishing and ensuring a correct one-to-one correspondence between vertebral key points detected in different views is a critical and complex issue.

[0011] 4. Balancing Computational Resource Consumption and Accuracy: For volume-based reconstruction methods, positioning accuracy depends heavily on the spatial resolution of the reconstructed volume. Increasing resolution for more precise positioning often results in a dramatic increase in computational costs (including memory consumption and computation time), sometimes quadratically or cubically. This limits their application in intraoperative scenarios requiring real-time feedback. Summary of the Invention

[0012] The purpose of this application is to provide a method and system for three-dimensional positioning of key points of the spine in dual-plane imaging, which can improve the accuracy and robustness of three-dimensional positioning of key points of the spine under dual-plane spinal X-ray images, especially the positioning capability in complex clinical situations such as occlusion and deformity.

[0013] This application discloses a method for three-dimensional positioning of key points of the spine in dual-plane imaging, comprising:

[0014] Obtaining a biplane spinal radiographic image of the subject to be treated, including at least an anteroposterior and lateral views;

[0015] receiving a user-specified point guide marker on the biplane spinal radiograph corresponding to a reference vertebra at the same level in the anteroposterior and lateral views, respectively;

[0016] extracting initial two-dimensional visual features of each view based on the biplane spinal radiographs;

[0017] generating a guidance mask for the reference vertebra based on the position of the point guidance marker; enhancing the initial two-dimensional visual features corresponding to other vertebrae in the biplane image by first attention calculation using the guidance mask and the initial two-dimensional visual features of the reference vertebra to obtain enhanced features;

[0018] Utilizing the learnable vertebral representation and the enhanced features, a dynamic sparse attention mechanism is used to extract vertebral semantic context to obtain two-dimensional vertebral heat maps in the anteroposterior and lateral views;

[0019] fusing the two-dimensional vertebral heat maps of the anteroposterior and lateral views with the corresponding enhanced features or the initial two-dimensional visual features, and constructing a pseudo three-dimensional feature volume by triangulation or feature projection method;

[0020] The pseudo three-dimensional feature volume is processed to obtain the three-dimensional coordinates of other vertebrae except the reference vertebrae in the biplane image.

[0021] In a preferred example, the dynamic sparse attention mechanism includes: downsampling the enhanced features to obtain low-resolution features, identifying high-attention area indexes through a second attention calculation based on the low-resolution features; mapping the high-attention area indexes back to the non-downsampled enhanced features to obtain a sparse high-resolution foreground feature subset; generating a context-rich conical feature representation through a third attention calculation based on the sparse high-resolution foreground feature subset, and converting it into a two-dimensional conical heat map of each view.

[0022] In a preferred example, in the dynamic sparse attention mechanism, the learnable pyramidal representation is used as a query, the low-resolution features are used as keys for the second attention calculation, and the sparse high-resolution foreground feature subset is used as a key for the third attention calculation.

[0023] In a preferred example, in the first attention calculation, the initial two-dimensional visual features corresponding to the other vertebrae are used as queries, and the guiding features extracted from the initial two-dimensional visual features of the reference vertebrae using the guiding mask are used as keys and values.

[0024] In a preferred embodiment, the guide mask includes at least a binary guide mask and a one-way distance mask;

[0025] The unidirectional distance mask contains the vertical distance information of each pixel to the point guide marker and is combined with the enhanced features before the dynamic sparse attention mechanism extracts the pyramidal semantic context.

[0026] In a preferred embodiment, the step of extracting the initial two-dimensional visual features of each view uses a two-dimensional U-shaped network structure to extract multi-scale visual features.

[0027] In a preferred example, the step of processing the pseudo three-dimensional feature volume includes: using three-dimensional convolution to process the aggregated multi-view feature volume to obtain a three-dimensional vertebral heat map; and processing the three-dimensional vertebral heat map through a soft weighted sum function to obtain the three-dimensional coordinates.

[0028] In a preferred example, the method also includes: defining an overall loss function including two-dimensional context loss and three-dimensional positioning loss, and using an integrated training method to simultaneously optimize the generation of the two-dimensional vertebral heat map and the positioning of the three-dimensional coordinates, without the need for a pre-trained two-dimensional feature extraction backbone network.

[0029] This application also discloses a dual-plane imaging spine key point three-dimensional positioning system, comprising:

[0030] a memory for storing computer-executable instructions; and

[0031] A processor, coupled to the memory, is configured to implement the steps in the method described above when executing the computer-executable instructions.

[0032] The present application also discloses a computer program product, comprising computer executable instructions, which implement the steps of the method described above when executed by a processor.

[0033] In the implementation of the present application, by receiving the point guidance mark corresponding to the reference vertebra of the same level specified by the user on the dual-plane image, and generating a guidance mask based on this mark, combining the first attention calculation to enhance the initial two-dimensional visual features of other vertebrae, and then using the learnable vertebrae representation and enhanced features, the vertebrae semantic context is extracted through the dynamic sparse attention mechanism to obtain a two-dimensional vertebrae heat map, and finally the two-dimensional heat map is fused with the features to construct a pseudo three-dimensional feature volume and process it to obtain the three-dimensional coordinates of other vertebrae. The overall method flow can effectively integrate the user's prior knowledge (point guidance) and the feature extraction capability of the deep learning network, and realize the mapping from sparse user input to precise three-dimensional positioning by guiding and enhancing features layer by layer and extracting context information. This method can overcome the problems of depth uncertainty, difficulty in distinguishing vertebrae, difficulty in ensuring perspective consistency, and high consumption of computing resources in traditional methods. Specifically, point guidance provides a clear starting point and reference, reducing the blindness of the network's search throughout the spine; feature enhancement based on reference vertebrae helps to distinguish vertebrae with similar morphology; the extraction of semantic context can utilize the anatomical relationship between vertebrae; and the subsequent multi-view fusion and 3D reconstruction integrate information from different planes, thereby improving the accuracy, robustness and computational efficiency of 3D positioning even in complex situations (such as occlusion, deformity, and metal implants).

[0034] Furthermore, by adopting a specific dual attention processing flow in the dynamic sparse attention mechanism to downsample the enhanced features, perform a second attention calculation based on the low-resolution features to identify the high-attention area index, and then map the index back to the non-downsampled enhanced features to obtain a sparse high-resolution foreground feature subset, and finally perform a third attention calculation based on the subset to generate a context-rich vertebral feature representation, it can significantly reduce the amount of computation while still focusing on the foreground feature area that is most important for vertebral positioning and perform refined context extraction, thereby significantly improving the computational efficiency of the model while ensuring high positioning accuracy.

[0035] Furthermore, by assigning specific roles in the dynamic sparse attention mechanism, namely, using the learnable vertebral representation as the query, the downsampled low-resolution features as the key for the second attention calculation, and the sparse high-resolution foreground feature subset as the key for the third attention calculation, the attention mechanism can efficiently scan the global information using low-resolution features to identify potential foreground areas in the first stage, and combine the learnable vertebral anatomical prior (as the query) and the filtered high-resolution foreground features (as the key) to extract fine-grained contextual information more accurately and targetedly in the second stage, thereby optimizing the effectiveness and pertinence of the dual attention calculation.

[0036] Furthermore, by using the initial two-dimensional visual features corresponding to other vertebrae as queries in the first attention calculation of the point-guided feature enhancement unit, and using the guiding features extracted from the initial two-dimensional visual features of the reference vertebra using the guiding mask as both the key and value settings, the network can effectively utilize the user-specified reference vertebra information (guiding features) to enhance and associate the feature expressions of other vertebrae to be detected, thereby strengthening the guiding role of the reference point in the learning of the entire spinal feature and improving the recognition and relevance of the features.

[0037] Furthermore, by including not only a binary guidance mask but also a unidirectional distance mask in the guidance mask, and combining this unidirectional distance mask with the enhanced features before the dynamic sparse attention mechanism extracts semantic context, explicit vertical spatial distance prior information about the reference vertebral position can be incorporated into the features. This not only facilitates feature extraction within the reference region delineated by the binary mask, but also provides the network with the relative positional relationship between different pixels and the reference point. This is particularly helpful in establishing more accurate semantic correspondence and spatial alignment between different views, enhancing the perception and utilization of the relative positions of the vertebral bodies in subsequent semantic context extraction.

[0038] Furthermore, by adopting a two-dimensional U-shaped network structure in the step of extracting the initial two-dimensional visual features of each view, its encoder-decoder structure and skip connection characteristics can be utilized to effectively capture and integrate multi-scale visual cues in the image, thereby providing a richer and more robust initial feature map for subsequent point-guided feature enhancement and semantic context extraction, which helps to improve the overall positioning accuracy.

[0039] Furthermore, by first using three-dimensional convolution to process the aggregated multi-view feature volume to obtain a three-dimensional vertebral heat map when processing the pseudo three-dimensional feature volume, and then processing the heat map through a soft weighted sum function to obtain three-dimensional coordinates, the network can learn the probability distribution (heat map) in three-dimensional space from the fused multi-view information, and convert the probability distribution into precise, sub-pixel three-dimensional coordinates through a differentiable soft weighted sum method, thereby improving the accuracy and continuity of the final three-dimensional positioning result.

[0040] Furthermore, by defining an overall loss function that includes two-dimensional context loss and three-dimensional positioning loss, and adopting an integrated training method to simultaneously optimize two-dimensional vertebral heat map generation and three-dimensional coordinate positioning, and without the need for a pre-trained two-dimensional feature extraction backbone network, end-to-end training can be achieved, which simplifies the model training process, avoids the suboptimal solution problem that may be caused by staged training, and promotes more comprehensive collaborative learning between network modules, thereby improving the performance and convergence efficiency of the overall model.

[0041] The various technical features disclosed in the above invention content, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (all of which should be deemed to have been recorded in this specification), unless such a combination of technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed. Features C and D are equivalent technical means that play the same role. Technically, only one of them needs to be used, and it is impossible to use them at the same time. Feature E can be technically combined with feature C. In this case, the solution of A+B+C+D should not be deemed to have been recorded because it is technically infeasible, while the solution of A+B+C+E should be deemed to have been recorded. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a method for three-dimensional positioning of spine key points using dual-plane imaging according to one embodiment of the present application. DETAILED DESCRIPTION

[0043] In the following description, many technical details are provided to help readers better understand this application. However, those skilled in the art will understand that even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in this application can be implemented.

[0044] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0045] One embodiment of the present application relates to a method for three-dimensional positioning of key points of the spine in dual-plane imaging, the process of which is as follows: Figure 1 As shown, the method includes the following steps:

[0046] In step 101, a biplane spinal radiographic image of a subject to be treated, including at least an anteroposterior view and a lateral view, is obtained.

[0047] Thereafter, the process proceeds to step 102 to receive point guide marks designated by the user on the biplane spinal radiograph, which correspond to reference vertebrae at the same level in the anteroposterior and lateral views, respectively.

[0048] Then, the process proceeds to step 103, where initial two-dimensional visual features of each view are extracted based on the bi-plane spinal radiographs. Preferably, a two-dimensional U-shaped network structure can be used to extract multi-scale visual features.

[0049] Thereafter, step 104 is entered to generate a guidance mask for the reference vertebra based on the position of the point guidance marker. Using the guidance mask and the initial two-dimensional visual features of the reference vertebra, the initial two-dimensional visual features corresponding to other vertebrae in the dual-plane image are enhanced through the first attention calculation to obtain enhanced features. Preferably, in the first attention calculation, the initial two-dimensional visual features corresponding to other vertebrae are used as queries, and the guidance features extracted from the initial two-dimensional visual features of the reference vertebra using the guidance mask are used as keys and values. Preferably, the guidance mask includes at least a binary guidance mask and a one-way distance mask. The one-way distance mask contains the vertical distance information from each pixel to the point guidance marker, and is combined with the enhanced features before the dynamic sparse attention mechanism extracts the semantic context of the vertebra.

[0050] Thereafter, step 105 is entered, and the semantic context of the vertebra is extracted by a dynamic sparse attention mechanism using the learnable vertebral representation and enhanced features to obtain a two-dimensional vertebral heat map for the anteroposterior and lateral views. Preferably, the dynamic sparse attention mechanism includes: downsampling the enhanced features to obtain low-resolution features, identifying high-attention region indexes through a second attention calculation based on the low-resolution features; mapping the high-attention region indexes back to the non-downsampled enhanced features to obtain a sparse high-resolution foreground feature subset; generating a context-rich vertebral feature representation through a third attention calculation based on the sparse high-resolution foreground feature subset, and converting it into a two-dimensional vertebral heat map for each view. In the dynamic sparse attention mechanism, the learnable vertebral representation is used as a query, the low-resolution features are used as keys for the second attention calculation, and the sparse high-resolution foreground feature subset is used as keys for the third attention calculation.

[0051] Then, step 106 is entered to fuse the two-dimensional vertebral heat maps of the anteroposterior and lateral views with their corresponding enhanced features or initial two-dimensional visual features, and construct a pseudo three-dimensional feature volume through triangulation or feature projection method.

[0052] Then, step 107 is performed to process the pseudo 3D feature volume to obtain the 3D coordinates of the vertebrae other than the reference vertebra in the biplane image. Preferably, the aggregated multi-view feature volume can be processed using 3D convolution to obtain a 3D vertebral heat map, and the 3D vertebral heat map can be processed using a soft weighted sum function to obtain the 3D coordinates.

[0053] Optionally, in one embodiment, an overall loss function including two-dimensional context loss and three-dimensional positioning loss can be defined, and an integrated training method can be used to simultaneously optimize the generation of two-dimensional vertebral heat maps and the positioning of three-dimensional coordinates without the need for a pre-trained two-dimensional feature extraction backbone network.

[0054] In order to better understand the technical solution of the present application, a specific embodiment is provided below for illustration. The details listed in this example are mainly for ease of understanding and are not intended to limit the scope of protection of the present application.

[0055] This embodiment enhances the features of vertebrae at other levels by utilizing the position information and visual features of the reference vertebra (indicated by user markers). Subsequently, the two-dimensional vertebral context is captured through a series of learnable vertebral representations to generate a set of two-dimensional vertebral heat maps. These two-dimensional heat maps are integrated with high-resolution multi-view image features to construct a pseudo three-dimensional volume for estimating the three-dimensional coordinates of the vertebra. Unlike previous vertebral localization methods that require pre-training of a two-dimensional feature extraction backbone network, this embodiment utilizes simultaneous learning of two-dimensional vertebral context and three-dimensional vertebral position to achieve integrated training.

[0056] First, perform two-dimensional visual feature extraction, which is shared on the anteroposterior and lateral spine radiographic images. For a given pair of input images, the set can be represented as the anteroposterior and lateral views, each with a dimension of 512×512 pixels and 1 channel. To capture multi-scale visual cues, a two-dimensional U-shaped structure can be used, which contains four levels and a spatial pyramid pooling module for feature integration. Subsequently, the two-dimensional visual feature extraction unit generates a pair of feature maps corresponding to the respective lateral and anteroposterior views, with 64 channels. For long images with a height exceeding 512 pixels, a sliding window technique can be used to sample consecutive image patches.

[0057] Next, perform point-guided feature enhancement. For a given pair of input images, the point-guided feature enhancement unit allows the user to specify dot markers representing the center of each vertebra in each image at the corresponding spinal level. These marker points are used as references to guide the network to predict the positions of the centers of the vertebrae at the remaining levels. For long images divided into a series of consecutive image patches, the bottom vertebra in the previous patch is used as the guiding point for the subsequent patch.

[0058] The guiding point can be represented as coordinates (x0, y0), where x0 and y0 correspond to the horizontal and vertical coordinates of the point marker respectively. The input guiding point can be preprocessed first to create two types of masks: a binary guiding mask and a one-way distance mask. Specifically, the binary guiding mask is constructed as follows:

[0059] Mask = {

[0060] 1, if (x0 - r ≤ x < x0 + r) and (y0 - r ≤ y < y0 + r)

[0061] 0, otherwise

[0062] }

[0063] where (x, y) represents pixel coordinates, and r = 40 is an empirical value used to define the range that can cover the reference vertebral body area.

[0064] Assuming the feature representation comes from the features of the two-dimensional visual feature extraction unit, the guiding mask can be used to obtain the guiding features, that is, the non-zero feature values within the mask area.

[0065] At the same time, the one-way distance mask is calculated as follows:

[0066] Distance mask = the vertical distance of each pixel point to the y coordinate of the guiding point

[0067] This distance mask is crucial for establishing cross-view correspondence.

[0068] After the preprocessing step, both the features and the guide features are downsampled by a factor of 8 and then fed into a 4-layer attention block with 8 heads and 64 hidden dimensions. In the attention block, the guide features serve as both keys and values, enhancing the recurring cone features present in the original features. The output of the attention layer can be represented as follows:

[0069] Output = attention weight calculation (query, key, value)

[0070] The query comes from the original feature, and the key and value come from the guided feature.

[0071] Finally, the point-guided feature enhancement unit generates a set of enhanced features, which are passed through the semantic context extraction unit to extract important contextual information of the remaining vertebrae.

[0072] After that, semantic context extraction is performed. Spatial context information can be aggregated using learnable pyramidal representations. These pyramidal representations are structured as a series of spatial feature maps with a spatial dimension of 16×16 and 64 channels. The maximum number of predicted pyramids is n.

[0073] In traditional attention mechanisms, the computational cost grows quadratically with the spatial dimensions of the key and query matrices. To alleviate this problem, this application proposes a novel architectural variant called the dynamic sparse attention mechanism. This mechanism reduces computational cost by introducing dual attention computation. Specifically, in the first attention layer, it automatically distinguishes foreground regions from background using low-resolution feature maps; while in the second attention layer, it optimizes computation by extracting pyramidal context by processing only selected high-resolution but sparse foreground features.

[0074] Mathematically, the enhanced features can be first combined with the distance mask as follows:

[0075] Mask feature = enhanced feature × binary distance mask + distance mask

[0076] Binary distance mask (x, y) = {

[0077] 0, if distance mask (x, y) = 0

[0078] 1, if distance mask (x, y) > 0

[0079] }

[0080] Here, the mask feature integrates the perpendicular distance of each pixel to the reference cone indicated by the guiding points, thus facilitating the spatial alignment of 2D features extracted from different views.

[0081] Next, the mask features are subjected to a maximum pooling operation with a ratio of α to obtain low-resolution features, which are guided to the first attention layer of the dynamic sparse attention mechanism. Specifically, in the first attention layer, the attention matrix is ​​calculated as follows:

[0082] Attention Matrix 1 = Attention Weight Calculation (Cone-representation Query, Low-resolution Feature Key)

[0083] After the attention calculation, the elements of the attention matrix can be sorted by their attention scores in descending order. This sorting operation allows the indices of the top k elements with the highest scores to be identified, resulting in a set of high-attention indices. These indices can then be mapped to the corresponding positions on the high-resolution mask features using a mapping relationship to obtain a new set of indices, which will be used to extract a subset of foreground features. These foreground features are then directed to the second attention layer of the dynamic sparse attention mechanism for fine attention calculation, where the attention matrix is ​​calculated as follows:

[0084] Attention Matrix 2 = Attention Weight Calculation (Cone-representation Query, Foreground Feature Key)

[0085] Finally, the output of the dynamic sparse attention mechanism is a set of context-rich cone feature representations, which are converted into predicted 2D cone heatmaps through a linear layer.

[0086] To summarize, in a standard attention mechanism with L attention layers, the complexity of attention computation is O(LNMd), where N represents the spatial dimension of the mask feature, d is the hidden dimension, and M represents the product of the spatial dimension and the total number of pyramidal representations. In contrast, the proposed dynamic sparse attention mechanism significantly reduces the computational complexity to L / 2[O1(NMd / α 2 )+O2(kαMd)]. Here, the first term corresponds to the attention computation on the low-resolution features to identify the foreground pyramidal region. The second term represents the fine-grained attention computation on the high-resolution but sparse foreground features, aiming to improve the localization accuracy while keeping the computational cost low.

[0087] This is followed by 3D multi-view feature fusion. The predicted 2D vertebral heatmaps for the lateral and anteroposterior views are rescaled to match the input size. These heatmaps are combined with their respective feature maps and forwarded to the 3D feature fusion unit. In this unit, features can be projected into a fixed-size volume using triangulation, and then the volumes of the multiple views are aggregated and processed through 3D convolution to output heatmaps representing the 3D vertebral locations. These 3D vertebral heatmaps are then processed through a soft weighted sum function, which is used to convert the heatmaps into precise vertebral coordinates. Finally, the 3D feature fusion unit generates a set of predicted 3D coordinates for the vertebrae.

[0088] In order to predict the 2D vertebral position, assuming that the size of the true 2D vertebral heat map on each radiograph is n×16×16, the mean square error loss and similarity loss can be calculated as follows:

[0089] Mean square error loss = (1 / (n×16×16))×the sum of the squares of the differences between the predicted values ​​and the true values ​​of all pixels

[0090] as well as

[0091] Similarity loss = 1-(2×dot product of predicted value and true value) / (sum of squares of predicted value + sum of squares of true value + smoothing term)

[0092] The predicted value and true value represent the true and predicted probabilities of the pixel at position (j, k) of the i-th vertebra.

[0093] Therefore, the two-dimensional context loss can be obtained as follows:

[0094] 2D context loss = mean square error loss + similarity loss

[0095] To predict the 3D coordinates of the vertebra, assume that the true vertebra position is a set of 3D coordinates, and m is the number of vertebrae present in the current input. The mean squared error loss can be calculated for each vertebra as follows:

[0096] 3D positioning loss = (1 / m) × the sum of the squares of the differences between the predicted coordinates of all vertebrae and the true coordinates. The overall detection loss is:

[0097] Overall loss = 2D context loss + 3D positioning loss

[0098] This example uses a synthetic biplane digitally reconstructed spine radiograph dataset. This dataset contains 337 pairs of biplane spinal radiographs, each corresponding to a tomographic volume. The number of vertebrae in each radiograph ranges from 3 to 24. 95% of the dataset is used as the training and test sets (in an 8:2 ratio), and 5% is used for validation.

[0099] The evaluation indicators used in this example are as follows:

[0100] 1. Percentage of correct labeling: It is defined as the percentage of predicted vertebral positions within a given Euclidean distance threshold, compared with the corresponding true positions. The calculation formula is:

[0101] Percentage of correct labels = (1 / N) × number of labels that meet the threshold condition

[0102] The threshold condition is: if the Euclidean distance between the predicted position and the true position is less than the preset threshold τ, then it is 1, otherwise it is 0. N represents the total number of annotated vertebral positions.

[0103] 2. Average position error: measures the average Euclidean distance in millimeters in 3D measurements. The calculation formula is:

[0104] Average position error = (1 / N) × the sum of the Euclidean distances between the predicted and true positions of all vertebrae.

[0105] 3. Area under the curve: A measure of the area enclosed by the correctly labeled percentage curve between the upper and lower limits on the horizontal axis.

[0106] For the 3D vertebral positioning assessment, the threshold τ was set to 10 mm and 20 mm, respectively. Similarly, for the estimation of the 3D area under the curve, the upper limit was set to 10 mm and the lower limit was set to 50 mm.

[0107] The technical solution of this embodiment was developed using a deep learning framework in a high-level programming language and trained on a workstation equipped with a high-performance graphics processor. The input image size was 512×512. The network was trained in an integrated manner from scratch using an adaptive optimizer, a weight decay of 0.05, a batch size of 2, and approximately 100 epochs.

[0108] For parameter tuning, we set the maximum number of prediction cones to n = 10, and the dynamic sparse attention mechanism parameters to α = 32 and k = 8 for synthetic datasets. In contrast, for real data testing, we use n = 5, α = 8, and k = 4. These parameter choices are guided by an approximate estimate of the foreground to background ratio present in each dataset.

[0109] During training, random cropping can be used as a data augmentation technique, and the centroid of the top vertebra of each input lateral and anteroposterior pair is selected as a guide point. During inference, the user can input two point-matching markers, corresponding to the vertebra at the same level in the lateral and anteroposterior views of the radiograph, respectively. The network simultaneously generates n heatmaps and n 3D vertebral coordinates for each radiograph. For each predicted heatmap, the vertebra is considered present if the maximum probability is above a threshold of 0.5. Therefore, the maximum probability value can be used as a criterion for evaluating the validity of the 3D coordinate predictions.

[0110] In other embodiments, the spatial dimension of the vertebral feature representation can also be set to 4×4, 8×8, or 16×16, corresponding to the resolution of the predicted 2D vertebral heat map. Higher resolution provides more contextual information, but also increases computational cost.

[0111] In other embodiments, the pooling ratio α may also be set to 1, 2, or 4. A larger α corresponds to a larger area of ​​selected regions for fine attention calculation, allowing more local context to be aggregated, thereby improving localization accuracy.

[0112] In other embodiments, the top k value may also be set to 2, 4, or 8. A larger k means that more features participate in the fine attention calculation.

[0113] In other embodiments, the two-dimensional visual feature extraction unit may also use different backbone network architectures, such as a residual network, a densely connected network, or an efficient network, to extract multi-scale visual features.

[0114] In other embodiments, in addition to single-point guidance, the method can be extended to support multi-point guidance, allowing users to mark multiple reference vertebrae on each view to further enhance feature extraction and context understanding.

[0115] Compared with the existing technology, the technical solution of this application has the following advantages:

[0116] 1. High Detection Accuracy: This method demonstrated excellent vertebral localization accuracy on a large synthetic dataset, achieving significantly higher percentages of correct labeling than existing methods under both low and high threshold evaluation conditions. The system's mean position error was low, and the area under the curve (AUC) metric approached ideal values, demonstrating highly accurate and stable localization results. Testing on actual clinical datasets also validated the method's superior performance, confirming its practical value and potential for widespread adoption in real-world applications.

[0117] 2. Effectively handle occlusion and deformation: By integrating point-guided reference vertebrae and vertebral semantic context into the detection framework, this application can effectively handle challenging situations such as vertebral occlusion, spinal deformity, and metal implants.

[0118] 3. Computational efficiency: Through dual attention calculation in the dynamic sparse attention mechanism, this application significantly reduces the complexity of attention calculation while maintaining high detection accuracy.

[0119] 4. Integrated training: Unlike existing methods that require pre-training of a 2D feature extraction backbone network, this application supports integrated training, which simplifies the training process and improves performance.

[0120] 5. Cross-view semantic consistency: Through unidirectional distance masking and point-guided feature enhancement, this application can effectively solve the problem of perspective differences between multi-view radiographic images and establish more accurate cross-view semantic correspondence.

[0121] The technical solution of this application can be used for real-time spinal navigation, spinal morphology assessment, education and training, postoperative evaluation (by comparing the vertebral positions before and after surgery to evaluate the surgical effect and degree of spinal correction), etc.

[0122] Accordingly, the embodiments of the present application also provide a computer-readable storage medium having computer-executable instructions stored therein, which implement the various method embodiments of the present application when executed by a processor. Computer-readable storage media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include transient computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0123] In addition, the embodiment of the present application also provides a dual-plane imaging spine key point three-dimensional positioning system, which includes a memory for storing computer executable instructions, and a processor; the processor is used to implement the steps in the above-mentioned method implementation when executing the computer executable instructions in the memory. Among them, the processor can be a central processing unit (Central Processing Unit, referred to as "CPU"), an image processor (Graphic Processing Unit, referred to as "GPU"), a digital signal processor (Digital Signal Processor, referred to as "DSP"), a microcontroller unit (Microcontroller Unit, referred to as "MCU"), a neural network processor (referred to as "NPU"), an application specific integrated circuit (Application Specific Integrated Circuit, referred to as "ASIC"), a field programmable gate array (Field Programmable Gate Array, referred to as "FPGA") or other programmable logic devices, etc. The aforementioned memory can be a read-only memory (read-only memory, referred to as "ROM"), a random access memory (random access memory, referred to as "RAM"), a flash memory (Flash), a hard disk or a solid-state drive, etc. The steps of the method disclosed in each embodiment of the present invention can be directly embodied as being executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0124] In addition, an embodiment of the present application further provides a computer program product, which includes computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] It should be noted that, in this application, relational terms such as first and second are merely used to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. Without further restriction, an element defined by the phrase "comprising a" does not exclude the presence of other identical elements in the process, method, article, or device comprising the element. In this application, if it is mentioned that an action is performed according to a certain element, it means that the action is performed at least according to that element, including two situations: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "multiple," and "multiple" include 2, 2 times, 2 kinds, and more than 2, more than 2 times, and more than 2 kinds.

[0126] The serial numbers used in describing the steps of a method do not themselves limit the order of these steps. For example, a step with a larger serial number does not necessarily have to be executed after a step with a smaller serial number. The step with a larger serial number can be executed first and then the step with a smaller serial number, or they can be executed in parallel, as long as this execution order is reasonable to those skilled in the art. For another example, having multiple steps with consecutive serial numbers (e.g., step 101, step 102, step 103, etc.) does not limit other steps that can be executed in between. For example, there can be other steps between step 101 and step 102.

[0127] This specification includes combinations of the various embodiments described herein. Individual references to embodiments (e.g., "one embodiment" or "some embodiments" or "preferred embodiments") are not mutually exclusive unless indicated as such or clear to one skilled in the art. It should be noted that the word "or" is used in this specification in a non-exclusive sense unless the context clearly indicates or requires otherwise.

[0128] In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Additionally, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A method for three-dimensional positioning of key points of the spine in dual-plane images, characterized in that: include: Obtaining a biplane spinal radiographic image of the subject to be treated, including at least an anteroposterior and lateral views; receiving a user-specified point guide marker on the biplane spinal radiograph corresponding to a reference vertebra at the same level in the anteroposterior and lateral views, respectively; extracting initial two-dimensional visual features of each view based on the biplane spinal radiographs; generating a guidance mask for the reference vertebra based on the position of the point guidance marker; enhancing the initial two-dimensional visual features corresponding to other vertebrae in the biplane image by first attention calculation using the guidance mask and the initial two-dimensional visual features of the reference vertebra to obtain enhanced features; Utilizing the learnable vertebral representation and the enhanced features, a dynamic sparse attention mechanism is used to extract vertebral semantic context to obtain two-dimensional vertebral heat maps in the anteroposterior and lateral views; fusing the two-dimensional vertebral heat maps of the anteroposterior and lateral views with the corresponding enhanced features or the initial two-dimensional visual features, and constructing a pseudo three-dimensional feature volume by triangulation or feature projection method; The pseudo three-dimensional feature volume is processed to obtain the three-dimensional coordinates of other vertebrae except the reference vertebrae in the biplane image.

2. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 1, wherein: The dynamic sparse attention mechanism includes: downsampling the enhanced features to obtain low-resolution features, identifying high-attention area indexes through a second attention calculation based on the low-resolution features; mapping the high-attention area indexes back to the un-downsampled enhanced features to obtain a sparse high-resolution foreground feature subset; generating a context-rich cone feature representation through a third attention calculation based on the sparse high-resolution foreground feature subset, and converting it into a two-dimensional cone heat map for each view.

3. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 2, wherein: In the dynamic sparse attention mechanism, the learnable pyramidal representation serves as a query, the low-resolution features serve as keys for a second attention computation, and the sparse high-resolution foreground feature subset serves as keys for a third attention computation.

4. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 1, wherein: In the first attention calculation, the initial two-dimensional visual features corresponding to the other vertebrae are used as queries, and the guiding features extracted from the initial two-dimensional visual features of the reference vertebrae using the guiding mask are used as keys and values.

5. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 1, wherein: The guide mask at least includes a binary guide mask and a one-way distance mask; The unidirectional distance mask contains the vertical distance information of each pixel to the point guide marker and is combined with the enhanced features before the dynamic sparse attention mechanism extracts the pyramidal semantic context.

6. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 1, wherein: The step of extracting the initial two-dimensional visual features of each view uses a two-dimensional U-shaped network structure to extract multi-scale visual features.

7. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 1, wherein: The step of processing the pseudo three-dimensional feature volume includes: using three-dimensional convolution to process the aggregated multi-view feature volume to obtain a three-dimensional vertebral heat map; and processing the three-dimensional vertebral heat map through a soft weighted sum function to obtain the three-dimensional coordinates.

8. The method for three-dimensional positioning of key points of the spine using dual-plane imaging according to claim 1, wherein: The method also includes: defining an overall loss function including a two-dimensional context loss and a three-dimensional positioning loss, and using an integrated training method to simultaneously optimize the generation of the two-dimensional vertebral heat map and the positioning of the three-dimensional coordinates, without the need for a pre-trained two-dimensional feature extraction backbone network.

9. A dual-plane imaging spine key point three-dimensional positioning system, characterized by: include: a memory for storing computer-executable instructions; as well as, A processor, coupled to the memory, configured to implement the steps of the method according to any one of claims 1 to 8 when executing the computer-executable instructions.

10. A computer program product comprising computer executable instructions, characterized in that: When the computer executable instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images