Tibial intramedullary nail locking navigation system and method based on mixed reality and two-dimensional code

Abstract

The application relates to the field of mixed reality, in particular to a tibial intramedullary nail locking navigation system and method based on mixed reality and a two-dimensional code, which comprises the following steps: determining the connector pose of a proximal connector of the tibial intramedullary nail according to a first two-dimensional code, and the first two-dimensional code corresponds to a CT image of a patient; generating a locking operation blood vessel segmentation graph through a CT image segmentation model of a U-Net architecture; a mixed reality device scans a second two-dimensional code arranged on a distal locking drill head guide sleeve, and determines a locking pose according to the second two-dimensional code; the mixed reality device generates a three-dimensional tibial blood vessel virtual model based on the locking operation blood vessel segmentation graph, and generates a navigation prompt for avoiding blood vessels in the three-dimensional tibial blood vessel virtual model to guide the locking drill to penetrate into a locking hole of the connector. The application realizes accurate segmentation of a three-dimensional model of a tibial blood vessel, and provides reliable prompts for tibial intramedullary nail locking navigation for blood vessel avoidance.

Description

Technical Field

[0001] This invention relates to the field of mixed reality, and more particularly to a tibial intramedullary nail locking navigation system and method based on mixed reality and QR codes. Background Technology

[0002] Tibial fractures are a common type of long bone fracture in clinical practice, accounting for 10% to 15% of all fractures, and are particularly common in high-energy injuries such as traffic accidents and falls from heights. For significantly displaced, comminuted, or unstable tibial fractures, surgical treatment is the preferred option for restoring skeletal continuity and limb function. Among these procedures, intramedullary nailing of the tibia has become the mainstream surgical method in clinical practice due to its advantages such as good biomechanical stability, wide fixation range, and minimal disruption to the blood supply to the fracture ends.

[0003] One of the core steps in tibial intramedullary nailing is the distal locking screw procedure, which involves precisely inserting a locking screw into the locking hole at the distal end of the tibial intramedullary nail to achieve rigid fixation between the intramedullary nail and the tibia, preventing postoperative loosening of the intramedullary nail. However, due to the complex anatomy of the distal tibia, surrounded by important blood vessels such as the anterior and posterior tibial arteries, and the deep location of the distal locking hole with difficult visualization, the accuracy and safety of the distal locking screw procedure face significant challenges.

[0004] Currently, commonly used distal pin-guided techniques in clinical practice mainly include traditional freehand pinning, C-arm X-ray-assisted pinning, and optical navigation-assisted pinning. Traditional freehand pinning relies on the surgeon's clinical experience and touch for blind pin placement, which is not only difficult to perform and has a long learning curve, but also has low pinning accuracy, often resulting in complications such as pin deviation from the pin hole, penetration of blood vessels or nerves. Statistics show that the complication rate is as high as 15% to 30%, seriously affecting postoperative recovery. Furthermore, to confirm pin placement, the surgeon needs to repeatedly perform fluoroscopic examinations, exposing both patients and medical staff to large amounts of ionizing radiation, which poses long-term health risks.

[0005] While C-arm X-ray machine-assisted locking pins can provide skeletal image references through real-time fluoroscopy, which can improve the accuracy of locking pins to some extent, the fluoroscopic images are two-dimensional plane images and cannot intuitively present the three-dimensional spatial relationship between the bone connectors, blood vessels and locking pins. Surgeons need to mentally fill in the three-dimensional structure through two-dimensional images, which can easily lead to spatial positioning errors.

[0006] To address these issues, optical navigation-assisted pin-locking technology has emerged. By installing optical markers on the bone connector and surgical instruments, and combining them with preoperative CT images to construct a three-dimensional bone model, real-time tracking and guidance of pin positions can be achieved. Existing optical navigation systems only utilize a single plain CT image for modeling, failing to fully integrate enhanced vascular imaging CT images, thus failing to accurately present vascular distribution information and making it difficult to achieve precise navigation for vascular avoidance.

[0007] In recent years, Mixed Reality (MR) technology has shown promising application prospects in the field of medical surgical navigation due to its ability to integrate virtual 3D models with real surgical scenes in real time. MR navigation can directly overlay the 3D model reconstructed from preoperative medical images onto the patient's surgical area, providing surgeons with an intuitive 3D spatial reference without the need to switch between the surgical field of view and the navigation interface. However, the current application of MR in intramedullary nailing navigation for tibial screw locking still has significant shortcomings. The feature extraction and fusion effects of preoperative CT images are not good, especially the insufficient fusion of multimodal data from plain CT and enhanced angiography CT, resulting in insufficient segmentation accuracy of blood vessels. This leads to the constructed 3D model failing to accurately reflect the distribution of blood vessels, failing to provide reliable vascular avoidance navigation prompts for screw locking operations, and failing to meet the precise navigation needs of clinical surgery.

[0008] Therefore, how to enable mixed reality devices to accurately segment the 3D model of tibial vessels in order to achieve reliable vessel avoidance and intramedullary nail locking navigation for tibial tuberosity is a technical problem that needs to be solved. Summary of the Invention

[0009] To this end, the present invention provides a tibial intramedullary nail locking navigation system and method based on mixed reality and QR codes. The mixed reality device achieves accurate segmentation of tibial vessels and three-dimensional modeling through dual-modal vessel segmentation using QR code positioning, plain CT images and enhanced vascular imaging CT images, providing reliable prompts for tibial intramedullary nail locking navigation for vessel avoidance.

[0010] To achieve the above objectives, this invention proposes a tibial intramedullary nail locking navigation method based on mixed reality and QR codes, comprising:

[0011] The mixed reality device scans the first QR code set on the proximal connector of the tibial intramedullary nail, and determines the connector pose of the proximal connector of the tibial intramedullary nail based on the first QR code, as well as the patient's preoperative plain CT image and enhanced vascular imaging CT image corresponding to the first QR code;

[0012] The plain CT images and enhanced angiography CT images are passed through a dual encoder based on a gated mechanism architecture to generate high-level features for plain CT and high-level features for enhanced angiography CT.

[0013] The high-level features of plain CT and enhanced vascular CT are passed through a bottleneck layer based on a cross-attention mechanism architecture to generate fused CT features;

[0014] The fused CT features, the plain CT high-level features, and the enhanced vascular CT high-level features are passed through a decoding output layer based on cross-modal skip connections to generate a lock-nail surgery vascular segmentation map, wherein the dual encoder, bottleneck layer, and decoding output layer constitute a CT image segmentation model with a U-Net architecture.

[0015] The mixed reality device scans the second QR code set on the remote locking nail drill bit guide sleeve, and determines the locking nail position based on the second QR code;

[0016] The mixed reality device generates a three-dimensional tibial vessel virtual model based on the vascular segmentation map of the locking pin surgery. After registering the connector fixed coordinate system based on the connector pose and locking pin pose, it generates navigation prompts in the three-dimensional tibial vessel virtual model to guide the locking pin to pass through the connector locking pin hole while avoiding the blood vessel.

[0017] Furthermore, the process of generating high-level features for plain CT and enhanced vascular CT using dual encoders includes:

[0018] The plain CT image is processed through a first branch coding block to generate the original plain CT features;

[0019] The enhanced vascular imaging CT image is passed through a second branch coding block to generate original enhanced vascular CT features;

[0020] The original enhanced vascular CT features are passed through a gating network to generate plain scan feature gating vectors;

[0021] Based on the gating vector of the plain scan feature, the original plain scan CT feature is subjected to channel weighting calculation to generate enhanced plain scan CT feature, and the enhanced plain scan CT feature is passed through the first downsampling layer to generate the next level first coding block input feature or the plain scan CT high-level feature;

[0022] The original enhanced vascular CT features are passed through a second downsampling layer to generate the next level second coding block input features or the enhanced vascular CT high-level features;

[0023] The dual encoder includes a first branch coding block, a second branch coding block, a gating network, a first downsampling layer, and a second downsampling layer, wherein the convolution kernel sizes of the first branch coding block and the second branch coding block are equal.

[0024] Furthermore, the process of generating the flat-scan feature gating vector through the gating network includes:

[0025] The original enhanced vascular CT features are concatenated with the first feature generated by the global average pooling layer and the second feature generated by the global maximum pooling layer to generate the original gating vector.

[0026] The original gate vector is used for feature mapping through a fully connected sub-network to generate the flat-scan feature gate vector;

[0027] The gated network includes a global average pooling layer, a global max pooling layer, and a fully connected sub-network.

[0028] Furthermore, the process of generating the flat-scan feature gating vector through the fully connected sub-network includes:

[0029] The original gated vector is passed through a first fully connected layer to perform feature dimensionality reduction, thereby generating a dimensionality-reduced gated vector;

[0030] The dimension-reduced gating vector is increased in dimension by passing it through a second fully connected layer to generate the flat-scan feature gating vector;

[0031] The fully connected sub-network includes a first fully connected layer and a second fully connected layer.

[0032] Furthermore, the process of generating fused CT features through the bottleneck layer includes:

[0033] Based on the high-level features of the plain CT scan, an attention query vector is generated, and based on the high-level features of the enhanced vascular CT scan, an attention key vector and an attention value vector are generated.

[0034] Calculate the cross-attention weights based on the attention query vector, attention key vector, and attention value vector;

[0035] The cross-attention weights and the residual connection features of the high-level features of the plain CT scan are normalized through layers to generate the original cross features;

[0036] The splicing features of the plain CT high-level features, enhanced vascular CT high-level features, and original cross features are passed through multi-level convolutional layers to generate the fused CT features;

[0037] The bottleneck layer includes layer normalization and multiple levels of convolutional layers.

[0038] Furthermore, the process of generating a vascular segmentation map for nail-locking surgery through the decoding output layer includes:

[0039] The high-level features of plain CT and high-level features of enhanced vascular CT are connected through a cross-modal skip connection layer to generate cross-modal fusion features;

[0040] The decoded features of the previous level or the fused CT features are upsampled to generate upsampled features;

[0041] The cross-modal fused features and upsampled features are passed through a fused convolutional layer to generate the current level decoding features or decoder output features;

[0042] The decoder output features are passed through a convolutional output head to generate the vascular segmentation map for the nail-locking surgery;

[0043] The vascular segmentation map for the locking and nailing surgery includes a cross-modal skip connection layer, a fusion convolutional layer, and a convolutional output head.

[0044] Furthermore, the intramedullary nail locking method for tibial tuberculosis also includes:

[0045] A sample set was constructed based on big data on the tibial anatomy of Chinese people;

[0046] The sample set is used to generate the segmentation map of the vascular segmentation in the locking pin surgery through a CT image segmentation model, and the vascular segmentation map of the locking pin surgery and the true labels of the sample set are used to train and optimize the CT image segmentation model through a composite segmentation loss function;

[0047] The composite segmentation loss function includes a contour Dice loss term and a cross-entropy loss term.

[0048] Furthermore, the process of generating a three-dimensional virtual model of the tibial vessels includes:

[0049] The processing unit uses 3D Slicer software to generate a three-dimensional virtual model of the tibial vessels from the segmented vascular map of the locking screw surgery, and then sends the three-dimensional virtual model of the tibial vessels to the mixed reality device.

[0050] Furthermore, the process of generating navigation prompts for inserting the locking pin into the connector locking pin hole while avoiding blood vessels includes:

[0051] The color change of the three-dimensional tibial blood vessel virtual model provides navigation prompts for the pin to be inserted into the connector pin hole, and the visual and audio units of the mixed reality device provide navigation prompts for the pin to approach the blood vessel.

[0052] This invention also proposes a system for tibial intramedullary nail locking navigation based on mixed reality and QR codes, comprising:

[0053] Mixed reality devices, including MR glasses;

[0054] The processing unit communicates with the mixed reality device and carries the CT image segmentation model;

[0055] The proximal connector of the tibial intramedullary nail has a detachable first QR code.

[0056] The remote locking drill bit guide sleeve is detachably equipped with a second QR code.

[0057] Compared with existing technologies, the advantages of this invention lie in its connector fixed coordinate system registration mechanism, which effectively avoids the positioning deviation problems caused by marker point occlusion and cumbersome registration processes in traditional navigation technologies. The QR code positioning method eliminates the need for complex optical marker installation and calibration processes, simplifying operation and enabling real-time precise alignment between the virtual model and the actual surgical scene. This provides a stable and reliable spatial reference for nail insertion, significantly improving the accuracy of nail insertion into the connector's nail holes, effectively reducing complications such as nail deviation and bone penetration, and ensuring the effectiveness of internal fixation. Through dual-modal precise segmentation of blood vessels using plain CT images and enhanced vascular imaging CT images, a three-dimensional model of precisely segmented tibial vessels is achieved, providing reliable guidance for tibial intramedullary nailing navigation with vessel avoidance.

[0058] In particular, this invention constructs a high-precision vascular visualization model to achieve accurate vascular avoidance in the nail-locking path. Through an architecture-optimized U-Net segmentation model, multi-module collaboration enables deep feature fusion and accurate segmentation of multimodal CT images, providing high-quality data support for vascular avoidance navigation. Dual encoders leverage gating networks to deeply mine the guiding value of enhanced vascular CT features. The bottleneck layer establishes a precise correlation between plain CT and enhanced vascular CT features through a cross-attention mechanism. The decoding output layer fully reuses features extracted from each level by the encoder through cross-modal skip connection layers, combined with upsampling and fusion convolution to achieve accurate feature recovery, ultimately generating a detailed and clearly defined vascular segmentation map for nail-locking surgery.

[0059] In particular, the navigation system of this invention features a simple and efficient architecture. Its core components include a processing unit equipped with a CT image segmentation model, MR glasses integrating vision and sound units, and a detachable guide sleeve for the proximal connector and distal locking screw drill bit of the tibial intramedullary nail, which can be detachably equipped with first and second QR codes. By using a mixed reality device, a three-dimensional virtual model of the tibial vessels and locking screw navigation prompts are directly superimposed on the patient's surgical area. The surgeon can obtain accurate navigation information simultaneously without frequently switching their line of sight, significantly improving the continuity of operation. At the same time, the position and orientation of the connector and locking screw can be quickly completed by scanning the QR code with the MR glasses, greatly simplifying the preoperative preparation and intraoperative positioning steps, reducing the complexity of multi-person collaboration, effectively shortening the operation time, and reducing the dependence on the surgeon's experience. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating the intramedullary nail locking navigation method for tibial tuberosity based on mixed reality and QR codes, according to an embodiment of the present invention.

[0061] Figure 2 This is a schematic diagram of the dual encoder process of the tibial intramedullary nail locking navigation method based on mixed reality and QR codes in an embodiment of the present invention.

[0062] Figure 3 This is a schematic diagram of the tibial intramedullary nail locking navigation system based on mixed reality and QR codes, according to an embodiment of the present invention.

[0063] Figure 4 This is a schematic diagram of coordinate registration for a tibial intramedullary nail locking navigation system based on mixed reality and QR codes, according to an embodiment of the present invention.

[0064] Figure 5 This is a schematic diagram illustrating the mixed reality surgical display effect of the tibial intramedullary nail locking system based on mixed reality and QR codes, according to an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0066] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0067] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0068] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0069] like Figures 1 to 5 As shown, this invention provides a tibial intramedullary nail locking navigation system and method based on mixed reality and QR codes. The mixed reality device achieves accurate segmentation of tibial vessels in three-dimensional modeling through dual-modal vessel segmentation using QR code positioning, plain CT images, and enhanced vascular imaging CT images, providing reliable prompts for tibial intramedullary nail locking navigation for vessel avoidance.

[0070] like Figure 1As shown, this embodiment proposes a tibial intramedullary nail locking navigation method based on mixed reality and QR codes, including:

[0071] The mixed reality device scans the first QR code set on the proximal connector of the tibial intramedullary nail, and determines the connector pose of the proximal connector of the tibial intramedullary nail based on the first QR code, as well as the patient's preoperative plain CT image and enhanced vascular imaging CT image corresponding to the first QR code;

[0072] The plain CT images and enhanced angiography CT images are passed through a dual encoder based on a gated mechanism architecture to generate high-level features for plain CT and high-level features for enhanced angiography CT.

[0073] The high-level features of plain CT and enhanced vascular CT are passed through a bottleneck layer based on a cross-attention mechanism architecture to generate fused CT features.

[0074] The fused CT features, the plain CT high-level features, and the enhanced vascular CT high-level features are passed through a decoding output layer based on cross-modal skip connections to generate a lock-nail surgery vascular segmentation map, wherein the dual encoder, bottleneck layer, and decoding output layer constitute a CT image segmentation model with a U-Net architecture.

[0075] The mixed reality device scans the second QR code set on the remote locking nail drill bit guide sleeve, and determines the locking nail position based on the second QR code;

[0076] The mixed reality device generates a three-dimensional tibial vessel virtual model based on the vascular segmentation map of the locking pin surgery. After registering the connector fixed coordinate system based on the connector pose and locking pin pose, it generates navigation prompts in the three-dimensional tibial vessel virtual model to guide the locking pin to pass through the connector locking pin hole while avoiding the blood vessel.

[0077] Specifically, such as Figure 3 As shown, the mixed reality device is preferably MR glasses, which are preferably holographic glasses, Microsoft HoloLens. The Microsoft HoloLens communicates with the host computer of the processing unit. The host computer carries the algorithm model of this embodiment to generate an accurate three-dimensional tibial blood vessel virtual model. After the Microsoft HoloLens obtains the three-dimensional tibial blood vessel virtual model, it displays it and after actually aligning the connector pose and locking pin pose, it displays or shows it in the field of view of the three-dimensional tibial blood vessel virtual model of the Microsoft HoloLens. This is to determine whether the position and angle of the distal locking pin drill guide sleeve driving the locking pin into the larger connector locking pin hole avoids important large blood vessels such as the anterior tibial artery and posterior tibial artery, thereby realizing screw navigation with blood vessel avoidance.

[0078] More specifically, the host computer of the processing unit performs three-dimensional visualization modeling of the vascular segmentation map of the locking screw surgery using the 3D Slicer open-source software. The 3D Slicer open-source software is connected to the CT image segmentation model of the U-Net architecture through an interface.

[0079] Specifically, such as Figure 4 As shown, the connector pose determined by the first QR code set by the proximal connector of the tibial intramedullary nail is a coordinate system including the center of the target locking hole as the origin. coordinate system This includes the position and rotation angle information of the proximal connector of the tibial intramedullary nail. The locking pin pose, determined by the second QR code located on the distal locking pin drill guide sleeve, is a coordinate system with the drill tip as the origin. The PnP (Perspective-n-Point) algorithm formula Calculate the relative transformation between two coordinate systems. This enables precise positioning of the tool tip within the intramedullary nail coordinate system.

[0080] Specifically, the non-contrast CT image (NCCT) is volumetric data obtained by scanning the human body with an X-ray beam without injecting any contrast agent. The enhanced angiography CT image is generated by performing a CT scan after intravenous injection of iodine contrast agent. Utilizing the high attenuation properties of iodine contrast agent on X-rays, the vascular system exhibits a significantly high signal, and the reconstructed vascular-specific volumetric data is generated. This embodiment uses a U-Net architecture CT image segmentation model for end-to-end joint vascular segmentation. This allows for focused learning of clear bone edges in non-contrast CT and significant vascular contrast in enhanced angiography CT, avoiding feature confusion that occurs when a single encoder processes vascular and skeletal features. Furthermore, the decoding output layer can simultaneously utilize the anatomical framework information of the skeleton and the angiographic enhancement information of the blood vessels, thereby more accurately determining their spatial relationship, especially in boundary regions.

[0081] Preferably, when the plain CT image and the enhanced angiography CT image are not precisely aligned in space, the images are registered using the VoxelMorph registration algorithm and then input into the CT image segmentation model of the U-Net architecture for image vessel segmentation.

[0082] like Figure 2 As shown, the process of generating high-level features for plain CT and enhanced vascular CT using dual encoders further includes:

[0083] The plain CT image is processed through a first branch coding block to generate the original plain CT features;

[0084] The enhanced vascular imaging CT image is passed through a second branch coding block to generate original enhanced vascular CT features;

[0085] The original enhanced vascular CT features are passed through a gating network to generate plain scan feature gating vectors;

[0086] Based on the gating vector of the plain scan feature, the original plain scan CT feature is subjected to channel weighting calculation to generate enhanced plain scan CT feature, and the enhanced plain scan CT feature is passed through the first downsampling layer to generate the next level first coding block input feature or the plain scan CT high-level feature;

[0087] The original enhanced vascular CT features are passed through a second downsampling layer to generate the next level second coding block input features or the enhanced vascular CT high-level features;

[0088] The dual encoder includes a first branch coding block, a second branch coding block, a gating network, a first downsampling layer, and a second downsampling layer, wherein the convolution kernel sizes of the first branch coding block and the second branch coding block are equal.

[0089] like Figure 2 As shown, the process of generating the flat-scan feature gating vector through the gating network further includes:

[0090] The original enhanced vascular CT features are concatenated with the first feature generated by the global average pooling layer and the second feature generated by the global maximum pooling layer to generate the original gating vector.

[0091] The original gate vector is used for feature mapping through a fully connected sub-network to generate the flat-scan feature gate vector;

[0092] The gated network includes a global average pooling layer, a global max pooling layer, and a fully connected sub-network.

[0093] Furthermore, the process of generating the flat-scan feature gating vector through the fully connected sub-network includes:

[0094] The original gated vector is passed through a first fully connected layer to perform feature dimensionality reduction, thereby generating a dimensionality-reduced gated vector;

[0095] The dimension-reduced gating vector is increased in dimension by passing it through a second fully connected layer to generate the flat-scan feature gating vector;

[0096] The fully connected sub-network includes a first fully connected layer and a second fully connected layer.

[0097] Specifically, the process of generating high-level features of plain CT and enhanced vascular CT using dual encoders can be represented as follows:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] In the formula, , These represent the original plain CT features and the original enhanced vascular CT features, respectively. express Activation function Indicates group normalization, This indicates that a convolution operation is performed on the output features of the previous level (i.e., the input features of the first coding block of the current level) or on a plain CT image with l=1. This indicates that a convolution operation is performed on the output features of the previous level (i.e., the input features of the second coding block of the current level) or on the enhanced angiography CT image when l is 1. The preferred kernel size for this convolution operation is 3x3x3. This represents the original gating vector of the c-th channel. This represents the original gate vector containing all channels. This indicates that the original enhanced vascular CT features are stitched together using the first feature from the global average pooling layer and the second feature from the global max pooling layer. Represents a dimension-reduced gated vector. , This represents the convolutional weight matrix and bias vector of the first fully connected layer, which has a compression factor preferably of 16 for dimensionality reduction, resulting in fewer parameters in the fully connected network. , This represents the convolutional weight matrix and bias vector of the second fully connected layer, restoring the dimension to be equal to the number of channels in the original plain CT features. Represents the feature gating vector for flat scanning. This represents channel-level multiplication. Indicates features of enhanced plain CT scan. This represents the input features of the first coding block of the next level, or the high-level features of plain CT scans when l is 3. This represents the input features of the second coding block at the next level, or the high-level features of enhanced vascular CT when l is 3. and The preferred method for convolution operations is to use a 1x1x1 convolution kernel. Convolution or max pooling with a stride of 2 is preferred.

[0107] In particular, by enhancing the strong signal of blood vessels in CT through a gating network, the channels of the skeletal region that may be adjacent to blood vessels in the plain CT features are enhanced. Furthermore, the generation of the gating vector depends only on the vector after global pooling, and the computational cost is much less than that of performing attention calculations across the entire feature map.

[0108] Furthermore, the process of generating fused CT features through the bottleneck layer includes:

[0109] Based on the high-level features of the plain CT scan, an attention query vector is generated, and based on the high-level features of the enhanced vascular CT scan, an attention key vector and an attention value vector are generated.

[0110] Calculate the cross-attention weights based on the attention query vector, attention key vector, and attention value vector;

[0111] The cross-attention weights and the residual connection features of the high-level features of the plain CT scan are normalized through layers to generate the original cross features;

[0112] The spliced ​​features of the plain CT high-level features, enhanced vascular CT high-level features, and original cross features are passed through multi-level convolutional layers to generate the fused CT features;

[0113] The bottleneck layer includes layer normalization and multiple levels of convolutional layers.

[0114] Specifically, the process of generating fused CT features through the bottleneck layer can be represented as:

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123] In the formula, , , These represent the attention query vector, attention key vector, and attention value vector, respectively. , These represent high-level features of plain CT and high-level features of enhanced vascular CT, respectively. , , Represents the mapping weight matrix. Indicates the cross-attention weights. Represents the dimension of attention space. Represents the original cross features. Representation layer normalization, This represents the residual connectivity features between cross-attention weights and high-level features of plain CT scans. This represents the stitching feature of high-level features from plain CT, high-level features from enhanced vascular CT, and the original crossover features. The convolution operation preferably uses a 1x1 convolution kernel. , , These represent the original fusion feature, the first compressed feature, and the fused CT feature, respectively. Indicates batch normalization. The number of output channels for the convolution operation is C / 2 (C is the original number of channels in the plain CT image). The number of output channels for the convolution operation is C / 4.

[0124] Furthermore, the process of generating a vascular segmentation map for nail-locking surgery through the decoding output layer includes:

[0125] The high-level features of plain CT and high-level features of enhanced vascular CT are connected through a cross-modal skip connection layer to generate cross-modal fusion features;

[0126] The decoded features of the previous level or the fused CT features are upsampled to generate upsampled features;

[0127] The cross-modal fused features and upsampled features are passed through a fused convolutional layer to generate the current level decoding features or decoder output features;

[0128] The decoder output features are passed through a convolutional output head to generate the vascular segmentation map for the nail-locking surgery;

[0129] The vascular segmentation map for the locking and nailing surgery includes a cross-modal skip connection layer, a fusion convolutional layer, and a convolutional output head.

[0130] Specifically, the process of generating a vascular segmentation map for nail-locking surgery through the decoding output layer can be represented as follows:

[0131]

[0132]

[0133]

[0134]

[0135] In the formula, This represents the cross-modal fusion feature of the l-th layer corresponding to the encoder. The preferred method for convolution operations is to use a 3x3 convolution kernel. Indicates batch normalization, The convolution operation preferably uses a 1x1 convolution kernel. This represents the upsampled features of the l-th layer. This represents the decoding feature of the previous level, or the fused CT feature when l equals 0. This indicates a convolutional layer consisting of 3×3×3 convolutions, group normalization, and ReLU activation. This represents the decoding features of the current level or the decoder output features when l is 2. This is a diagram showing the segmentation of blood vessels during a locking pin surgery. The convolution operation preferably uses a 1x1 convolution kernel.

[0136] Furthermore, the tibial intramedullary nail locking navigation method based on mixed reality and QR codes also includes:

[0137] A sample set was constructed based on the Chinese Anatomical Database;

[0138] The sample set is used to generate the segmentation map of the vascular segmentation in the locking pin surgery through a CT image segmentation model, and the vascular segmentation map of the locking pin surgery and the true labels of the sample set are used to train and optimize the CT image segmentation model through a composite segmentation loss function;

[0139] The composite segmentation loss function includes a contour Dice loss term and a cross-entropy loss term.

[0140] Specifically, the composite segmentation loss function can be expressed as:

[0141]

[0142] In the formula, This represents the composite segmentation loss function. This represents the contour Dice loss term (ClDice). Represents the cross-entropy loss term. , These are hyperparameters, with the preferred starting values ​​being 0.6 and 0.4, which are the default settings for the U-Net framework and have been proven by many studies to be robust starting points.

[0143] Specifically, in the comparative experiment, the segmentation accuracy (Dice) of the CT image segmentation model of this embodiment and the conventional U-Net model using only plain CT images were 0.8146 and 0.7546 respectively. It can be seen that the CT image segmentation model of this embodiment achieves multi-stage fine-grained blood vessel segmentation.

[0144] Furthermore, the process of generating a three-dimensional virtual model of the tibial vessels includes:

[0145] The processing unit uses 3D Slicer software to generate a three-dimensional virtual model of the tibial vessels from the segmented vascular map of the locking screw surgery, and then sends the three-dimensional virtual model of the tibial vessels to the mixed reality device.

[0146] Furthermore, the process of generating navigation prompts for inserting the locking pin into the connector locking pin hole while avoiding blood vessels includes:

[0147] The color change of the three-dimensional tibial blood vessel virtual model provides navigation prompts for the pin to be inserted into the connector pin hole, and the visual and audio units of the mixed reality device provide navigation prompts for the pin to approach the blood vessel.

[0148] like Figure 5 As shown, specifically, since conventional CT scans struggle to clearly visualize fine nerves, such as the deep peroneal nerve, the model can map a standardized vascular and nerve model onto the patient's tibial model using non-rigid registration, based on the probability distribution of large-scale data statistics. This completes the image by revealing soft tissue structures that are not visible in the images. The MR glasses' camera captures a second QR code B attached to the surgical tool (such as a drill bit) in real time. Using the PnP (Perspective-n-Point) algorithm, the system refreshes the tool tip's three-dimensional coordinates and axial vector in space every second (e.g., 60Hz). Under a unified world coordinate system, the processing unit calculates the shortest Euclidean distance D from the tool tip coordinates to the surface of the virtual vascular or nerve model in real time.

[0149] The system's preset safety threshold is preferably 5mm. The system compares the shortest Euclidean distance D calculated in real time with the safety threshold: if D is greater than the safety threshold, the virtual extension line of the surgical tool will be displayed in green in the mixed reality field of view, and the remaining depth of the drill bit from the distal keyhole will be displayed in real time. If D is less than the safety threshold, it is a dangerous situation, and the system determines that there is a risk of damage. Once a dangerous situation is determined, the system immediately performs the following actions: Visual warning: The virtual path in the MR glasses instantly turns red, and a flashing "Caution: Blood Vessels / Nerves" icon pops up in the center of the field of view. Auditory warning: The speaker emits a high-frequency, rapid beeping sound, prompting the doctor to immediately stop needle insertion or change the angle.

[0150] This embodiment also provides a system for the tibial intramedullary nail locking navigation method based on mixed reality and QR codes, including:

[0151] Mixed reality devices, including MR glasses;

[0152] The processing unit communicates with the mixed reality device and carries the CT image segmentation model;

[0153] The proximal connector of the tibial intramedullary nail has a detachable first QR code.

[0154] The remote locking drill bit guide sleeve has a detachable second QR code.

[0155] In this embodiment, the fixed coordinate system registration mechanism of the connector effectively avoids the positioning deviation problems caused by marker point occlusion and cumbersome registration processes in traditional navigation technologies. The QR code positioning method eliminates the need for complex optical marker installation and calibration processes, simplifying operation and enabling real-time and accurate alignment between the virtual model and the actual surgical scene. This provides a stable and reliable spatial reference for nail insertion, significantly improving the accuracy of nail insertion into the connector nail holes, effectively reducing complications such as nail deviation and bone penetration, and ensuring the effectiveness of internal fixation. Through dual-modal precise vascular segmentation of plain CT images and enhanced vascular imaging CT images, a three-dimensional model of precisely segmented tibial vessels is achieved, providing reliable guidance for tibial intramedullary nailing navigation with vessel avoidance. A high-precision vascular visualization model is constructed to achieve precise vascular avoidance of the nail path. An optimized U-Net segmentation model, through multi-module collaboration, achieves deep feature fusion and precise segmentation of multi-modal CT images, providing high-quality data support for vascular avoidance navigation. Dual encoders leverage gating networks to deeply mine the guiding value of enhanced vascular CT features. The bottleneck layer establishes a precise correlation between plain CT and enhanced vascular CT features through a cross-attention mechanism. The decoding output layer fully reuses features extracted from each level by the encoder through a cross-modal skip connection layer, and achieves precise feature recovery through upsampling and fusion convolution, ultimately generating a detailed and clearly defined segmentation map of the vascular segmentation for the locking screw surgery. The navigation system has a simple and efficient architecture. Its core includes a processing unit with a CT image segmentation model, MR glasses with integrated vision and sound units, and a detachable guide sleeve for the proximal connector and distal locking screw drill bit of the tibial intramedullary nail, which can be set with the first and second QR codes. Through mixed reality devices, a three-dimensional tibial vascular virtual model and locking screw navigation prompts are directly superimposed on the patient's surgical area. The surgeon can obtain accurate navigation information simultaneously without frequently switching their line of sight, significantly improving the continuity of operation. At the same time, the pose positioning of the connector and locking screw can be quickly completed by scanning the QR code with the MR glasses, greatly simplifying the preoperative preparation and intraoperative positioning steps, reducing the complexity of multi-person collaboration, effectively shortening the operation time, and reducing the dependence on the surgeon's experience.

[0156] Those skilled in the art will recognize that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0158] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A tibial intramedullary nail locking method based on mixed reality and QR codes, characterized in that, include: The mixed reality device scans the first QR code set on the proximal connector of the tibial intramedullary nail, and determines the connector pose of the proximal connector of the tibial intramedullary nail based on the first QR code, as well as the patient's preoperative plain CT image and enhanced vascular imaging CT image corresponding to the first QR code; The plain CT images and enhanced angiography CT images are passed through a dual encoder based on a gated mechanism architecture to generate high-level features for plain CT and high-level features for enhanced angiography CT. The high-level features of plain CT and enhanced vascular CT are passed through a bottleneck layer based on a cross-attention mechanism architecture to generate fused CT features. The fused CT features, the plain CT high-level features, and the enhanced vascular CT high-level features are passed through a decoding output layer based on cross-modal skip connections to generate a lock-nail surgery vascular segmentation map, wherein the dual encoder, bottleneck layer, and decoding output layer constitute a CT image segmentation model with a U-Net architecture. The mixed reality device scans the second QR code set on the remote locking nail drill bit guide sleeve, and determines the locking nail position based on the second QR code; The mixed reality device generates a three-dimensional tibial vessel virtual model based on the vascular segmentation map of the locking pin surgery. After registering the connector fixed coordinate system based on the connector pose and locking pin pose, it generates navigation prompts in the three-dimensional tibial vessel virtual model to guide the locking pin to pass through the connector locking pin hole while avoiding the blood vessel.

2. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to claim 1, characterized in that, The process of generating high-level features for plain CT and enhanced vascular CT using dual encoders includes: The plain CT image is processed through a first branch coding block to generate the original plain CT features; The enhanced vascular imaging CT image is passed through a second branch coding block to generate original enhanced vascular CT features; The original enhanced vascular CT features are passed through a gating network to generate plain scan feature gating vectors; Based on the gating vector of the plain scan feature, the original plain scan CT feature is subjected to channel weighting calculation to generate enhanced plain scan CT feature, and the enhanced plain scan CT feature is passed through the first downsampling layer to generate the next level first coding block input feature or the plain scan CT high-level feature; The original enhanced vascular CT features are passed through a second downsampling layer to generate the next level second coding block input features or the enhanced vascular CT high-level features; The dual encoder includes a first branch coding block, a second branch coding block, a gating network, a first downsampling layer, and a second downsampling layer, wherein the convolution kernel sizes of the first branch coding block and the second branch coding block are equal.

3. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to claim 2, characterized in that, The process of generating flat-scan feature gating vectors through a gating network includes: The original enhanced vascular CT features are concatenated with the first feature generated by the global average pooling layer and the second feature generated by the global maximum pooling layer to generate the original gating vector. The original gate vector is used for feature mapping through a fully connected sub-network to generate the flat-scan feature gate vector; The gated network includes a global average pooling layer, a global max pooling layer, and a fully connected sub-network.

4. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to claim 3, characterized in that, The process of generating flat-scan feature gating vectors through a fully connected subnetwork includes: The original gated vector is passed through a first fully connected layer to perform feature dimensionality reduction, thereby generating a dimensionality-reduced gated vector; The dimension-reduced gating vector is increased in dimension by passing it through a second fully connected layer to generate the flat-scan feature gating vector; The fully connected sub-network includes a first fully connected layer and a second fully connected layer.

5. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to claim 1, characterized in that, The process of generating fused CT features through the bottleneck layer includes: Based on the high-level features of the plain CT scan, an attention query vector is generated, and based on the high-level features of the enhanced vascular CT scan, an attention key vector and an attention value vector are generated. Calculate the cross-attention weights based on the attention query vector, attention key vector, and attention value vector; The cross-attention weights and the residual connection features of the high-level features of the plain CT scan are normalized through layers to generate the original cross features; The spliced ​​features of the plain CT high-level features, enhanced vascular CT high-level features, and original cross features are passed through multi-level convolutional layers to generate the fused CT features; The bottleneck layer includes layer normalization and multiple levels of convolutional layers.

6. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to claim 1, characterized in that, The process of generating a vascular segmentation map for nail-fastening surgery through the decoded output layer includes: The high-level features of plain CT and high-level features of enhanced vascular CT are connected through a cross-modal skip connection layer to generate cross-modal fusion features; The decoded features of the previous level or the fused CT features are upsampled to generate upsampled features; The cross-modal fused features and upsampled features are passed through a fused convolutional layer to generate the current level decoding features or decoder output features; The decoder output features are passed through a convolutional output head to generate the vascular segmentation map for the nail-locking surgery; The vascular segmentation map for the locking and nailing surgery includes a cross-modal skip connection layer, a fusion convolutional layer, and a convolutional output head.

7. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to claim 1, characterized in that, Also includes: A sample set was constructed based on big data on the tibial anatomy of Chinese people; The sample set is used to generate the segmentation map of the vascular segmentation in the locking pin surgery through a CT image segmentation model, and the vascular segmentation map of the locking pin surgery and the true labels of the sample set are used to train and optimize the CT image segmentation model through a composite segmentation loss function; The composite segmentation loss function includes a contour Dice loss term and a cross-entropy loss term.

8. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to any one of claims 1 to 7, characterized in that, The process of generating a three-dimensional virtual model of tibial vessels includes: The processing unit uses 3D Slicer software to generate a three-dimensional virtual model of the tibial vessels from the segmented vascular map of the locking screw surgery, and then sends the three-dimensional virtual model of the tibial vessels to the mixed reality device.

9. The tibial intramedullary nail locking navigation method based on mixed reality and QR codes according to any one of claims 1 to 7, characterized in that, The process of generating navigation prompts for inserting the locking pin into the connector locking pin hole while avoiding blood vessels includes: The color change of the three-dimensional tibial blood vessel virtual model provides navigation prompts for the pin to be inserted into the connector pin hole, and the visual and audio units of the mixed reality device provide navigation prompts for the pin to approach the blood vessel.

10. A system applying the tibial intramedullary nail locking navigation method based on mixed reality and QR codes as described in any one of claims 1 to 9, characterized in that, include: Mixed reality devices, including MR glasses; The processing unit communicates with the mixed reality device and carries the CT image segmentation model; The proximal connector of the tibial intramedullary nail has a detachable first QR code. The remote locking drill bit guide sleeve is detachably equipped with a second QR code.

Images