Medical navigation method and navigation system based on mixed reality technology

Abstract

The application discloses a medical navigation method and a navigation system based on a mixed reality technology, and the medical navigation system comprises a mixed reality device, an X-ray imaging device and a surgical instrument, and the method comprises the following steps: performing orthotopic perspective and lateral perspective on a target bone region of a to-be-detected object by using the X-ray imaging device, and determining a first screwing position of a target screwing point of the target bone in a first coordinate system of the X-ray imaging device; and according to the first screwing position, first pose parameters, second pose parameters, third pose parameters and a first surgical direction of a second coordinate system of the mixed reality device relative to the first coordinate system, the relative spatial relationship between the surgical instrument and the first screwing position is displayed in the mixed reality device, so as to guide the surgical instrument to move to the screwing position and operate along the screwing direction. Based on the technical scheme, the working efficiency of positioning the ideal screwing position and the screwing direction can be effectively improved, and the positioning time is shortened.

Description

Technical Field

[0001] This disclosure relates to the field of medical device technology, specifically to a medical navigation method and navigation system based on mixed reality technology. Background Technology

[0002] Intramedullary nailing is commonly used to treat proximal femoral fractures and fractures of the shafts of long bones such as the tibia and humerus. The selection of the nail's insertion position and direction is crucial to the surgical outcome. Improper nail placement and direction can affect the establishment of the medullary canal and the placement of the master nail, leading to difficulty in the master nail entering the medullary canal, uneven stress, or deformation. In severe cases, it can cause delayed union, nonunion, or malunion of the fracture.

[0003] Currently, clinicians mostly use a "blind insertion + fluoroscopic correction" method to locate the screw insertion point. After estimating the insertion point by palpating bony landmarks, the doctor inserts a guide pin, then uses C-arm fluoroscopy to confirm the pin's position and direction. If there is a deviation, a second guide pin is inserted without removing the first, and fluoroscopy is repeated for confirmation. This process is repeated until the ideal insertion position is achieved. In this process, the initial insertion position and direction rely entirely on the doctor's experience, making it difficult to achieve the desired result on the first attempt. Subsequent adjustments require multiple fluoroscopic verifications. Therefore, current techniques suffer from low efficiency and long positioning times in locating the ideal screw insertion position and direction. Summary of the Invention

[0004] This application provides a medical navigation method and navigation system based on mixed reality technology, which can effectively shorten the positioning time for locating the ideal nail insertion position and direction, and improve the work efficiency for locating the ideal nail insertion position and direction.

[0005] In a first aspect, embodiments of this application provide a medical navigation method based on mixed reality technology, applied to a medical navigation system, the medical navigation system including a mixed reality device, an X-ray imaging device, and surgical instruments, the method comprising: The X-ray imaging device is used to perform anteroposterior and lateral fluoroscopy on the target bone region of the object to be detected, thereby obtaining X-ray anteroposterior and lateral X-ray images of the target bone. Based on the X-ray anterior and lateral images, a three-dimensional reconstruction is performed to obtain a three-dimensional skeleton model of the target bone. Based on the three-dimensional skeleton model, the first nail insertion point of the target bone in the first coordinate system of the X-ray imaging device is determined. The first coordinate system is generated based on a first marker set on the X-ray imaging device. The first nail insertion position includes the nail insertion position and nail insertion direction of the target nail insertion point. Obtain the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device; Obtain the second pose parameters of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on the second marker set on the body surface of the object to be detected; The tool coordinate system of the surgical instrument is obtained relative to the second coordinate system as a third pose parameter, and a first surgical direction of the surgical instrument is determined in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; Based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, the relative spatial relationship between the surgical instrument and the first nail insertion pose is displayed in the mixed reality device to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

[0006] Secondly, embodiments of this application provide a medical navigation method based on mixed reality technology, applied to a mixed reality device in a medical navigation system, wherein the medical navigation system further includes an X-ray imaging device and surgical instruments, and the method includes: The X-ray imaging device receives a first nail insertion pose of the target nail insertion point in the first coordinate system of the X-ray imaging device. The first nail insertion pose is determined by the X-ray imaging device after performing three-dimensional reconstruction based on the X-ray anterior and lateral X-ray images of the target bone of the object to be detected. The first coordinate system is generated based on a first marker set on the X-ray imaging device. The first nail insertion pose includes the nail insertion position and nail insertion direction of the target nail insertion point. Obtain the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device; Obtain the second pose parameters of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on the second marker set on the body surface of the object to be detected; The tool coordinate system of the surgical instrument is obtained relative to the second coordinate system as a third pose parameter, and a first surgical direction of the surgical instrument is determined in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; Based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, the relative spatial relationship between the surgical instrument and the first nail insertion pose is displayed in the mixed reality device to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

[0007] Thirdly, embodiments of this application also provide a medical navigation system based on mixed reality technology. The medical navigation system includes a mixed reality device, an X-ray imaging device, and surgical instruments. The X-ray imaging device is used to perform anteroposterior and lateral fluoroscopy on the target bone region of the object to be examined, acquiring X-ray anteroposterior and lateral X-ray images of the target bone. Based on the X-ray anteroposterior and lateral X-ray images, three-dimensional reconstruction is performed to obtain a three-dimensional bone model of the target bone. According to the three-dimensional bone model, a first nail insertion pose of the target nail insertion point in a first coordinate system of the X-ray imaging device is determined, and the first nail insertion pose is sent to the mixed reality device. The first coordinate system is generated based on a first marker set on the X-ray imaging device, and the first nail insertion pose includes the nail insertion position and direction of the target nail insertion point. The mixed reality device is used to acquire the first... The system acquires a first pose parameter of a coordinate system relative to a second coordinate system of the mixed reality device; acquires a second pose parameter of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on a second marker set on the body surface of the object to be detected; acquires a third pose parameter of the tool coordinate system of the surgical instrument relative to the second coordinate system, and determines a first surgical direction of the surgical instrument in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; and displays the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, so as to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

[0008] The scheme of this application embodiment involves using the X-ray imaging device to perform anteroposterior and lateral fluoroscopy on the target bone region of the object to be detected, acquiring X-ray anteroposterior and lateral X-ray images of the target bone; performing three-dimensional reconstruction based on the X-ray anteroposterior and lateral X-ray images to obtain a three-dimensional bone model of the target bone; and determining the first nail insertion point of the target bone in the first coordinate system of the X-ray imaging device according to the three-dimensional bone model; acquiring the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device; acquiring the second pose parameter of the body surface coordinate system of the object to be detected relative to the second coordinate system; and acquiring the third pose parameter of the tool coordinate system of the surgical instrument relative to the second coordinate system, and determining the first surgical direction of the surgical instrument in the tool coordinate system. Based on the first nail insertion pose, the first pose parameters, the second pose parameters, the third pose parameters, and the first surgical direction, the relative spatial relationship between the surgical instrument and the first nail insertion pose is displayed in the mixed reality device to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction. Thus, by determining the first nail insertion pose of the target nail insertion point through 3D reconstruction, and then unifying the first nail insertion pose and the surgical instrument into a second coordinate system based on the first pose parameters, the relative spatial relationship between the surgical instrument and the first nail insertion pose can be displayed in real time in the mixed reality device. This allows the surgeon to quickly move the surgical instrument to the nail insertion position and operate along the nail insertion direction based on visual guidance, effectively shortening the positioning time for the ideal nail insertion position and direction, and improving the efficiency of positioning the ideal nail insertion position and direction. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a system architecture diagram of the medical navigation system provided in the embodiments of this application; Figure 2 This is a schematic diagram of an embodiment of creating a body surface coordinate system based on a second marker provided in this application; Figure 3 This is a schematic diagram of an embodiment of creating a first coordinate system based on a first marker provided in this application; Figure 4 This is a schematic diagram of the first embodiment of creating a tool coordinate system based on a third marker provided in this application; Figure 5 This is a flowchart illustrating a medical navigation method based on mixed reality technology provided in the embodiments of this application; Figure 6 This is a schematic diagram of an embodiment of creating a target coordinate system on a three-dimensional skeleton model provided in this application; Figure 7 This is a schematic diagram of an embodiment of displaying the nail insertion position and direction in a mixed reality space provided in this application; Figure 8 This is a schematic diagram of an embodiment of displaying displacement deviation and a first direction indicator line in a mixed reality space provided in this application. Figure 9 This is a schematic diagram of an embodiment of displaying angle deviation and orientation adjustment indicator lines in a mixed reality space provided in this application. Figure 10 This is a schematic diagram of the coordinate transformation relationship of the medical navigation system provided in the embodiments of this application; Figure 11 This is another flowchart illustrating the medical navigation method based on mixed reality technology provided in the embodiments of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. At the same time, in the description of the embodiments of this application, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0012] The medical navigation method based on mixed reality technology in this application embodiment is applied to a medical navigation system. Please refer to... Figure 1 , Figure 1 This is a system architecture diagram of a medical navigation system provided in an embodiment of this application. The medical navigation system includes a mixed reality device 100, an X-ray imaging device 200, and surgical instruments 300.

[0013] The mixed reality device 100 can be, for example, any of a mixed reality (MR) display helmet or MR glasses. The mixed reality device 100 incorporates high-precision spatial positioning sensors (such as depth cameras and inertial measurement units) to track the three-dimensional spatial position and orientation of markers attached to objects in real time. The mixed reality device 100 also includes a data processing module (such as a built-in processor or one that communicates with an external workstation) for performing coordinate transformation calculations and generating navigation guidance information.

[0014] In some embodiments, the X-ray imaging device 200 may be, for example, a C-arm device, a G-arm device, or an O-arm device, or other intraoperative fluoroscopic imaging devices. The X-ray imaging device 200 is used to perform anteroposterior and lateral fluoroscopy on the target bone region of the patient during surgery to obtain X-ray anteroposterior and lateral images of the target bone. For example, the X-ray imaging device 200 may be a mobile C-arm X-ray machine, which can be flexibly moved to the operating table for fluoroscopic imaging.

[0015] In some embodiments, the surgical instrument 300 may be, for example, an electric drill (such as a medical bone drill with a chuck), a holder, a guide needle inserter, or other operating tools used to drive a guide needle into the bone. The surgical instrument 300 holds the guide needle, which is used to create an entry channel in the bone to guide the subsequent insertion of the intramedullary nail master nail. The surgical instrument 300 may be an electric bone drill with its front end holding the guide needle, which, under navigation guidance, drills the guide needle into the bone in a predetermined direction and position.

[0016] In some embodiments, markers can be set on the object to be detected, the X-ray imaging device 200, and the surgical instruments 300. These markers can be, for example, calibration plates with asymmetrically distributed optical recognition patterns on their surfaces. Each marker has unique pattern features, facilitating differentiation and identification by the spatial positioning system of the mixed reality device 100. After the marker is fixed to the corresponding object, the spatial position information of the marker is acquired in real time by the built-in sensors of the mixed reality device 100. Combined with the mechanical mounting dimensions between the marker and the object, real-time three-dimensional spatial positioning of the target can be achieved.

[0017] The following example uses a second marker fixed to the surface of the object to be tested. (See also...) Figure 2The second marker 41 is fixed to the surface of the object to be detected 400, with no relative movement between them. The mixed reality device 100 identifies the optical pattern of the second marker 41 in real time using its built-in spatial positioning sensor and calculates the second pose parameter of the marker coordinate system (surface coordinate system 51) relative to the coordinate system of the mixed reality device 100 (i.e., the second coordinate system), where the second pose parameter is denoted as R, and R can be a homogeneous matrix of order 4. Furthermore, the second marker 41 is fixed to the tracking target of the object to be detected (e.g., the target entry point). Based on the mechanical mounting dimensions and orientation of the second marker 41 on the surface of the object to be detected, the pose parameter of the object coordinate system relative to the surface coordinate system 51, denoted as A, can be pre-calculated. This pose parameter A can be a homogeneous matrix of order 4, containing the displacement (translation vector) of the object coordinate system relative to the origin of the surface coordinate system 51 and the orientation (rotation matrix) of the object coordinate system relative to each coordinate axis of the surface coordinate system 51. Since there is no relative movement between the second marker 41 and the surface of the object to be detected, A is constant. Based on R and A, the formula for calculating the pose parameter T (i.e., the pose parameter of the object coordinate system relative to the second coordinate system) in the mixed reality device 100 can be expressed by the following formula (1), as follows: T = R × A (1) Among them, the pose parameters T, R, and A are all fourth-order homogeneous matrices, which contain position and attitude information. Specifically, the pose parameter T can also be represented by the following formula (2), as follows: (2) In formula (2), matrix r (the 3×3 matrix in the upper left corner) represents the pose transformation of the object coordinate system relative to the second coordinate system, also known as the direction cosine matrix, which is a 3x3 unit orthogonal matrix; the three-dimensional vector (x, y, z) represents the displacement of the object coordinate system relative to the second coordinate system, that is, the coordinates of the origin of the object coordinate system in the body surface coordinate system. Thus, based on the pose parameter T, any point in the object coordinate system can be... Coordinates in the second coordinate system It can be expressed by the following formula (3), as follows: (3) Thus, the coordinates of any point in the object's coordinate system in the second coordinate system can be obtained in real time using formula (3). Taking the target entry point of the target bone in the object to be detected 400 as an example, if The coordinates of the target nail entry point in the object coordinate system are given. Then, the position coordinates of the target nail entry point in the second coordinate system can be calculated in real time using the above formula (3). Since the second marker 41 is fixedly connected to the object to be detected 400, the target nail entry point is represented as a fixed constant in the body surface coordinate system.

[0018] In some embodiments, see Figure 3 A first marker 42 can be set on the X-ray imaging device 200. The first marker 42 can be fixed to the main casing of the X-ray imaging device 200, and is rigidly connected to the X-ray imaging device 200 with no relative movement between them. The mixed reality device 100 can identify and track the first marker 42 in real time through its built-in spatial positioning sensor, thereby determining the pose of the first marker 42 in the second coordinate system; and can also create a first coordinate system for the X-ray imaging device 200 based on the first marker 42. Thus, based on the pose of the first marker 42 in the second coordinate system, the first pose parameter of the first coordinate system relative to the second coordinate system can be determined.

[0019] In some embodiments, see Figure 4 A third marker 43 can be set on the surgical instrument 300. The third marker 43 can be fixed to the main body housing of the surgical instrument 300, and is rigidly connected to the surgical instrument 300 with no relative movement between them. The mixed reality device 100 can identify and track the third marker 43 in real time through its built-in spatial positioning sensor, thereby determining the pose of the third marker 43 in the second coordinate system. Furthermore, a tool coordinate system 53 of the surgical instrument 300 can be created based on the third marker 43. Thus, based on the pose of the third marker 43 in the second coordinate system, the third pose parameters of the tool coordinate system relative to the second coordinate system can be determined.

[0020] In traditional intramedullary nailing surgery, surgeons rely on C-arm fluoroscopy to confirm the placement and direction of the guide pin. Each adjustment requires further fluoroscopy verification, exposing both the surgeon and patient to X-ray radiation. This repeated adjustment prolongs the surgery and increases the risks of infection and anesthesia. The technical solution adopted in this application transforms the coordinate systems of the X-ray imaging device 200, surgical instruments 300, and the object under test 400 to the second coordinate system of the mixed reality device 100. This allows the first nail insertion pose to be acquired preoperatively or intraoperatively in a single step, and its positioning and anchoring can be completed in the virtual space corresponding to the mixed reality device 100. Furthermore, during guide pin insertion using the surgical instruments 300, the surgeon only needs to be guided by the medical navigation information displayed on the mixed reality device 100, eliminating the need for repeated X-ray fluoroscopy verification during adjustments. This effectively reduces the number and duration of intraoperative X-ray fluoroscopy, lowers the cumulative radiation dose for both the surgeon and patient, and protects the radiation safety of both parties. At the same time, since there is no need for repeated fluoroscopy and adjustments, the operation time is greatly shortened, reducing the patient's intraoperative bleeding and infection risk, and improving the efficiency and safety of the operation.

[0021] The above is an overall description of the medical navigation system in the embodiments of this application. The following will further describe the medical navigation method based on mixed reality technology in the embodiments of this application in conjunction with the medical navigation system.

[0022] Please see Figure 5 This application provides a medical navigation method based on mixed reality technology, applied to a medical navigation system. The medical navigation system includes a mixed reality device, an X-ray imaging device, and surgical instruments. The method includes steps S501-S506, as follows: Step S501: Use X-ray imaging equipment to perform frontal and lateral fluoroscopy on the target bone region of the object to be inspected, and obtain X-ray frontal and lateral X-ray images of the target bone.

[0023] In some embodiments, the subject 400 to be tested may be, for example, a patient who needs to undergo intramedullary nailing surgery. The target bone may be, for example, a long bone such as the proximal femur (for intertrochanteric fractures of the femur, femoral neck fractures, etc.), the tibia (for tibial shaft fractures), or the humerus (for humeral shaft fractures).

[0024] In some embodiments, the subject 400 may also be a spinal disease patient who needs to undergo pedicle screw placement, in which case the first screw insertion position corresponds to the ideal insertion point and direction of the pedicle screw; or, the subject 400 may be a bone disease patient who needs to undergo percutaneous biopsy, in which case the first screw insertion position corresponds to the ideal puncture point and direction of the puncture needle.

[0025] In some embodiments, the object to be tested 400 may also be an animal patient in veterinary medicine (such as large animals such as dogs and horses), which is suitable for navigation scenarios in internal fixation surgery for animal fractures; the object to be tested 400 may also be a human bone specimen or an artificial bone model, which is suitable for orthopedic surgical teaching and training or accuracy testing and calibration of navigation equipment.

[0026] This application does not limit the specific type of the object to be detected 400. Any scenario where the medical navigation method of this application can be used to locate the position and orientation of the target bone under X-ray fluoroscopy guidance falls within the scope of protection of this application. This application also does not limit the specific location of the target bone region; it can be any long bone or irregular bone requiring surgical localization, such as the proximal femur, femoral shaft, tibia, humerus, vertebrae, or pelvis. For ease of description, the following embodiments use proximal femoral intramedullary nailing surgery as an example.

[0027] In some embodiments, before the surgery begins, the X-ray imaging device 200 can be moved to the side of the operating table, with the X-ray source and image receiver of the X-ray imaging device 200 positioned on either side of the target bone region of the subject 400. The position and angle of the X-ray imaging device 200 are adjusted so that it can clearly image the target bone region (such as the greater trochanter, femoral neck, and proximal portion of the femoral shaft). Then, the X-ray imaging device 200 is adjusted to an anteroposterior projection direction, so that the X-ray beam penetrates the target bone region along the anterior-posterior direction (i.e., the coronal plane direction) of the subject 400. Specifically, the X-ray source of the X-ray imaging device 200 is located in front of (or behind) the subject 400, and the image receiver is located behind (or in front of) the body. The X-ray beam passes through the proximal femoral region in an anterior-posterior direction, forming an anteroposterior X-ray image on the image receiver. On an anteroposterior X-ray image, the morphology of the femoral head, femoral neck, greater trochanter, lesser trochanter, and proximal femoral medullary canal can be observed, which helps determine the ideal location of the insertion point on the anteroposterior film (e.g., at the apex of the greater trochanter) and the ideal orientation of the guide needle on the anteroposterior film (e.g., at an external angle of approximately 5° to the femoral shaft axis). Furthermore, after completing the anteroposterior fluoroscopy, the X-ray imaging device 200 can be rotated 90° (or the C-arm rotated to the lateral direction) and adjusted to the lateral projection direction, allowing the X-ray beam to penetrate the target bone region along the left-right direction (i.e., the sagittal plane direction) of the subject 400. Specifically, the X-ray source of the X-ray imaging device 200 is located on the left (or right) side of the subject 400, and the image receiver is located on the right (or left) side of the body. The X-ray beam passes through the proximal femoral region along the left-right direction, forming an X-ray lateral image on the image receiver. On lateral X-ray images, the lateral contours of the femoral head and neck, as well as the lateral morphology of the femoral shaft, can be observed, which can be used to determine the ideal orientation of the guide pin on the lateral radiograph (e.g., the direction of the guide pin on the lateral radiograph coincides with or is parallel to the axis of the femoral neck).

[0028] In some embodiments, the order of frontal and lateral fluoroscopy can be interchanged; that is, lateral fluoroscopy can be performed first, followed by frontal fluoroscopy, as long as X-ray projection images of the target bone region in two orthogonal or nearly orthogonal directions can be obtained. Optionally, the shooting directions of frontal and lateral fluoroscopy are not limited to a strict 0° and 90° orthogonal relationship; they can also be two oblique directions with known included angles and not parallel (e.g., orthogonal oblique and lateral oblique), as long as there is sufficient spatial intersection between the two X-ray images, they can be used for subsequent three-dimensional reconstruction steps.

[0029] In some embodiments, taking the X-ray imaging device 200 as an example, see [link to example]. Figure 3 During fluoroscopic imaging, a first marker 42 fixed on the X-ray imaging device 200 (e.g., a C-arm device) moves along with the X-ray imaging device 200. The mixed reality device 100 tracks the first marker 42 to acquire the spatial position and orientation of the X-ray imaging device 200 in a second coordinate system in real time. When the X-ray imaging device 200 rotates from an orthopedic to a lateral view, the mixed reality device 100 synchronously tracks the pose change of the first marker 42, thereby acquiring precise spatial pose information of the X-ray imaging device 200 during orthopedic and lateral fluoroscopy, providing a basis for spatial mapping between X-ray lateral images.

[0030] In some embodiments, the image acquisition parameters (such as tube voltage, tube current, exposure time, etc.) of the frontal and lateral fluoroscopy can be appropriately adjusted according to factors such as the body thickness and bone density of the object to be inspected 400, so as to obtain X-ray images with clarity that meet the requirements of three-dimensional reconstruction. The acquired X-ray frontal and lateral images can be stored in the built-in memory of the X-ray imaging device 200 or in an external workstation for subsequent three-dimensional reconstruction processing.

[0031] The aforementioned anteroposterior and lateral X-ray images can be stored in the Digital Imaging and Communications in Medicine (DICOM) format to facilitate transmission and processing between different devices.

[0032] In some embodiments, to ensure the accuracy of subsequent three-dimensional reconstruction, during fluoroscopic imaging, it should be ensured that the target skeletal region of the object under test 400 (such as the proximal femur) is clearly visible in both the anteroposterior and lateral X-ray images, and does not significantly overlap with other anatomical structures (such as the contralateral femur, pelvis, etc.). If necessary, the position of the object under test 400 can be adjusted (such as slight internal or external rotation of the hip joint) to obtain the optimal projection angle.

[0033] Step S502: Perform three-dimensional reconstruction based on the X-ray anterior and lateral images to obtain a three-dimensional skeleton model of the target bone, and determine the first nail insertion pose of the target bone in the first coordinate system of the X-ray imaging device based on the three-dimensional skeleton model.

[0034] The first coordinate system is generated based on the first marker set on the X-ray imaging equipment, and the first nail insertion pose includes the nail insertion position and the nail insertion direction of the target nail insertion point.

[0035] In some embodiments, a three-dimensional bone model of the target bone is obtained by three-dimensional reconstruction based on X-ray anterior and lateral images, which can be achieved using a three-dimensional reconstruction algorithm based on X-ray images.

[0036] Specifically, the contour features and internal structural features of the target bone are first extracted from the anteroposterior and lateral X-ray images, respectively. For example, image segmentation algorithms (such as thresholding, region growing, active contour models, or deep learning-based bone segmentation networks) are used to extract anatomical features such as the outer contour of the proximal femur, the cortical boundary, the medullary canal contour, the apex of the greater trochanter, and the femoral neck axis from the anteroposterior X-ray image, while corresponding anatomical features are extracted from the lateral X-ray image. Then, the correspondence between feature points in the anteroposterior and lateral X-ray images is established. Since the anteroposterior and lateral X-ray images are obtained by projecting the same target bone from two different angles, the same anatomical feature point has different projection positions in the two images. Therefore, feature matching algorithms (such as feature matching based on scale-invariant feature transform, feature matching based on local descriptors, etc.) can be used to establish the correspondence between the same feature points in the two images. Next, based on the geometric imaging parameters of the X-ray imaging device 200, the feature points on the two-dimensional image are reprojected into three-dimensional space, thereby obtaining a three-dimensional bone model of the target bone. The geometric imaging parameters of the X-ray imaging device 200 include the spatial position of the X-ray source, the spatial position and orientation of the image receiver, the distance from the X-ray source to the image receiver (i.e., source-image distance), and pixel size. Furthermore, these geometric imaging parameters can be obtained from the DICOM file of the X-ray imaging device 200 or through a pre-calibrated procedure; this specification does not impose specific limitations.

[0037] In some embodiments, the three-dimensional reconstruction method can be any of the following: Poisson surface reconstruction, Delaunay triangulation, and moving least squares method. The three-dimensional reconstruction method can also be a three-dimensional reconstruction method based on a statistical shape model, which registers a pre-established statistical shape model of the skeleton (including the mean and main variation patterns of the skeleton shape) with X-ray anteroposterior and lateral X-ray images. By adjusting the parameters of the statistical shape model, the anteroposterior and lateral projections of the model are made to achieve the best match with the actual X-ray images, thereby realizing personalized three-dimensional skeleton model reconstruction.

[0038] In some embodiments, if the X-ray imaging device 200 has a three-dimensional imaging function (such as cone-beam CT mode), the three-dimensional volume data of the target bone can be directly acquired using the X-ray imaging device 200, and then a three-dimensional bone model can be extracted from the three-dimensional volume data.

[0039] In some embodiments, after obtaining a three-dimensional skeletal model of the target bone, the insertion position and direction of the target screw insertion point are determined within the three-dimensional skeletal model. Specifically, the insertion position and direction can be automatically determined based on skeletal anatomical features. For example, by identifying anatomical landmarks in the three-dimensional skeletal model (such as the apex of the greater trochanter, the center of the femoral neck, the femoral shaft axis, etc.), and based on preset anatomical rules and clinical experience (such as "the insertion point is located at the intersection of the medial edge of the apex of the greater trochanter and the femoral neck axis"), the insertion position is automatically calculated and determined. Then, the insertion direction is determined according to a preset insertion direction (approximately 4°~6° outward deviation from the femoral shaft axis). Thus, the insertion position and direction can be determined within the three-dimensional skeletal model. Of course, after determining the insertion position and direction in the three-dimensional skeletal model, medical personnel can also fine-tune the insertion position and direction. Furthermore, after determining the insertion position and direction, the first insertion pose of the insertion position and direction in the first coordinate system can be determined and denoted as... .

[0040] In some embodiments, a target coordinate system is constructed in the three-dimensional skeleton model, wherein the origin of the target coordinate system is the nail insertion position of the target nail insertion point, the direction of the first coordinate axis in the target coordinate system is the nail insertion direction of the target nail insertion point, and the first coordinate axis is any coordinate axis in the target coordinate system; the pose of the target coordinate system in the first coordinate system is used as the first nail insertion pose.

[0041] Specifically, a target coordinate system can be constructed in the 3D skeleton model based on the pin insertion position and direction. The pose of the target coordinate system in the first coordinate system can then be used as the first pin insertion pose. Specifically, when creating the target coordinate system, the pin insertion position can be set as the origin of the target coordinate system, and the pin insertion direction can be set as the direction of the first coordinate axis. The first coordinate axis can be the Z-axis of the target coordinate system; of course, it can also be the X-axis or Y-axis. The following example uses the origin of the target coordinate system as the pin insertion position and the Z-axis as the pin insertion direction. See [link to example]. Figure 6 After creating the 3D skeletal model 60, a target coordinate system 63 can be constructed with the nail insertion position 61 as the origin and the nail insertion direction 62 as the Z-axis. The X and Y axes of the target coordinate system 63 can be automatically generated based on the Z-axis direction. Thus, by creating the target coordinate system through the nail insertion position and direction, subsequent homogeneous matrix transformations of the target coordinate system are only required to simultaneously complete the position and orientation coordinate transformations, simplifying the calculation process and reducing the computational load. Furthermore, the pose of the target coordinate system in the first coordinate system establishes a precise mapping relationship between the target nail insertion point and the 3D skeletal model, allowing the target nail insertion point to freely transform between the first coordinate system, the second coordinate system, and the body surface coordinate system, and ensuring higher accuracy of the transformed target nail insertion pose.

[0042] In some embodiments, the X and Y axes of the target coordinate system are used only to complete the mathematical integrity of the rotation matrix and construct the fourth-order homogeneous transformation matrix, and are not used as the basis for navigation guidance. In the subsequent navigation process, the origin position of the target coordinate system (i.e., the entry point position) and the Y-axis direction of the target coordinate system (i.e., the entry direction) are used for positioning guidance.

[0043] In some embodiments, the X-axis and Y-axis of the target coordinate system can also be generated in other ways. For example, an arbitrary direction perpendicular to the Z-axis can be selected as the X-axis direction, and then orthogonal X-axis, Y-axis, and Z-axis can be obtained through orthogonalization. This application does not limit the specific generation method of the X-axis and Y-axis in its embodiments.

[0044] Step S503: Obtain the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device.

[0045] In some embodiments, see Figure 3 The X-ray imaging device 200 is provided with a first marker 42. The first marker 42 can be fixed to the main body shell of the X-ray imaging device 200 by means of magnetic base, clamp or bolt, etc. The first marker 42 and the X-ray imaging device 200 are rigidly connected and there is no relative movement between the two.

[0046] Specifically, since the first marker 42 is rigidly connected to the X-ray imaging device 200, there is no relative movement between them. Therefore, the coordinate system created based on the first marker 42 can be used as the first coordinate system 52 of the X-ray imaging device 200. Furthermore, the surface of the first marker 42 is provided with asymmetrically distributed optical recognition patterns, each pattern having a unique identifier, to facilitate identification and tracking by the spatial positioning system of the mixed reality device 100. The mixed reality device 100 acquires the optical patterns or spatial features of the first marker 42 in real time using its built-in spatial positioning sensors (such as depth cameras, inertial measurement units, etc.), and calculates the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device 100 through image recognition and spatial positioning algorithms. The first pose parameter can be directly read from the data processing module of the mixed reality device 100.

[0047] Specifically, the mixed reality device 100 identifies the first marker 42 in real time through its built-in spatial positioning sensor. It can obtain the pose parameter R1 (i.e., 3×3 rotation matrix) of the marker coordinate system (first coordinate system 52) where the first marker 42 is located relative to the second coordinate system of the mixed reality device 100, and the three-dimensional coordinate d1 (i.e., 3×1 translation vector) of the origin of the first coordinate system 52 in the second coordinate system. The first pose parameter is constructed based on the pose parameter R1 and d1. The first pose parameter can be a 4th order homogeneous matrix. For details, please refer to formula (2). For the sake of brevity, it will not be elaborated here.

[0048] Step S504: Obtain the second pose parameters of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on the second marker set on the body surface of the object to be detected.

[0049] In some embodiments, see Figure 2The second marker 41 is fixed to the surface of the subject 400 and rigidly connected to it, with no relative movement between them. Taking proximal femoral intramedullary nailing surgery as an example, the second marker 41 can be fixed to the affected thigh near the knee, or to a bony prominence such as the anterior superior iliac spine where surface displacement is minimal. The second marker 41 can be fixed to the patient's skin surface using medical double-sided adhesive tape, bandages, negative pressure suction devices, or disposable sterile protective sleeves to ensure no relative displacement between the second marker 41 and the subject 400 during surgery. Furthermore, the surface of the second marker 41 is provided with asymmetrically distributed optical recognition patterns, each pattern having a unique identifier. Thus, based on the geometry of the second marker 41, a marker coordinate system (i.e., surface coordinate system 51) for the second marker 41 can be established. The origin of the surface coordinate system 51 can be, for example, the geometric center of the second marker 41, and the coordinate axis directions can be, for example, specific edge directions of the second marker 41.

[0050] Specifically, a surface coordinate system 51 for the subject 400 can be defined based on the fixed position and orientation of the second marker 41 on the surface of the subject 400. The surface coordinate system 51 can, for example, have its origin at the geometric center of the second marker 41, its Z-axis perpendicular to the body surface, and its X-axis or Y-axis along anatomical directions (such as the head-side or lateral direction). Optionally, the surface coordinate system 51 can also have its origin at a specific reference point on the second marker 41, and its coordinate axis directions along a specific edge of the second marker 41. Because the second marker 41 is rigidly connected to the surface of the subject 400, there is no relative movement between the surface coordinate system 51 and the surface of the subject 400. When the subject 400 moves due to breathing, repositioning, or surgical procedures, the surface coordinate system 51 moves synchronously with the subject 400.

[0051] Specifically, the mixed reality device 100 identifies the second marker 41 in real time through its built-in spatial positioning sensor. It can obtain the pose parameter R2 (i.e., 3×3 rotation matrix) of the marker coordinate system (body surface coordinate system 51) of the second marker 41 relative to the second coordinate system of the mixed reality device 100, as well as the three-dimensional coordinate d2 (i.e., 3×1 translation vector) of the origin of the body surface coordinate system 51 in the second coordinate system. The second pose parameter is constructed based on the pose parameter R2 and d2. The second pose parameter can be a 4th order homogeneous matrix. For details, please refer to formula (2). For the sake of brevity, it will not be elaborated here.

[0052] Step S505: Obtain the third pose parameters of the surgical instrument's tool coordinate system relative to the second coordinate system, and determine the first surgical direction of the surgical instrument in the tool coordinate system.

[0053] The tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument.

[0054] In some embodiments, such as Figure 4 As shown, the third marker 43 is fixed to the outer casing of the surgical instrument 300 and is rigidly connected to the surgical instrument 300, with no relative movement between them. Specifically, the third marker 43 can be detachably fixed to the outer casing of the surgical instrument 300 by means of clamps, straps, magnetic bases, or medical tape. The surface of the third marker 43 is provided with asymmetrically distributed optical recognition patterns, each pattern having a unique identifier, so as to facilitate identification and tracking by the spatial positioning system of the mixed reality device 100.

[0055] In some embodiments, the third marker 43 may be fixed on the surgical instrument 300 in a location that is not easily obstructed (e.g., the side or top of the drill body) to ensure that the spatial positioning sensor of the mixed reality device 100 can continuously track the third marker 43 during the operation.

[0056] Specifically, since the third marker 43 is rigidly connected to the surgical instrument 300, there is no relative movement between them. Therefore, the coordinate system created based on the third marker 43 can be directly used as the tool coordinate system of the surgical instrument 300. Furthermore, the surface of the third marker 43 is provided with asymmetrically distributed optical recognition patterns, each pattern having a unique identifier, to facilitate identification and tracking by the spatial positioning system of the mixed reality device 100. The mixed reality device 100 acquires the optical patterns or spatial features of the third marker 43 in real time through its built-in spatial positioning sensors (such as depth cameras, inertial measurement units, etc.), and calculates the third pose parameters of the tool coordinate system 53 relative to the second coordinate system of the mixed reality device 100 through image recognition and spatial positioning algorithms. These third pose parameters can be directly read from the data processing module of the mixed reality device 100.

[0057] Specifically, the mixed reality device 100 identifies the third marker 43 in real time through its built-in spatial positioning sensor, and can obtain the pose parameter R3 (i.e., 3×3 rotation matrix) of the tool coordinate system 53 created based on the third marker 43 relative to the second coordinate system, as well as the three-dimensional coordinate d3 (i.e., 3×1 translation vector) of the origin of the tool coordinate system 53 in the second coordinate system. The third pose parameter is constructed based on the pose parameter R3 and d3. The third pose parameter can be a 4th order homogeneous matrix. For details, please refer to formula (2). For the sake of brevity, it will not be elaborated here.

[0058] In some embodiments, during surgery, when the surgeon moves and rotates the surgical instrument 300, the third marker 43 moves and rotates along with the surgical instrument 300. The mixed reality device 100 continuously tracks the third marker 43 and updates the third pose parameters in real time at a predetermined frequency (e.g., 30 or 60 frames per second) to ensure that the spatial pose of the tool coordinate system in the second coordinate system is always consistent with the actual physical position of the surgical instrument 300.

[0059] In some embodiments, a first surgical direction of the surgical instrument 300 needs to be determined in the tool coordinate system for subsequent comparison with the target insertion direction. If the surgical instrument 300 includes a clamped guide pin, the first surgical direction of the surgical instrument 300 refers to the axial direction of the guide pin clamped by the surgical instrument 300. Specifically, when the surgical instrument 300 is an electric drill, its front end clamps a guide pin, which extends along its own axial direction, and this axial direction is the first surgical direction.

[0060] Specifically, due to the rigid connection between the surgical instrument 300 and the third marker 43, the first surgical direction of the surgical instrument 300 is represented as a fixed constant in the tool coordinate system. Specifically, after creating the tool coordinate system based on the third marker 43, the direction vector v1 of the guide needle axis in the tool coordinate system can be directly calculated based on the actual mechanical mounting dimensions and orientation of the third marker 43 on the surgical instrument 300, and the direction vector v1 can be stored in the mixed reality device's storage space for subsequent use.

[0061] In some embodiments, to reduce computational load, the first surgical direction can be set as the direction of the second coordinate axis in the tool coordinate system, where the second coordinate axis can be any one of the X-axis, Y-axis, and Z-axis in the tool coordinate system. Specifically, when creating the tool coordinate system based on the third marker 43, the second coordinate axis in the tool coordinate system can be parallel to the first surgical direction. In this case, the direction of the second coordinate axis can represent the first surgical direction. For example, when the surgical instrument 300 is an electric drill with a guide needle clamped at its front end, the Y-axis in the tool coordinate system can be parallel to the guide needle axis direction, and the Y-axis direction is the first surgical direction. Of course, the Z-axis or X-axis in the tool coordinate system can also be parallel to the guide needle axis direction, so that the Z-axis or X-axis direction of the tool coordinate system can represent the first surgical direction.

[0062] Step S506: Based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, display the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

[0063] In some embodiments, a second nail insertion pose in the body surface coordinate system can be determined based on a first nail insertion pose, a first pose parameter, and a second pose parameter; then, a third nail insertion pose in the second coordinate system can be determined based on the second nail insertion pose and the second pose parameter; and finally, the third nail insertion pose can be displayed in the mixed reality space of the mixed reality device.

[0064] Specifically, the third nail insertion pose of the target nail insertion point in the second coordinate system can be determined based on the first pose parameter and the first nail insertion pose; correspondingly, the third nail insertion pose of the target nail insertion point in the second coordinate system can also be determined based on the second pose parameter and the second nail insertion pose of the target nail insertion point in the body surface coordinate system.

[0065] For example, taking the first pose parameter as The first nail-feeding position is The third nail-feeding position is For example, the third pin-feeding pose can be represented by the following formula (4), as follows: (4) Among them, formula (4) , and All of them can be 4th order homogeneous square matrices. Thus, the third pin pose can be obtained in real time through formula (4).

[0066] Accordingly, the second pose parameters are... The second nail-feeding position is For example, the third pin-feeding pose can be represented by the following formula (5), as follows: = × (5) Therefore, formula (5) can be substituted into formula (4) to determine... ,in, It can be expressed by the following formula (6), as follows: (6) Among them, it can be determined in real time by formula (6). Since the second marker on the surface of the object to be tested is fixed to the object, the nail insertion pose and direction of the target nail insertion point are invariants in the surface coordinate system, that is, the second nail insertion pose calculated by formula (6) is... This is a fixed constant. Therefore, even if the C-arm or the mixed reality system is moved, The pose remains unchanged. Thus, based on the pre-calibrated first pose parameters, second pose parameters, and first insertion pose, the second insertion pose of the target insertion point in the body surface coordinate system can be determined. .

[0067] In some embodiments, in order to reduce the computational load of the third nail insertion pose of the target nail insertion point in the second coordinate system and to improve the real-time performance of the acquired third nail insertion pose, after determining the second nail insertion pose of the target nail insertion point in the body surface coordinate system for the first time based on the first nail insertion pose, the first pose parameter, and the second pose parameter, the second nail insertion pose can be stored in the storage space of the mixed reality device. This allows the third nail insertion pose of the target nail insertion point in the second coordinate system to be determined subsequently based on the second pose parameter and the second nail insertion pose read from the storage space when acquiring the third nail insertion pose. Thus, when subsequently obtaining the third nail insertion pose of the target nail insertion point in the second coordinate system, the second nail insertion pose can be directly read from the storage space in the mixed reality device. Then, based on the real-time acquired second pose parameters and the second nail insertion pose, the third nail insertion pose of the target nail insertion point in the second coordinate system can be obtained in real time. Since the second nail insertion pose is directly read from the storage space in the mixed reality device, the computation time required for each calculation of the second nail insertion pose is saved, and the real-time computation load can be effectively reduced, reducing the occupation of computing resources. This improves the response speed and display frame rate of the medical navigation system, ensuring the real-time performance and smoothness of the guidance information displayed in the mixed reality device.

[0068] Specifically, based on the second nail insertion pose and its parameters, the third nail insertion pose in the second coordinate system can be determined. Based on this third pose, the nail insertion position and direction can be displayed in the mixed reality space of the mixed reality device. Alternatively, the target skeleton can be displayed in the mixed reality space, and the nail insertion position and direction can be displayed at the corresponding positions on the target skeleton. For example... Figure 7 As shown, the target bone 70 can be displayed in mixed reality space, along with the screw insertion position 71 and the screw insertion direction indicated by the guide line 72, to provide intuitive visual guidance to the surgeon. Thus, by displaying a virtual target bone in mixed reality space and overlaying the screw insertion position and guide line on it, the surgeon can directly see the screw insertion position and direction. This allows for guidance in adjusting the posture of surgical instruments via the guide line, making the adjustment of the screw insertion direction visible and quantifiable. This solves the problem of difficulty in accurately positioning the screw insertion direction due to insensitivity to angles in traditional surgery.

[0069] In some embodiments, the second point of action of the surgical instrument in the second coordinate system of the mixed reality device can be determined based on the first point of action position and the third pose parameter of the surgical instrument in the tool coordinate system; the second surgical direction of the first surgical direction in the second coordinate system can be determined based on the first surgical direction and the third pose parameter; and the instrument point of action and the first surgical direction can be displayed in the second coordinate system of the mixed reality device based on the second point of action position and the second surgical direction.

[0070] In some embodiments, when the first surgical direction is the direction of the second coordinate axis in the tool coordinate system, the second surgical direction in the second coordinate system can be determined according to the direction of the second coordinate axis and the third pose parameter, and the instrument point of action and the first surgical direction can be displayed in the second coordinate system of the mixed reality device according to the position of the second point of action and the second surgical direction.

[0071] Specifically, the first surgical direction in the second coordinate system is obtained by multiplying the first surgical direction by the rotation matrix in the third pose parameter. The second point of action position can be obtained by performing a homogeneous coordinate transformation between the first point of action position and the third pose parameter. In this way, the first surgical direction and the instrument point of action can be accurately determined in the second coordinate system through the second point of action position and the second surgical direction. This allows the instrument point of action and the first surgical direction to be displayed in real time in the mixed reality space of the mixed reality device, and ensures that the displayed instrument point of action and the first surgical direction are more accurate, so as to provide doctors with intuitive surgical instrument pose guidance.

[0072] In some embodiments, the instrument point of action refers to the location on the surgical instrument that interacts with the target bone. When the surgical instrument holds a guide needle, the instrument point of action refers to the position of the tip (i.e., the foremost point) of the guide needle held by the surgical instrument in the tool coordinate system.

[0073] In some embodiments, the nail insertion position can be displayed in a mixed reality space using a predefined display method. This predefined display method may involve overlaying a virtual marker of a preset shape at the nail insertion position in the mixed reality space. The virtual marker may include any one of the following: a crosshair, a bullseye pattern, a colored sphere, a ring, a star-shaped marker, or a flashing light spot. Using these virtual markers, doctors can visually identify the precise location of the target nail insertion point on the actual patient's skin.

[0074] Furthermore, in the mixed reality space, directional indicator lines or rays extend from the screw insertion point along the screw insertion direction. These directional indicator lines can be any of the following: solid colored lines, dashed colored lines, gradient lines, rays marked with arrows, or path indicators composed of multiple directional arrows. The length of the directional indicator lines can be a preset fixed length, a dynamically adjustable length according to surgical needs, or a length extending to the cortex opposite the target bone or to the end of the medullary canal. The colors of the directional indicator lines include, but are not limited to, striking colors such as red, blue, green, or orange to facilitate rapid identification within the surgical field.

[0075] In some embodiments, the displacement deviation between the instrument point of action and the nail insertion position of the surgical instrument is displayed in the mixed reality space of the mixed reality device, and a first direction indicator line pointing from the instrument point of action to the nail insertion position is displayed in the mixed reality space to guide the instrument point of action to move to the nail insertion position.

[0076] Specifically, since the third nail-feeding pose includes the nail-feeding position and nail-feeding direction of the target nail-feeding point, the nail-feeding position of the target nail-feeding point in the third nail-feeding pose can be obtained, that is, the real-time nail-feeding coordinates of the target nail-feeding point in the second coordinate system can be obtained; thus, based on the nail-feeding position of the target nail-feeding point and the position of the second point of action in the third nail-feeding pose, the displacement deviation is determined, and the displacement deviation is the distance from the nail-feeding position of the target nail-feeding point to the position of the second point of action in the third nail-feeding pose.

[0077] In some embodiments, the displacement deviation can be displayed in a first preset display mode, wherein the first preset display mode includes a first set display format, which may include one or more of the following: numerical format, text format, display color, and display font. Taking the first set display format as numerical and text format as an example, the displacement deviation can be displayed in real time in mixed reality space in numerical and text format, such as displaying "Tip offset: 2.3mm in front, 1.5mm outward, 0.8mm above" or "Distance from target: 2.8mm", etc. The display position of the displacement deviation can be between the insertion position and the instrument's point of action.

[0078] In some embodiments, the mixed reality device also displays a first directional indicator line in the mixed reality space, pointing from the instrument's point of action to the insertion position, to indicate the direction the surgeon needs to move. The first directional indicator line is a virtual line extending from the second point of action to the insertion position. The virtual line can take at least one of the following forms: a solid colored line, a dashed colored line, a gradient line, a ray with arrow markings, or a path indicator composed of multiple directional arrows. Thus, the first directional indicator line visually indicates the direction the surgeon needs to move the surgical instrument.

[0079] For example, see Figure 8The system can display the target bone 70 in mixed reality space, and the insertion position 71 can be displayed using red dots. A first direction indicator line 73 points from the instrument's point of action to the insertion position 71, and the indicator line 73 has an arrow pointing in the same direction. Below the indicator line 73, a displacement deviation of "3.2mm" is displayed in red. Additionally, the instrument's point of action 74 and the first surgical direction are also displayed in mixed reality space via a solid line 75 with an arrowhead. By displaying the specific value of the displacement deviation and the first direction indicator line in real time, the surgeon can accurately know the difference between the instrument's point of action and the insertion position, allowing for targeted adjustments to the direction and amount of movement. This avoids the inefficient operation of adjusting the surgical instrument's displacement based on subjective experience in traditional surgery, effectively improving the efficiency of moving the instrument's point of action to the insertion position.

[0080] In some embodiments, in the mixed reality space, a first surgical direction is displayed by a second direction indicator line, and a screw insertion direction is displayed by a third direction indicator line. An angular deviation between the second direction indicator line and the third direction indicator line and a surgical instrument orientation adjustment indicator line are also displayed. The surgical instrument is guided to rotate by the angular deviation and orientation adjustment indicator lines so that the second direction indicator line overlaps with the third direction indicator line.

[0081] Specifically, when the displacement deviation between the instrument's point of action and the nail insertion position is less than a first preset threshold (e.g., 1.0mm, 1.5mm, or 2.0mm), the instrument's point of action is determined to have moved to the nail insertion position. At this point, the mixed reality device automatically switches from position guidance mode to direction guidance mode, highlighting the direction guidance information in the mixed reality space. The first preset threshold can be set or adjusted according to the surgical precision requirements and the doctor's operating habits.

[0082] In some embodiments, directional guidance and position guidance can be performed simultaneously. That is, regardless of whether the instrument's point of action has moved to the nail insertion position, the mixed reality device continuously displays directional indicator lines, angle deviation, and directional adjustment indicator lines for the surgical direction and nail insertion direction in the mixed reality space. This allows the doctor to adjust the surgical direction simultaneously while moving the surgical instrument, achieving parallel guidance of position and direction without having to wait for the position to be aligned before making directional adjustments.

[0083] In some embodiments, doctors can also manually switch the display mode via voice commands, gesture commands, or foot switches to enhance the flexibility and operability of the medical navigation system.

[0084] In some embodiments, since the third nail insertion pose includes the nail insertion position and the nail insertion direction of the target nail insertion point, the nail insertion direction of the target nail insertion point in the third nail insertion pose can be obtained, that is, the real-time nail insertion direction of the target nail insertion point in the second coordinate system can be obtained; thus, based on the nail insertion direction of the target nail insertion point in the third nail insertion pose and the second surgical direction, the angle deviation is determined, and the angle deviation is the angle between the nail insertion direction of the target nail insertion point in the third nail insertion pose and the second surgical direction.

[0085] In some embodiments, the angle deviation can be displayed in a second preset display mode, wherein the second preset display mode includes a second set display format, which may include one or more of the following: numerical format, text format, display color, and display font. Taking the second set display format as numerical and text format as an example, the displacement deviation can be displayed in real time in the mixed reality space in numerical and text format, for example, displaying "Angle deviation: 3.2°".

[0086] In some embodiments, the direction adjustment indicator line is used to indicate the direction adjustment of the surgical direction towards the screw insertion direction. The direction adjustment indicator line is at least one of the following: an arc-shaped arrow, a curved arrow, or an arc segment with a direction indicator mark (such as an arrow) displayed between the second and third direction indicator lines. The direction of the direction adjustment indicator line (e.g., clockwise or counterclockwise, or the direction from the start to the end of the arc) indicates the rotation direction of the surgical instrument as it adjusts from the current surgical direction to the screw insertion direction. The arc length or angle range of the direction adjustment indicator line is related to the magnitude of the angular deviation. For example, when the surgical direction deviates 5° to the right relative to the screw insertion direction, the direction adjustment indicator line is displayed as a rightward-curving arc-shaped arrow, the arc length or angle range of which corresponds to the 5° deviation. As the surgeon rotates the surgical instrument, the angular deviation gradually decreases, and the arc length or angle range of the direction adjustment indicator line shortens synchronously until the surgical direction is aligned with the screw insertion direction, at which point the direction adjustment indicator line automatically disappears or switches to a green confirmation state.

[0087] For example, see Figure 9The system can display the target bone 70 in mixed reality space. When the instrument's point of action moves to the insertion position, the insertion direction is indicated by a yellow dashed line 76, the first surgical direction by a solid line 75 with an arrow, the angle deviation is displayed in red as "5.4°", and the direction adjustment indicator line is displayed using a yellow arc 77 with an arrow, where the arrow of the yellow arc 77 points from the solid line 75 to the dashed line 76. In this way, the angle deviation and the yellow arc 77 can guide the rotation of the surgical instrument until the solid line 75 and the dashed line 76 overlap, ending the adjustment. This display of angle deviation and direction adjustment indicator lines allows the surgeon to accurately know the difference between the current surgical direction and the insertion direction of the surgical instrument, thus enabling targeted rotation of the surgical instrument. This avoids the inefficient operation of rotating based on subjective experience in traditional surgery and effectively improves the efficiency of moving the surgical instrument from the first surgical direction to the insertion direction.

[0088] In some embodiments, Figure 10 This is a schematic diagram of the coordinate transformation relationship of the medical navigation system provided in the embodiments of this application. Taking an X-ray imaging device as a C-arm device as an example, as follows... Figure 10 As shown, a first marker is fixed to the C-arm device, a second marker is fixed to the surface of the object to be examined, and a third marker is fixed to the surgical instrument 300. A coordinate system for the C-arm device (i.e., the first coordinate system) can be established based on the first marker, a coordinate system for the object surface can be established based on the second marker, and a coordinate system for the tool can be established based on the third marker. The mixed reality device 100 has a second coordinate system. Figure 10 middle, This represents the transformation relationship between the first coordinate system and the second coordinate system, i.e., the first pose parameter; This represents the transformation relationship between the body surface coordinate system and the second coordinate system, i.e., the second pose parameter; The transformation relationship between the tool coordinate system and the second coordinate system is represented by the third pose parameter. During the surgical planning phase, the C-arm device performs anteroposterior and lateral fluoroscopy on the target bone region of the object to be examined, acquiring anteroposterior and lateral X-ray images. Based on these two images, a three-dimensional reconstruction is performed to obtain a three-dimensional bone model of the target bone. The insertion position 61 and insertion direction 62 of the target insertion point are determined on the three-dimensional bone model, thus determining the first insertion pose of the target insertion point in the first coordinate system. During the coordinate transformation stage, based on Figure 10 The coordinate transformation relationship shown allows the mixed reality device to obtain the first pose parameter in real time by tracking the first marker. The second pose parameters are obtained in real time by tracking the second marker. The third pose parameters are obtained in real time by tracking the third marker. Since the three pose parameters mentioned above have been unified to the second coordinate system, the screw insertion position and direction at the target insertion point, the relative spatial relationship between the object to be tested and the surgical instrument, can be displayed in real time in the second coordinate system. Furthermore, it is also possible to... , and The second nail insertion pose of the target nail insertion point relative to the body surface coordinate system is determined. And because Since it is a fixed constant, therefore, it can be Stored in the mixed reality device's storage space. And, during the navigation guidance phase, the mixed reality device, based on the second pose parameters... Second nail position It can quickly and accurately determine the pose of the target's entry point in the second coordinate system. Simultaneously, the mixed reality device 100 determines the pose based on the third pose parameters. This allows for real-time acquisition of the position and orientation of surgical instruments in the second coordinate system. Based on this, the mixed reality device can display the relative positional and directional deviations between the surgical instruments and the target insertion point in real-time within the mixed reality space, guiding the surgeon to move the instruments to the target insertion position and operate along the target insertion direction. Furthermore, if the first insertion pose of the target insertion point is represented by the target coordinate system, then the surgeon can determine the insertion point based on the target coordinate system and the second pose parameters. and third pose parameters Determine the fourth nail-feeding pose of the target nail-feeding point in the tool coordinate system. .

[0089] Thus, through Figure 10 The coordinate transformation relationship shown transforms the first coordinate system, the body surface coordinate system, and the tool coordinate system into the second coordinate system. This enables precise anchoring of the target insertion point on the patient and real-time tracking of surgical instruments, providing doctors with intuitive and accurate navigation guidance. Furthermore, by integrating the three coordinate systems into the second coordinate system, the relative spatial relationship between the surgical instrument and the target insertion point can be directly calculated and displayed within the same coordinate system. This avoids the complex operations of frequently switching and transforming between different coordinate systems, improving the computational efficiency and guidance accuracy of the medical navigation system, and helping to shorten surgical time and reduce radiation exposure.

[0090] This application also provides a medical navigation method based on mixed reality technology, applied to a mixed reality device in a medical navigation system. The medical navigation system further includes X-ray imaging equipment and surgical instruments, such as... Figure 11 As shown, the method includes: Step S110: Receive the first nail insertion pose of the target nail insertion point in the first coordinate system of the X-ray imaging device, sent by the X-ray imaging device.

[0091] The first nail insertion pose is determined by the X-ray imaging device after performing three-dimensional reconstruction based on the X-ray anterior and lateral X-ray images of the target bone of the object to be detected. The first coordinate system is generated based on the first marker set on the X-ray imaging device. The first nail insertion pose includes the nail insertion position and the nail insertion direction of the target nail insertion point.

[0092] Step S111: Obtain the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device.

[0093] Step S112: Obtain the second pose parameters of the object to be detected relative to the second coordinate system.

[0094] The body surface coordinate system is generated based on a second marker set on the body surface of the object to be detected.

[0095] Step S113: Obtain the third pose parameters of the surgical instrument's tool coordinate system relative to the second coordinate system, and determine the first surgical direction of the surgical instrument in the tool coordinate system.

[0096] The tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument.

[0097] Step S114: Based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, display the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device, so as to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

[0098] In some embodiments, displaying the relative spatial relationship between surgical instruments and a first screw insertion pose in a mixed reality device includes: The displacement deviation between the instrument's point of action and the nail insertion position is displayed in the mixed reality space of the mixed reality device, and a first direction indicator line pointing from the instrument's point of action to the nail insertion position is displayed in the mixed reality space to guide the instrument's point of action to move to the nail insertion position.

[0099] In some embodiments, when the instrument's point of action moves to the nail insertion position, displaying the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device includes: In the mixed reality space, the first surgical direction is displayed by a second direction indicator line, and the screw insertion direction is displayed by a third direction indicator line. The angular deviation between the second direction indicator line and the third direction indicator line is also displayed, as well as the direction adjustment indicator line of the surgical instrument. The surgical instrument is guided to rotate by the angular deviation and the direction adjustment indicator line so that the second direction indicator line overlaps with the third direction indicator line.

[0100] In some embodiments, determining the first nail insertion point of the target bone in the first coordinate system of the X-ray imaging device based on the three-dimensional bone model includes: In the 3D skeleton model, a target coordinate system is constructed, where the origin of the target coordinate system is the nail insertion position of the target nail insertion point, the direction of the first coordinate axis in the target coordinate system is the nail insertion direction of the target nail insertion point, and the first axis can be any coordinate axis in the target coordinate system; The pose of the target coordinate system in the first coordinate system is taken as the first nail-feeding pose.

[0101] In some embodiments, displaying the relative spatial relationship between the surgical instrument and the first screw insertion pose in a mixed reality device based on a first screw insertion pose, a first pose parameter, a second pose parameter, a third pose parameter, and a first surgical direction includes: Based on the first nail insertion pose, the first pose parameter, and the second pose parameter, determine the second nail insertion pose of the target nail insertion point in the body surface coordinate system; Based on the second nail insertion pose and the second pose parameters, the third nail insertion pose of the target nail insertion point in the second coordinate system of the mixed reality device is determined. Based on the third nail insertion pose, the nail insertion position and nail insertion direction are displayed in the second coordinate system of the mixed reality device.

[0102] In some embodiments, after determining the second nail insertion pose of the target nail insertion point in the body surface coordinate system based on the first nail insertion pose, the first pose parameter, and the second pose parameter, the method further includes: The second nail insertion pose is stored in the storage space of the mixed reality device so that when the third nail insertion pose is subsequently obtained, the third nail insertion pose of the target nail insertion point in the second coordinate system can be determined based on the second pose parameters and the second nail insertion pose read from the storage space.

[0103] In some embodiments, the first surgical direction is the direction of the second coordinate axis in the tool coordinate system.

[0104] In some embodiments, displaying the relative spatial relationship between the surgical instrument and the first screw insertion pose in a mixed reality device based on a first screw insertion pose, a first pose parameter, a second pose parameter, a third pose parameter, and a first surgical direction includes: Based on the first point of action and the third pose parameter of the surgical instrument in the tool coordinate system, determine the second point of action of the instrument in the second coordinate system of the mixed reality device; Based on the second coordinate axis and the third pose parameter, the first surgical direction is determined as the second surgical direction in the second coordinate system. Based on the position of the second point of action and the second surgical direction, the instrument point of action and the first surgical direction are displayed in the second coordinate system of the mixed reality device.

[0105] This application embodiment also provides a medical navigation system based on mixed reality technology. The medical navigation system includes a mixed reality device, an X-ray imaging device, and surgical instruments. The X-ray imaging device is used to perform anteroposterior and lateral fluoroscopy on the target bone region of the object to be examined, acquiring X-ray anteroposterior and lateral X-ray images of the target bone. Based on the X-ray anteroposterior and lateral X-ray images, three-dimensional reconstruction is performed to obtain a three-dimensional bone model of the target bone. Based on the three-dimensional bone model, a first nail insertion pose of the target nail insertion point in a first coordinate system of the X-ray imaging device is determined, and the first nail insertion pose is sent to the mixed reality device. The first coordinate system is generated based on a first marker set on the X-ray imaging device, and the first nail insertion pose includes the nail insertion position and direction of the target nail insertion point. The mixed reality device uses… The system acquires a first pose parameter of a first coordinate system relative to a second coordinate system of a mixed reality device; acquires a second pose parameter of a body surface coordinate system of the object to be tested relative to the second coordinate system, wherein the body surface coordinate system is generated based on a second marker set on the body surface of the object to be tested; acquires a third pose parameter of a surgical instrument's tool coordinate system relative to the second coordinate system, and determines a first surgical direction of the surgical instrument in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; and displays the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, so as to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

[0106] In some embodiments, the mixed reality device is configured to display a displacement deviation between the instrument point of action and the insertion position of a surgical instrument in the mixed reality space of the mixed reality device, and to display a first direction indicator line pointing from the instrument point of action to the insertion position in the mixed reality space to guide the instrument point of action to move to the insertion position.

[0107] In some embodiments, the mixed reality device is configured to, in the mixed reality space, display a first surgical direction via a second direction indicator line and a nail insertion direction via a third direction indicator line, and display the angular deviation between the second direction indicator line and the third direction indicator line, as well as the orientation adjustment indicator line of the surgical instrument, and guide the surgical instrument to rotate via the angular deviation and the orientation adjustment indicator line so that the second direction indicator line overlaps with the third direction indicator line.

[0108] In some embodiments, the mixed reality device is used to construct a target coordinate system in a three-dimensional skeletal model, wherein the origin of the target coordinate system is the nail insertion position of the target nail insertion point, the direction of the first coordinate axis in the target coordinate system is the nail insertion direction of the target nail insertion point, and the first coordinate axis is any coordinate axis in the target coordinate system; the pose of the target coordinate system in the first coordinate system is used as the first nail insertion pose.

[0109] In some embodiments, the mixed reality device is configured to determine a second nail insertion pose of the target nail insertion point in a body surface coordinate system based on a first nail insertion pose, a first pose parameter, and a second pose parameter; determine a third nail insertion pose of the target nail insertion point in a second coordinate system of the mixed reality device based on the second nail insertion pose and the second pose parameter; and display the nail insertion position and nail insertion direction in the second coordinate system of the mixed reality device based on the third nail insertion pose.

[0110] In some embodiments, the mixed reality device is configured to, after determining the second nail insertion pose of the target nail insertion point in the body surface coordinate system based on the first nail insertion pose, the first pose parameter, and the second pose parameter, store the second nail insertion pose in the storage space of the mixed reality device, so that when the third nail insertion pose is subsequently obtained, the third nail insertion pose of the target nail insertion point in the second coordinate system can be determined based on the second pose parameter and the second nail insertion pose read from the storage space.

[0111] In some embodiments, the first surgical direction is the direction of the second coordinate axis in the tool coordinate system.

[0112] In some embodiments, the mixed reality device is configured to determine a second point of action of the surgical instrument in a second coordinate system of the mixed reality device based on a first point of action position and a third pose parameter in a tool coordinate system; determine a second surgical direction of a first surgical direction in the second coordinate system based on a second coordinate axis and a third pose parameter; and display the instrument point of action and the first surgical direction in the second coordinate system of the mixed reality device based on the second point of action position and the second surgical direction.

[0113] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0114] To this end, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute any of the mixed reality-based medical navigation methods provided in embodiments of this application. The computer program can perform the steps of the mixed reality-based medical navigation method described above.

[0115] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0116] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0117] Since the computer program stored in the computer-readable storage medium can execute any of the medical navigation methods based on mixed reality technology provided in the embodiments of this application, it can achieve the beneficial effects that any of the medical navigation methods based on mixed reality technology provided in the embodiments of this application can achieve, as detailed in the preceding embodiments, and will not be repeated here.

[0118] According to one aspect of this application, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various optional implementations of the above embodiments.

[0119] In the above embodiments of the mixed reality device, the medical navigation system based on mixed reality technology, the computer-readable storage medium, and the computer program product, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the mixed reality device, the medical navigation system based on mixed reality technology, the computer-readable storage medium, and the computer program product described above can be referred to the description of the medical navigation method based on mixed reality technology in the above embodiments, and will not be repeated here.

[0120] The foregoing has provided a detailed description of a medical navigation method, navigation system, mixed reality device, computer-readable storage medium, and computer program product based on mixed reality technology, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A medical navigation method based on mixed reality technology, characterized in that, Applied to a medical navigation system, the medical navigation system including a mixed reality device, an X-ray imaging device, and surgical instruments, the method includes: The X-ray imaging device is used to perform anteroposterior and lateral fluoroscopy on the target bone region of the object to be detected, thereby obtaining X-ray anteroposterior and lateral X-ray images of the target bone. Based on the X-ray anterior and lateral images, a three-dimensional reconstruction is performed to obtain a three-dimensional skeleton model of the target bone. Based on the three-dimensional skeleton model, the first nail insertion point of the target bone in the first coordinate system of the X-ray imaging device is determined. The first coordinate system is generated based on a first marker set on the X-ray imaging device. The first nail insertion position includes the nail insertion position and nail insertion direction of the target nail insertion point. Obtain the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device; Obtain the second pose parameters of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on the second marker set on the body surface of the object to be detected; The tool coordinate system of the surgical instrument is obtained relative to the second coordinate system as a third pose parameter, and a first surgical direction of the surgical instrument is determined in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; Based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, the relative spatial relationship between the surgical instrument and the first nail insertion pose is displayed in the mixed reality device to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

2. The method as described in claim 1, characterized in that, Displaying the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device includes: The displacement deviation between the instrument's point of action and the nail insertion position is displayed in the mixed reality space of the mixed reality device, and a first direction indicator line pointing from the instrument's point of action to the nail insertion position is displayed in the mixed reality space to guide the instrument's point of action to move to the nail insertion position.

3. The method as described in claim 2, characterized in that, When the instrument's point of action moves to the nail insertion position, displaying the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device includes: In the mixed reality space, the first surgical direction is displayed by a second direction indicator line, and the screw insertion direction is displayed by a third direction indicator line. The angular deviation between the second direction indicator line and the third direction indicator line is also displayed, as well as the orientation adjustment indicator line of the surgical instrument. The surgical instrument is guided to rotate by the angular deviation and the orientation adjustment indicator line so that the second direction indicator line overlaps with the third direction indicator line.

4. The method as described in claim 1, characterized in that, The step of determining the first nail insertion point of the target bone in the first coordinate system of the X-ray imaging device based on the three-dimensional bone model includes: A target coordinate system is constructed in the three-dimensional skeleton model, wherein the origin of the target coordinate system is the nail insertion position of the target nail insertion point, the direction of the first coordinate axis in the target coordinate system is the nail insertion direction of the target nail insertion point, and the first axis can be any coordinate axis in the target coordinate system; The pose of the target coordinate system in the first coordinate system is taken as the first nail-feeding pose.

5. The method according to any one of claims 1-4, characterized in that, The step of displaying the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device based on the first nail insertion pose, the first pose parameters, the second pose parameters, the third pose parameters, and the first surgical direction includes: Based on the first nail insertion pose, the first pose parameters, and the second pose parameters, determine the second nail insertion pose of the target nail insertion point in the body surface coordinate system; Based on the second nail insertion pose and the second pose parameters, a third nail insertion pose of the target nail insertion point in the second coordinate system of the mixed reality device is determined, and based on the third nail insertion pose, the nail insertion position and the nail insertion direction are displayed in the second coordinate system of the mixed reality device.

6. The method as described in claim 5, characterized in that, After determining the second nail insertion point in the body surface coordinate system based on the first nail insertion pose, the first pose parameters, and the second pose parameters, the method further includes: The second nail insertion pose is stored in the storage space of the mixed reality device so that when the third nail insertion pose is subsequently obtained, the third nail insertion pose of the target nail insertion point in the second coordinate system can be determined based on the second pose parameters and the second nail insertion pose read from the storage space.

7. The method according to any one of claims 1-4, characterized in that, The first surgical direction is the direction of the second coordinate axis in the tool coordinate system.

8. The method as described in claim 7, characterized in that, The step of displaying the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device based on the first nail insertion pose, the first pose parameters, the second pose parameters, the third pose parameters, and the first surgical direction includes: Based on the first point of action of the surgical instrument in the tool coordinate system and the third pose parameter, the second point of action of the instrument in the second coordinate system of the mixed reality device is determined; Based on the second coordinate axis and the third pose parameter, the second surgical direction in the second coordinate system of the first surgical direction is determined, and the instrument's point of action and the first surgical direction are displayed in the second coordinate system of the mixed reality device according to the position of the second point of action and the second surgical direction.

9. A medical navigation method based on mixed reality technology, characterized in that, A mixed reality device applied to a medical navigation system, the medical navigation system further including an X-ray imaging device and surgical instruments, the method comprising: The X-ray imaging device receives a first nail insertion pose of the target nail insertion point in the first coordinate system of the X-ray imaging device. The first nail insertion pose is determined by the X-ray imaging device after performing three-dimensional reconstruction based on the X-ray anterior and lateral X-ray images of the target bone of the object to be detected. The first coordinate system is generated based on a first marker set on the X-ray imaging device. The first nail insertion pose includes the nail insertion position and nail insertion direction of the target nail insertion point. Obtain the first pose parameter of the first coordinate system relative to the second coordinate system of the mixed reality device; Obtain the second pose parameters of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on the second marker set on the body surface of the object to be detected; The tool coordinate system of the surgical instrument is obtained relative to the second coordinate system as a third pose parameter, and a first surgical direction of the surgical instrument is determined in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; Based on the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter, and the first surgical direction, the relative spatial relationship between the surgical instrument and the first nail insertion pose is displayed in the mixed reality device to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

10. A medical navigation system based on mixed reality technology, characterized in that, The medical navigation system includes a mixed reality device, an X-ray imaging device, and surgical instruments. The X-ray imaging device is used to perform anteroposterior and lateral fluoroscopy on the target bone region of the object to be examined, acquiring X-ray anteroposterior and lateral X-ray images of the target bone. Based on the X-ray anteroposterior and lateral X-ray images, a three-dimensional reconstruction is performed to obtain a three-dimensional bone model of the target bone. Based on the three-dimensional bone model, a first nail insertion pose of the target nail insertion point in a first coordinate system of the X-ray imaging device is determined, and the first nail insertion pose is sent to the mixed reality device. The first coordinate system is generated based on a first marker set on the X-ray imaging device, and the first nail insertion pose includes the nail insertion position and direction of the target nail insertion point. The mixed reality device is used to acquire the first coordinate system relative to the mixed reality device's first coordinate system. The first pose parameter of the two coordinate systems; obtaining the second pose parameter of the body surface coordinate system of the object to be detected relative to the second coordinate system, wherein the body surface coordinate system is generated based on a second marker set on the body surface of the object to be detected; obtaining the third pose parameter of the tool coordinate system of the surgical instrument relative to the second coordinate system, and determining the first surgical direction of the surgical instrument in the tool coordinate system, wherein the tool coordinate system is generated based on a third marker set on the surgical instrument, and the first surgical direction is used to indicate the operating direction of the surgical instrument; and displaying the relative spatial relationship between the surgical instrument and the first nail insertion pose in the mixed reality device according to the first nail insertion pose, the first pose parameter, the second pose parameter, the third pose parameter and the first surgical direction, so as to guide the surgical instrument to move to the nail insertion position and operate along the nail insertion direction.

11. The system as claimed in claim 10, characterized in that, The mixed reality device is configured to display the displacement deviation between the instrument action point of the surgical instrument and the nail insertion position in the mixed reality space of the mixed reality device, and to display a first direction indicator line pointing from the instrument action point to the nail insertion position in the mixed reality space, so as to guide the instrument action point to move to the nail insertion position.

12. The system as claimed in claim 11, characterized in that, The mixed reality device is configured to, when the instrument's point of action moves to the nail insertion position, display the first surgical direction via a second direction indicator line and the nail insertion direction via a third direction indicator line in the mixed reality space, and display the angular deviation between the second direction indicator line and the third direction indicator line, as well as the direction adjustment indicator line of the surgical instrument, and guide the surgical instrument to rotate via the angular deviation and the direction adjustment indicator line so that the second direction indicator line overlaps with the third direction indicator line.

Images