AR-assisted surgical positioning system and method based on virtual axis of x-ray beam

Abstract

The present application belongs to the field of surgical medical technology, and particularly relates to a surgical positioning system and method based on an X-ray beam virtual axis under AR assistance. The present application utilizes cooperation between a C-arm and AR technology, determines the position of a target by using an X-ray machine display screen, visualizes the central axis of an X-ray beam by using AR technology, and ingeniously generates a fixed virtual guide line outside the human body. The present application utilizes a display device to collect images of surgical instruments in real time, displays the surgical instruments and the virtual guide line in the display device at the same time, guides the surgeon during surgery, and effectively ensures that the surgeon can accurately place a screw into a locking hole of a intramedullary nail or determine the position of a foreign object. The method of the present application is simple and reliable, and has great application and conversion value in the field of orthopedic clinics.

Description

Technical Field

[0001] This invention belongs to the field of surgical medical technology, specifically relating to a surgical positioning system and method based on an X-ray beam virtual axis under AR assistance. Background Technology

[0002] Orthopedic surgeries often require X-ray fluoroscopy to locate metallic objects within the body, such as intramedullary nail insertion and removal of metallic foreign bodies. These surgeries rely on real-time fluoroscopic results for guidance. Taking the distal locking screw insertion step in intramedullary nailing as an example, current methods in China mainly involve traditional manual manipulation, implant-compatible mechanical aiming devices, core drills, and orthopedic robots for positioning. These methods have several problems: 1. Traditional manual nail placement (blind locking): Relies entirely on touch, resulting in inconsistent accuracy and frequent adjustments, increasing surgical time and radiation exposure. 2. Core drill: The principle involves inserting a fine drill from inside the intramedullary nail, penetrating from the inside out. However, the drill bit is thin, soft, and flexible; if it breaks, it can remain inside the body and hinder proper screw insertion. Furthermore, core drills are disposable, leading to high overall costs. 3. Locking with implant-compatible mechanical aiming devices: These have complex mechanical structures and numerous connection points, each with some degree of looseness, increasing the final error. Furthermore, the intramedullary nail can undergo minute and unpredictable deformation after being placed into the medullary cavity, which can also lead to aiming device deviation. This approach has a low success rate; 4. Orthopedic robots: High accuracy, but complex operation, long learning time, and significant intraoperative time consumption. In addition, the equipment is extremely expensive. These factors limit its widespread adoption; 5. Augmented / mixed reality technology: There are very few domestic applications and reports. The technologies used all use surgical tools or the human body as a reference system, and the algorithms and processes are complex, making it difficult to promote and apply on a large scale.

[0003] Orthopedic positioning methods are used not only for intramedullary nailing but also for removing pelvic canal screws, foreign bodies, and internal fixation devices. Given the limitations of current in vivo positioning methods, this application proposes an AR-assisted in vivo positioning method based on X-ray projection direction. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a surgical positioning system and method based on a virtual axis of an X-ray beam under AR assistance. This invention is the first to propose the principle of using the geometric axis of an X-ray beam to guide the surgical path. In practice, the virtual spatial position of the X-ray beam's central axis is derived from the geometry of the C-arm X-ray machine. AR-assisted technology is then used to visualize this axis, thereby guiding the surgical path. The method of this invention is simple and reliable, and can be widely applied in hospitals, possessing significant application value.

[0005] The technical problem to be solved by this invention is achieved by the following technical solution: a surgical positioning method based on an X-ray beam virtual axis under AR assistance, comprising the following steps: S1. Before the operation, a positioning camera and a display device are set up on site. Visual positioning marks are fixedly installed on the C-arm and the display device respectively. The positioning camera is used to realize the three-dimensional transformation of the space from the X-ray machine to the display device. S2. During the operation, the human body is kept in a fixed position on the operating table. By adjusting the C-arm, X-ray images are obtained. The target in the X-ray image is placed in the center of the X-ray machine screen, and the position of the C-arm is kept unchanged. S3. Using AR technology, visualize the virtual connection between the geometric center of the X-ray transmitter and receiver on the C-arm, and generate a virtual guide line with a fixed spatial position on the display device. S4. During the operation, real-time images of surgical instruments are acquired through a display device, and virtual guide lines are displayed synchronously with the surgical instrument images on the display device; S5. Using the image displayed on the device, guide the surgical instruments to operate on the target along the direction of the virtual guide line.

[0006] Preferably, in step S3 of this invention, after the virtual axis is generated, the C-arm is moved aside to create space for subsequent operations. Since its rotation trajectory is a concentric circle and the angle is known, it does not affect the function of its reference frame.

[0007] Preferably, the display device in this invention is a monitor with a camera. Since continuous movement while wearing AR glasses affects the accuracy of the virtual navigation path, this invention utilizes a fixed display, which effectively improves operational accuracy.

[0008] Preferably, in step S1 of this invention, the specific steps for using a positioning camera to achieve the three-dimensional transformation of the space from the X-ray machine to the display device are as follows: A visual positioning marker is fixedly installed on the C-arm to determine the relative pose Tc from the visual positioning marker to the X-ray machine: The calibration from the visual positioning marker to the C-arm requires placing a checkerboard calibration board in the scene so that both the positioning camera and the camera of the C-arm can observe the calibration board. At the same time, the positioning camera can also observe the visual positioning marker. This allows us to calculate the pose T0 of the visual positioning marker, the pose T1 of the calibration board, and the pose T2 of the C-arm relative to the calibration board. Thus, the pose from the visual positioning marker to the C-arm is Tc = T2·T1·inv(T0). Install visual positioning markers on the display device and define the relative pose Ts from the visual positioning markers to the display device: The calibration from the visual positioning marker to the display device requires placing a checkerboard calibration board in the scene so that both the positioning camera and the display device's camera can observe the calibration board. At the same time, the positioning camera can also observe the visual positioning marker. This allows us to calculate the pose T0 of the visual positioning marker, the pose T1 of the calibration board, and the pose T2 of the display device relative to the calibration board. Thus, the pose from the visual positioning marker to the display device is Ts = T2·T1·inv(T0). The positioning camera simultaneously observes the visual positioning marks on the C-arm and the display device, and uses the positioning camera to calculate the relative pose Tm between the C-arm and the display device. The three-dimensional transformation T from the X-ray machine to the display device space is expressed as: T = Ts·inv(Tc)·Tm. The expression T = Ts·inv(Tc)·Tm describes a complete three-dimensional transformation process from the model coordinate system to the X-ray machine display screen coordinate system. In computer graphics and 3D vision, this is often referred to as a specific form of MVP (Model-View-Projection) transformation.

[0009] Preferably, in step S1 of this invention, the visual positioning mark is a planar graphic with a specific pattern, and the positioning camera only needs to be an ordinary RGB or grayscale camera, which is low cost.

[0010] Preferably, the present invention further includes step S6: in step S2, the C-arm angle is adjusted again to obtain another X-ray image, the target in the X-ray image is placed in the center of the X-ray machine display screen, the pose of the C-arm is kept unchanged, and step S3 is repeated to generate another virtual guide line that intersects with the first virtual guide line to obtain the target guide point. Then, operations are performed on the target based on the target guidance point, which can be used to determine the location of foreign objects inside the human body.

[0011] In a preferred embodiment of the present invention, in step S1, the visual positioning marker only needs to be marked once before the operation.

[0012] Preferably, in step S2 of this invention, the center of the X-ray machine display screen is determined by the two diagonals of the X-ray machine display screen, which can be determined by drawing lines with a marker pen, which is convenient and effective; alternatively, a positioning plate identical to the X-ray machine display screen can be set, with a positioning hole in the center of the positioning plate for quick positioning.

[0013] Preferably, in step S1 of this invention, the positioning camera and the display device are respectively supported and fixed by a tripod, which facilitates surgical operation and ensures the safety and reliability of the surgical operation.

[0014] This invention also discloses an AR-assisted surgical positioning system based on an X-ray beam virtual axis, used to perform the aforementioned AR-assisted surgical positioning method based on an X-ray beam virtual axis, the system comprising: The coordinate transformation module is used to perform three-dimensional spatial transformation between X-ray machine imaging and display device imaging; The navigation path generation module is used to acquire X-ray images, place the target in the X-ray image in the center of the X-ray machine display screen, visualize the virtual connection between the geometric center of the X-ray transmitter and receiver on the C-arm, and generate a virtual guide line with a fixed spatial position on the display device. The navigation guidance module is used to acquire real-time images of surgical instruments during surgery, and to display virtual guide lines and surgical instrument images synchronously on a display device. The images on the display device guide the surgical instruments to operate on the target along the direction of the virtual guide lines.

[0015] Advantages of this invention: Improvement 1: This invention utilizes AR technology to visualize the central axis of the X-ray beam to represent the nail insertion direction, and uses AR technology to guide nail placement; this solves the problem of poor accuracy caused by manual nail placement, shortens treatment time, and reduces the harm of radiation exposure.

[0016] Improvement point two: Traditional core drilling requires inserting the drill bit into the body for operation, which can easily lead to breakage and residue. This technology, however, positions the drill bit externally, eliminating the need for any internal devices and thus avoiding the risk of drill bit breakage or residue.

[0017] Improvement point three: This technology utilizes AR virtual surgical operation path, avoiding mechanical errors caused by multiple connection points of the aiming device and deformation of the master nail.

[0018] Fourthly, compared with orthopedic robots, this technology is simpler, cheaper, and easier to operate while maintaining accuracy, which will inevitably increase its adoption rate.

[0019] Fifthly, addressing the complexity of current augmented / mixed reality algorithms and procedures, this technology utilizes a virtual X-ray derived from the C-arm's appearance as a reference frame, simplifying the principles and algorithms and making the process more efficient. An AR display can be used instead of AR glasses, improving accuracy. Furthermore, because the reference frame of this technology is a third-party object, it is unaffected by patient characteristics or implant features.

[0020] Compared with the prior art, the beneficial effects of the present invention are: This invention utilizes a C-arm, an X-ray machine display screen, and AR technology in conjunction. First, the X-ray machine display screen determines the target's location. Then, AR technology visualizes the central axis of the X-ray beam, cleverly generating a fixed virtual guideline outside the body. The display device simultaneously captures images of surgical instruments and displays both the instruments and the virtual guideline, guiding the surgeon during operation. This effectively ensures the surgeon can accurately insert screws into the locking holes of intramedullary nails or locate foreign objects within the body. This method is simple and reliable, suitable for widespread application in hospitals, and has significant application and commercialization value in orthopedic clinical practice. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the principle of an AR-assisted surgical positioning method based on an X-ray beam virtual axis. Figure 2 This is a schematic diagram showing how to place a target in an X-ray image at the center of an X-ray machine screen. Figure 3 A schematic diagram illustrating the system setup and the relative pose calculation principle from the X-ray machine to the display device; Figure 4 A schematic diagram illustrating the calibration principle of the display device and the visual positioning marker; Figure 5 This is a schematic diagram illustrating the principle of synchronizing virtual guide lines with surgical instrument images on a display device. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0023] Example 1 like Figures 1-5 As shown, a surgical positioning method based on an X-ray beam virtual axis under AR assistance includes the following steps: S1. Before the surgery, a positioning camera and a display device are set up on-site. Visual positioning markers are fixedly installed on the C-arm and the display device, respectively. The positioning camera is used to achieve three-dimensional transformation of the space from the X-ray machine to the display device. The positioning camera is in a fixed position. In this embodiment, the visual positioning markers are binary encoded.

[0024] S2. During the procedure, the patient remains in a fixed position on the operating table. X-ray images are acquired by adjusting the C-arm, and the target in the X-ray image is placed in the center of the X-ray machine display screen while maintaining the C-arm's position. In this embodiment, as... Figure 2 As shown, the target is the locking hole of the intramedullary nail, with the center line of the locking hole perpendicular to the X-ray machine display screen and located at the center of the X-ray machine display screen.

[0025] S3. Using AR technology, a virtual connection between the geometric centers of the X-ray transmitter and receiver on the C-arm is visualized, generating a virtual guide line with a fixed spatial position on the display device. Both AR technology and the generation of virtual guide lines are existing technologies and will not be elaborated upon here.

[0026] S4. During the operation, real-time images of surgical instruments are acquired through a display device, and virtual guide lines are displayed synchronously with the surgical instrument images on the display device.

[0027] Specifically, such as Figure 3 and Figure 5 As shown, the processor constructs the coordinate transformation matrix T=Ts·Tc from the X-ray machine to the display device based on the preset three-dimensional spatial transformation relationship. - ¹·Tm, and load the coordinate transformation matrix into the OpenGL graphics rendering unit.

[0028] The OpenGL graphics rendering unit spatially calibrates the geometry and axes of the X-ray machine's transmitter and receiver based on the coordinate transformation matrix, generates a rendered image containing virtual axes, and outputs the rendered image to the display device.

[0029] The camera captures real-time images of the surgical instruments and transmits these images to a display device. The display device then merges the virtual axis rendering image with the real-time image of the surgical instruments and displays them synchronously.

[0030] S5. Using the image displayed on the device, guide the surgical instruments to operate on the target along the direction of the virtual guide line.

[0031] In step S3, after the virtual axis is generated, the C-arm can be removed.

[0032] In this embodiment, the display device is a monitor. The operator sees a virtual reference line and a real electric drill on the monitor and keeps them collinear.

[0033] In step S1, the specific steps for using a positioning camera to achieve the three-dimensional transformation from the X-ray machine to the display device space are as follows: A visual positioning marker is fixedly installed on the C-arm to determine the relative pose Tc from the visual positioning marker to the X-ray machine: The calibration from the visual positioning marker to the C-arm requires placing a checkerboard calibration board in the scene so that both the positioning camera and the camera on the C-arm can observe the calibration board. At the same time, the positioning camera can also observe the visual positioning marker. This allows us to calculate the pose T0 of the visual positioning marker, the pose T1 of the calibration board, and the pose T2 of the C-arm relative to the calibration board. Thus, the pose from the visual positioning marker to the C-arm is Tc = T2·T1·inv(T0).

[0034] Install visual positioning markers on the display device and define the relative pose Ts from the visual positioning markers to the display device: like Figure 4 As shown, the calibration from the visual positioning marker to the display device requires placing a checkerboard calibration board in the scene so that both the positioning camera and the display device's camera can observe the calibration board. At the same time, the positioning camera can also observe the visual positioning marker. In this way, the pose T0 of the visual positioning marker, the pose T1 of the calibration board, and the pose T2 of the display device relative to the calibration board can be calculated. Thus, the pose from the visual positioning marker to the display device is Ts = T2·T1·inv(T0).

[0035] The positioning camera simultaneously observes the visual positioning marks on the C-arm and the display device, and uses the positioning camera to calculate the relative pose Tm between the C-arm and the display device.

[0036] The three-dimensional transformation T from the X-ray machine to the display device space is expressed as: T = Ts·inv(Tc)·Tm.

[0037] In step S1, the visual positioning markers only need to be marked once before the operation.

[0038] In step S2, the exact center of the X-ray machine display screen can be determined using the two diagonals of the X-ray machine display screen.

[0039] Using the method of this embodiment, precise nail placement can be achieved. By using a virtual guide line with a fixed spatial position generated in the display device as the nail insertion direction, it can be ensured that the locking nail is accurately implanted into the locking hole of the intramedullary nail.

[0040] To implement the AR-assisted surgical positioning method based on an X-ray beam virtual axis described in this embodiment, the present invention provides an AR-assisted surgical positioning system based on an X-ray beam virtual axis, the system comprising: The coordinate transformation module is used to perform three-dimensional spatial transformation between X-ray machine imaging and display device imaging.

[0041] The navigation path generation module is used to acquire X-ray images, place the target in the X-ray image at the center of the X-ray machine display screen, visualize the virtual connection between the geometric centers of the X-ray transmitter and receiver on the C-arm, and generate a virtual guide line with a fixed spatial position on the display device.

[0042] The navigation guidance module is used to acquire real-time images of surgical instruments during surgery, and to display virtual guide lines and surgical instrument images synchronously on a display device. The images on the display device guide the surgical instruments to operate on the target along the direction of the virtual guide lines.

[0043] Example 2 The difference from Example 1 is that this example is used to confirm the location of foreign objects inside the human body. The principle is to determine a point using two non-parallel straight lines on the same plane.

[0044] The AR-assisted surgical positioning method based on the virtual axis of the X-ray beam further includes step S6: In step S2, the C-arm angle is adjusted again to obtain another X-ray image. The target in the X-ray image is placed in the center of the X-ray machine display screen. The pose of the C-arm is kept unchanged. Step S3 is repeated to generate another virtual guide line that intersects with the first virtual guide line. The intersection of the two virtual guide lines obtains the target guide point (not shown in the figure).

[0045] Then, operations are performed on the target based on the target guidance point. In this embodiment, the target is a foreign object inside the body.

[0046] The above method can be used to remove foreign objects from the body. The X-ray machine can visualize the foreign object from two different angles. Based on the two positions of the X-ray machine, two virtual axes of X-rays are obtained, and the intersection of the two axes indicates the location of the foreign object in the body.

Claims

1. A surgical positioning method based on an X-ray beam virtual axis under AR assistance, characterized in that, Includes the following steps: S1. Before the operation, a positioning camera and a display device are set up on site. Visual positioning marks are fixedly installed on the C-arm and the display device respectively. The positioning camera is used to realize the three-dimensional transformation of the space from the X-ray machine to the display device. S2. During the operation, the human body is kept in a fixed position on the operating table. By adjusting the C-arm, X-ray images are obtained. The target in the X-ray image is placed in the center of the X-ray machine screen, and the position of the C-arm is kept unchanged. S3. Using AR technology, visualize the virtual connection between the geometric center of the X-ray transmitter and receiver on the C-arm, and generate a virtual guide line with a fixed spatial position on the display device. S4. During the operation, real-time images of surgical instruments are acquired through a display device, and virtual guide lines are displayed synchronously with the surgical instrument images on the display device; S5. Using the image displayed on the device, guide the surgical instruments to operate on the target along the direction of the virtual guide line.

2. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that: In step S3, after the virtual axis is generated, the C-arm can be removed.

3. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that: The display device is a monitor with a camera.

4. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that, In step S1, the specific steps for using a positioning camera to achieve the three-dimensional transformation from the X-ray machine to the display device space are as follows: A visual positioning marker is fixedly installed on the C-arm to calibrate the relative pose Tc from the visual positioning marker to the X-ray machine; Install visual positioning markers on the display device and calibrate the relative pose Ts from the visual positioning markers to the display device; The positioning camera simultaneously observes the visual positioning marks on the C-arm and the display device, and uses the positioning camera to calculate the relative pose Tm between the C-arm and the display device. The three-dimensional transformation T from the X-ray machine to the display device space is expressed as: T = Ts·inv(Tc)·Tm.

5. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that: In step S1, the visual positioning marker is a two-dimensional graphic with a specific pattern.

6. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that, The procedure also includes step S6: In step S2, the C-arm angle is adjusted again to obtain another X-ray image. The target in the X-ray image is placed in the center of the X-ray machine display screen. The pose of the C-arm is kept unchanged. Step S3 is repeated to generate another virtual guide line that intersects with the first virtual guide line. The two virtual guide lines intersect to obtain the target guide point. Then, the target is operated on based on the target guide point.

7. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that: In step S1, the visual positioning markers only need to be marked once before the operation.

8. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that: In step S2, the exact center of the X-ray machine display screen is determined using the two diagonals of the X-ray machine display screen.

9. The surgical positioning method based on an X-ray beam virtual axis under AR assistance according to claim 1, characterized in that: In step S1, both the positioning camera and the display device are fixed using a tripod.

10. A surgical positioning system based on an X-ray beam virtual axis under AR assistance, characterized in that, For performing the AR-assisted surgical positioning method based on an X-ray beam virtual axis as described in any one of claims 1-9, the system comprises: The coordinate transformation module is used to perform three-dimensional spatial transformation between X-ray machine imaging and display device imaging; The navigation path generation module is used to acquire X-ray images, place the target in the X-ray image in the center of the X-ray machine display screen, visualize the virtual connection between the geometric center of the X-ray transmitter and receiver on the C-arm, and generate a virtual guide line with a fixed spatial position on the display device. The navigation guidance module is used to acquire real-time images of surgical instruments during surgery, and to display virtual guide lines and surgical instrument images synchronously on a display device. The images on the display device guide the surgical instruments to operate on the target along the direction of the virtual guide lines.

Images