An intelligent intraoperative image navigation system and a navigation image projection method

Abstract

This invention discloses an intelligent intraoperative image navigation system, comprising an industrial control computer, an SCRA robotic arm, and a navigation image projection device. The SCRA robotic arm is fixed at its head to a mobile trolley, and the navigation image projection device is mounted at its tail. The control cable of the SCRA robotic arm is connected to the industrial control computer. The industrial control computer adjusts the position and image projection direction of the navigation image projection device by controlling the posture of the SCRA robotic arm. The navigation image projection device includes an image acquisition device, a projector, and a laser galvanometer connected to the industrial control computer. This invention also provides a navigation image projection method. By combining the image acquisition device, projector, and laser galvanometer, this invention enables precise positioning of the navigation image projection device and accurate calibration of the navigation image, greatly improving navigation accuracy. Furthermore, the use of the laser galvanometer to project surgical markers significantly enhances the visual effect of the navigation image, making the surgical location clearer and thus reducing the difficulty of the surgery.

Description

Technical Field

[0001] This invention relates to an intelligent intraoperative image navigation system and a navigation image projection method, which can accurately project navigation images onto the patient's body, belonging to the technical field of surgical navigation systems. Background Technology

[0002] Surgical navigation systems utilize virtual probes to locate and track surgical instruments, registering and updating preoperative or intraoperative virtual images with the real-world scene for real-time display to the surgeon. This primarily involves two key technologies: positioning and display. Positioning technologies can be categorized into mechanical, ultrasonic, electromagnetic, and optical positioning; display technologies can be divided into video display, stereoscopic display, and projection display. The core of positioning technology is high-accuracy navigation and tracking registration, obtaining the mapping relationship between the virtual image coordinate system and the real-world scene coordinate system, and then superimposing and displaying the virtual organ image with the real organ scene to the surgeon. Registration methods are divided into static registration and real-time registration. Static registration is based on various medical imaging technologies, selecting appropriate rigid or non-rigid registration models to reconstruct tissue and organ models for intraoperative display. When there are no abnormal deformations between the reconstructed tissue and organ model and the actual tissue and organ model, rigid registration can be performed to meet the high accuracy requirements of the surgical navigation system. Real-time registration adds an optimized registration model to static registration to improve real-time registration accuracy; that is, it achieves real-time navigation by displaying the scene through motion compensation. Motion compensation can be divided into device tracking and algorithm implementation. Device tracking is achieved through tracking with optical, electromagnetic, and infrared devices; algorithm implementation uses algorithms such as feature matching tracking and optical flow to achieve compensation.

[0003] Intraoperative projection navigation offers advantages such as convenient observation and intuitive results, reducing complex indirect human-computer interaction processes and increasing algorithm reliability. Compared to VR navigation, projection navigation solves the problem of spatial vertigo experienced by some doctors with VR, greatly improving surgical safety. Compared to traditional surgery, especially percutaneous minimally invasive surgery, projection navigation reduces fluoroscopy, allowing even novice surgeons to accurately complete procedures with minimal fluoroscopy exposure, making it highly valuable for educational guidance and clinical reference.

[0004] However, existing intraoperative projection navigation systems still suffer from drawbacks such as large projection position errors and low registration accuracy between virtual and real images. Moreover, navigation images often become blurry under bright operating lights or on open soft tissues, severely affecting visual quality.

[0005] In addition, existing intraoperative projection navigation systems are generally fixed in place during use. Since the operating room space is relatively small, placing the navigation system in the operating room will seriously affect the activity space of medical staff and cause many inconveniences to their normal activities. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an intelligent intraoperative image navigation system and method to improve navigation accuracy, enhance the visual effect of navigation images, and reduce the difficulty of surgery.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] An intelligent intraoperative image navigation system includes an industrial control computer, a SCARA robotic arm, and a navigation image projection device. The head end of the SCARA robotic arm is fixed on a mobile cart, and the tail end is equipped with the navigation image projection device. The control cable of the SCARA robotic arm is connected to the industrial control computer. The industrial control computer adjusts the position and image projection direction of the navigation image projection device by controlling the posture of the SCARA robotic arm. The navigation image projection device includes an image acquisition device, a projector, and a laser galvanometer connected to the industrial control computer and corresponding to the surgical site of the patient.

[0009] The aforementioned intelligent intraoperative image navigation system includes a mobile vehicle that is an intelligent obstacle avoidance vehicle. The intelligent obstacle avoidance vehicle comprises a ROS intelligent chassis, a vehicle body, and a radar. The vehicle body is fixed to the ROS intelligent chassis, and the top of the vehicle body is fixedly connected to the tip of the SCARA robotic arm. The radar is fixed to the middle of the upper surface of the intelligent chassis and scans objects around the ROS intelligent chassis through transparent windows around the bottom of the vehicle body. The drive unit of the ROS intelligent chassis and the radar are both connected to an industrial control computer.

[0010] In the aforementioned intelligent intraoperative image navigation system, the industrial control computer is mounted on the body of a mobile trolley.

[0011] The aforementioned intelligent intraoperative image navigation system includes a touch screen and a voice recognition module on the top of the mobile vehicle, which are connected to an industrial control computer.

[0012] In the aforementioned intelligent intraoperative image navigation system, the image acquisition device is a CMOS image sensor.

[0013] The radar in the aforementioned intelligent intraoperative image navigation system is a lidar.

[0014] A navigation image projection method using the above-mentioned intelligent intraoperative image navigation system, the method comprising the following steps:

[0015] ① Positioning of the navigation image projection device:

[0016] a. An industrial control computer controls a laser galvanometer to project gratings of different sizes onto the patient's body;

[0017] b. The image acquisition device captures images of gratings of different sizes projected onto the patient's body and transmits the resulting grating images to the industrial control computer;

[0018] c. The industrial control computer calculates the surface morphology of the patient's body using an optical flow algorithm based on the distortion of the raster image morphology acquired by the image acquisition device, and compares it with the pre-stored surface morphology model of that part of the patient's body to determine the bony recognition points on the skin surface, and then determines the relative position of the navigation image projection device and the patient's body.

[0019] ② Image calibration:

[0020] a. An industrial control computer controls a projector to project calibration images onto the patient's body;

[0021] b. The image acquisition device captures calibration images and sends the obtained images back to the industrial control computer;

[0022] c. The industrial control computer outputs a calibration signal to the projector based on the image fed back by the image acquisition device, calibrating the boundary of the calibration image to the set position;

[0023] ③ Projection of navigation images:

[0024] When navigation is required, the industrial control computer controls the projector to project the pre-stored navigation image onto the patient's body; at the same time, the industrial control computer controls the laser galvanometer to project the pre-stored surgical markings onto the patient's body.

[0025] In the above-mentioned navigation image projection method, during the projection process, the industrial control computer controls the radar to scan the objects around the ROS smart chassis in real time. When the radar detects that the distance between an object in a certain direction and the ROS smart chassis is less than a set value, the industrial control computer controls the ROS smart chassis to move in the direction where there are no objects. At the same time, it outputs control signals to the SCARA robotic arm, changing the shape of the SCARA robotic arm to keep the navigation image projection device stationary. When changing the shape of the SCARA robotic arm fails to keep the navigation image projection device stationary, causing the navigation image projection device to leave its original position, the industrial control computer controls the navigation image projection device to reposition itself, calibrate the image ratio, and project the navigation image.

[0026] In the above-described navigation image projection method, the navigation images projected by the projector include X-ray films, vascular images, and neural images.

[0027] In the above-mentioned navigation image projection method, the surgical markers projected by the laser galvanometer can be the pin placement location, incision location, or outline of the excision area, or the shape and location of the implant.

[0028] This invention combines an image acquisition device, a projector, and a laser galvanometer, enabling precise positioning of the navigation image projection device and accurate calibration of the navigation image. This not only greatly improves navigation accuracy but also enhances the visual effect of the navigation image by projecting surgical markers using a laser galvanometer, making the surgical location clearer and thus reducing the difficulty of the surgery. Attached Figure Description

[0029] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0030] Figure 1 This is a block diagram of the electrical principle of the intelligent intraoperative image navigation system of the present invention;

[0031] Figure 2 This is an outline diagram of an intelligent intraoperative image navigation system;

[0032] Figure 3 This is a structural diagram of an intelligent obstacle avoidance vehicle;

[0033] Figure 4 This is a schematic diagram of the positioning of the navigation image projection device;

[0034] Figure 5 This is a schematic diagram of image calibration;

[0035] Figure 6 This is a schematic diagram of surgical signage and navigation image projection.

[0036] The following are the labels in the diagram: 1. Mobile vehicle, 2. Navigation image projection device, 3. SCARA robotic arm, 4. Image acquisition device, 5. Projector, 6. Laser galvanometer, 7. Patient, 8. Laser grid, 9. Calibration image, 10. Surgical marker, 11. Navigation image, 12. Industrial control computer, 13. Radar, 14. Transparent window, 15. ROS intelligent chassis, 16. Vehicle body, 17. Touch screen. Detailed Implementation

[0037] This invention combines a projector, a CMOS image sensor, and a laser galvanometer. It uses an optical flow algorithm to create a curved topographic map, determines the specific size of the bone distribution based on bony protrusions or depressions on the skin surface, and finally projects a proportionally scaled image registered with the real-time human body onto the human body surface, greatly improving navigation accuracy.

[0038] In addition, this invention also solves the problem of blurred navigation images caused by the high brightness of the shadowless lamp used in the operation, making the surgical location clearer.

[0039] See Figures 1-3The intelligent intraoperative image navigation system provided by the present invention mainly includes a mobile vehicle 1, a navigation image projection device 2, a SCARA robotic arm 3, an industrial control computer 12, a touch screen 17, a voice recognition module and a radar 13. The navigation image projection device 2 includes an image acquisition device 4, a projector 5 and a laser galvanometer 6. The mobile vehicle 1 includes a ROS intelligent chassis 15 and a body 16.

[0040] The industrial control computer 12 is the control device of this navigation system. The touch screen 17 and voice recognition module are used for human-machine interaction. The SCARA robotic arm 3 supports the navigation image projection device 2 and can adjust and control the position and projection angle of the navigation image projection device 2. The ROS intelligent chassis 15 can automatically avoid medical personnel under the control of the industrial control computer 12, making it easier for medical personnel to move around in the operating room. The radar 13 is used to scan objects (mainly medical personnel) around the ROS intelligent chassis 15. In this invention, the radar 13 adopts a G4 lidar.

[0041] The vehicle body 16 is fixed on the ROS intelligent chassis 15. The top of the vehicle body 16 is fixed with a SCARA robotic arm 3, a touch screen 17 and a voice recognition module. An industrial control computer 12 is installed inside. Transparent windows 14 are provided around the bottom. The photoelectric radar scans the objects around the ROS intelligent chassis 15 through the transparent windows 14.

[0042] The method for projecting navigation images includes the following steps:

[0043] ① Positioning of the navigation image projection device (see...) Figure 4 ):

[0044] a. The industrial control computer 12 controls the laser galvanometer 6 to project gratings of different sizes onto the patient's body. For example, first project a grating with a grating spacing of 1 cm, and then project a grating with a grating spacing of 5 cm.

[0045] b. The image acquisition device 4 (i.e., CMOS image sensor) captures images of gratings of different sizes projected onto the patient's body and transmits the obtained grating images to the industrial control computer 12;

[0046] c. The industrial control computer 12 calculates the surface morphology of the patient's body using an optical flow algorithm based on the distortion of the raster image morphology acquired by the image acquisition device 4, and compares it with the pre-stored surface morphology model of that part of the patient's body to determine the bony identification points on the skin surface (such as the protruding area of ​​the lumbar spinous process and posterior iliac spine of the back spine), and then determines the relative position of the navigation image projection device 2 and the patient's body (with the bony identification points as reference points).

[0047] ② Image calibration (see Figure 5 ):

[0048] a. The industrial computer 12 controls the projector 5 to project a calibration image 9 onto the patient's body. The calibration image 9 is a high-brightness blank image of a certain shape (usually a rectangle) to facilitate the identification of its boundaries.

[0049] b. The image acquisition device 4 captures the calibration image 9 and sends the obtained image back to the industrial control computer 12;

[0050] c. The industrial control computer 12 outputs a calibration signal to the projector 5 based on the image fed back by the image acquisition device 4, and calibrates the boundary of the calibration image 9 to the set position. The calibration of the calibration image 9 includes the calibration of size, shape and rotation angle. The calibration method can be to change the position and rotation angle of the navigation image projection device 2, or it can be calibrated through a program.

[0051] ③ Projection of navigation images (see) Figure 6 ):

[0052] When navigation is required, the industrial control computer 12 controls the projector 5 to project the pre-stored navigation images onto the patient's body. The navigation images include X-rays for indicating bone location, vascular images for indicating blood vessel location, and nerve images for indicating nerve location. At the same time, the industrial control computer 12 controls the laser galvanometer 6 to project the pre-stored surgical markings onto the patient's body. The surgical markings can be solid dot-shaped pin placement locations, linear incision locations, or closed wireframe-shaped resection area outlines, or the shape and location of implants.

[0053] ④ During the projection of navigation images, the industrial control computer 12 controls the radar 13 to scan the objects around the ROS smart chassis 15 in real time. When the radar 13 detects that the distance between an object in a certain direction and the ROS smart chassis 15 is less than a set value, the industrial control computer 12 controls the ROS smart chassis 15 to move in the direction where there are no objects. At the same time, it outputs control signals to the SCARA robotic arm 3 to keep the navigation image projection device 2 stationary by changing the shape of the SCARA robotic arm 3. When changing the shape of the SCARA robotic arm 3 fails to keep the navigation image projection device 2 stationary, causing the navigation image projection device 2 to leave its original position, the industrial control computer 12 controls the navigation image projection device 2 to reposition the navigation image projection device, calibrate the image ratio, and project the navigation image.

[0054] Using the above technical solution, the projector 5 can project navigation images, including X-ray films, blood vessel shapes, and nerve shapes, onto the patient's body, helping doctors accurately determine the surgical location. When the shadowless lamp is turned on and the navigation images become blurry, the surgical location can be determined by using the surgical markers projected by the laser galvanometer 6. The images projected by the laser galvanometer 6 also maintain high clarity under the shadowless lamp, greatly reducing the difficulty of the surgery.

Claims

1. An intelligent intraoperative navigation image projection method, characterized in that, The method includes the following steps: ① Positioning of the navigation image projection device: a. The industrial control computer (12) controls the laser galvanometer (6) to project gratings of different sizes onto the patient's body; b. The image acquisition device (4) takes pictures of gratings of different sizes projected onto the patient's body and transmits the obtained grating images to the industrial control computer (12). c. The industrial control computer (12) calculates the surface morphology of the patient's body using an optical flow algorithm based on the distortion of the grating image morphology acquired by the image acquisition device (4), compares it with the pre-stored surface morphology model of the patient's body, determines the bony identification points on the skin surface, and then determines the relative position of the navigation image projection device (2) and the patient's body. ② Image calibration: a. The industrial computer (12) controls the projector (5) to project calibration images (9) onto the patient's body; b. The image acquisition device (4) takes a picture of the calibration image (9) and feeds the obtained image back to the industrial control computer (12); c. The industrial control computer (12) outputs a calibration signal to the projector (5) based on the image fed back by the image acquisition device (4), and calibrates the boundary of the calibration image (9) to the set position; ③ Projection of navigation images: During navigation, the industrial control computer (12) controls the projector (5) to project the pre-stored navigation image onto the patient's body; at the same time, the industrial control computer (12) controls the laser galvanometer (6) to project the pre-stored surgical markings onto the patient's body.

2. The intelligent intraoperative navigation image projection method according to claim 1, characterized in that, During the projection of the navigation image, the industrial control computer (12) controls the radar (13) to scan the objects around the ROS smart chassis (15) in real time. When the radar (13) detects that the distance between the object in a certain direction and the ROS smart chassis (15) is less than the set value, the industrial control computer (12) controls the ROS smart chassis (15) to move in the direction where there are no objects. At the same time, it outputs control signals to the SCARA robotic arm (3) to keep the navigation image projection device (2) stationary by changing the shape of the SCARA robotic arm (3). When the navigation image projection device (2) cannot be kept stationary by changing the shape of the SCARA robotic arm (3), and the navigation image projection device (2) leaves its original position, the industrial control computer (12) controls the navigation image projection device (2) to reposition the navigation image projection device, calibrate the image ratio, and project the navigation image.

3. The intelligent intraoperative navigation image projection method according to claim 2, characterized in that, The navigation images projected by the projector (5) include X-ray images, vascular images, and neural images.

4. The intelligent intraoperative navigation image projection method according to claim 3, characterized in that, The surgical markers projected by the laser galvanometer (6) are the nail placement location, incision location, outline of the excision area, or shape and location of the implant.

5. An image navigation system used in the intelligent intraoperative navigation image projection method according to any one of claims 1 to 4, characterized in that, The system includes an industrial control computer (12), a SCARA robotic arm (3), and a navigation image projection device (2). The head end of the SCARA robotic arm (3) is fixed on a mobile trolley (1), and the tail end is equipped with the navigation image projection device (2). The control cable of the SCARA robotic arm (3) is connected to the industrial control computer (12). The industrial control computer (12) adjusts the position and image projection direction of the navigation image projection device (2) by controlling the posture of the SCARA robotic arm (3). The navigation image projection device (2) includes an image acquisition device (4), a projector (5), and a laser galvanometer (6) that are connected to the industrial control computer (12) and correspond to the surgical site of the patient. The mobile vehicle (1) is an intelligent obstacle avoidance vehicle. The intelligent obstacle avoidance vehicle includes a ROS intelligent chassis (15), a body (16), and a radar (13). The body (16) is fixed on the ROS intelligent chassis (15). The top of the body (16) is fixedly connected to the head end of the SCARA robotic arm (3). The radar (13) is fixed on the middle of the upper surface of the intelligent chassis (15) and scans the objects around the ROS intelligent chassis (15) through the transparent windows (14) around the bottom of the body (16). The drive device of the ROS intelligent chassis (15) and the radar (13) are both connected to the industrial control computer (12).

6. The image navigation system according to claim 5, characterized in that, The industrial control computer (12) is installed on the body (16) of the mobile trolley (1).

7. The image navigation system according to claim 6, characterized in that, The mobile vehicle (1) has a touch screen (17) and a voice recognition module on the top of its body (16), and the touch screen (17) and the voice recognition module are connected to the industrial control computer (12).

8. The image navigation system according to claim 7, characterized in that, The image acquisition device (4) is a CMOS image sensor.

9. The image navigation system according to claim 8, characterized in that, The radar (13) is a lidar.

Images