Surgical navigation registration system and method fused with reality in large-scale dynamic environment

Abstract

The invention provides a surgical navigation registration system and method fused with reality in a large-scale dynamic environment, and the method comprises the steps: obtaining a focus of a patient and a point cloud of a surrounding part through a HoloLens depth camera, and carrying out the PCA point cloud preprocessing; according to patient focus CT/MRI, marking an image equal to the focus through 3Dmax, according to coordinate point sets of the reference points in a CT image coordinate system and a world coordinate system, solving a rotation and translation transformation matrix between the two coordinate systems so that the distance between homologous points of the two coordinate systems is made to be minimum; performing point cloud tracking based on a Hellman filter; and introducing the Unity 3D model into the cloud, and performing Azure remote rendering. According to the method, a solution is provided for the real-time virtual-real registration problem of the mixed reality surgical navigation system in an unconventional scene, the registration real-time performance is improved, the precision is greatly improved, the collection time is shortened, the storage space is saved, and the later point cloud preprocessing quantity and time are saved.

Description

Technical field [0001] The present invention belongs to virtual reality, human-machine interaction, data processing technology, involving a large-scale dynamic environment, fusion reality, navigation registration system and method. Background technique [0002] Mixing Reality (MR, MIXED REALITY) refers to a new visual environment generated by the consolidated reality and virtual world, coexists in new visual environments, and interacts in real time. Mixed realistic technology is a leading innovation technology. It is also based on applications in medical field. At present, there are also relevant reports and literatures that many scientific research workers are developing corresponding products and methods, and will mix real-in-law applications. . With the development of image-guided surgical navigation system (IGS), it fundamentally changed the traditional surgical model, achieving precision transfer of preoperative planning programs to clinical surgery, greatly improved surgery...

AI Technical Summary

Problems solved by technology

[0004] 1. In mixed reality, there will be real-time registration problems in a dynamic environment. For example, when performing surgical rescue on an airplane, the airplane will turbulence from time to time, which may cause changes in the position of patients or medical staff

So far, most of the applications of surgical navigation systems based on mixed reality are carried out in laboratories or hospitals, where patients are almost motionless during surgery, and there is no need to design registration algorithms. It is necessary to consider the frequent movement of the target, so the current virtual-real registration algorithm designed in the mixed reality surgical navigation system is not suitable for real-time registration in a dynamic environment

[0005] 2. When complex algorithms are added to the system or a large amount of data is processed, the hardware configuration based on HoloLens cannot meet the actual needs

Synchronously displaying the absolute motion of the surgical instrument and the patient in the virtual image (that is, the absolute tracking based on the world coordinate system registration) is more in line with objective reality, real and intuitive, but this method needs real-time monitoring of the patient's STL format during the navigation process. The 3D model (generally about tens of megabytes) is calculated for moving, rotating and displaying, which inevitably has the disadvantages of large memory usage, easy to cause image display lag, and even crash and paralysis of the navigation system

(2) The traditional method of viewing 3D content on mobile devices needs to compress the model and remove polygons

This will require developers to simplify the model to the point where it can run on low-performance GPU hardware, but the result may be that important details needed for critical business and design decisions are lost

Images