Virtual navigation system based on mixed reality technology and use method thereof

Abstract

The invention discloses a virtual navigation system based on a mixed reality technology and a use method thereof, and relates to the technical field of medical equipment, the virtual navigation system comprises an intraoperative perspective positioning module, a spatial positioning and registration module, an optical reflection navigation module and a mixed reality display module, and the modules are connected through electric signals; the intraoperative perspective positioning module comprises a positioning plate and intraoperative perspective equipment. According to the method, the virtual image planned before the operation and the actual operation scene are overlapped in real time, a visual and accurate operation navigation view is provided for an operator, it is ensured that the virtual model is accurately aligned with the real operation area, the operator is allowed to check and adjust the position and state of the operation instrument at any time in the operation process, and in addition, the operation efficiency is improved. Through the high-precision optical reflection reference frame and the mixed reality display device, an operator can clearly see the relation between surgical instruments and surrounding key structures, so that accidental injury is avoided, and the precision and safety of an operation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and particularly relates to a virtual navigation system based on mixed reality technology and a method for using the same. Background Art

[0002] Currently, the orthopedic surgery navigation technology has made remarkable progress in accurate surgical positioning and reducing surgical trauma. However, existing systems usually rely on fluoroscopic images and invasive optical reference frames for intraoperative navigation, resulting in increased intraoperative radiation exposure and trauma. In addition, the visual feedback of traditional surgical navigation is relatively limited, mainly through screen display, which causes the surgeon to switch back and forth between real operation and screen display during intraoperative operation, and it is difficult to achieve real-time monitoring of the execution system operation progress under the premise of non-invasiveness, increasing the surgical difficulty.

[0003] In recent years, although the accuracy and stability of navigation surgery technology have been improved, in complex orthopedic surgery scenarios, how to further improve surgical accuracy, reduce intraoperative radiation exposure and patient trauma, and provide more intuitive visual feedback for surgeons is still a major challenge in the current technology. In addition, there is a problem of low accuracy in using mixed reality technology for navigation surgery. Therefore, the present invention proposes a virtual navigation system based on mixed reality technology, which combines non-invasive registration and mixed reality display technology, and realizes visualization and real-time monitoring of the surgical operation process by superimposing virtual images on the real scene, further optimizing intraoperative operation. Summary of the Invention

[0004] The purpose of the present invention is to provide a virtual navigation system based on mixed reality technology and a method for using the same to solve the problems raised in the above background art.

[0005] To solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0006] In a first aspect, a virtual navigation system based on mixed reality technology, the virtual navigation system includes an intraoperative fluoroscopic positioning module, a spatial positioning and registration module, an optical reflection navigation module, and a mixed reality display module, wherein, the modules are electrically connected to each other;

[0007] The intraoperative fluoroscopic positioning module includes a positioning plate and an intraoperative fluoroscopic device, and is used for collecting two-dimensional fluoroscopic images of the surgical area with the positioning plate, obtaining the immediate position of the surgical area for positioning registration, realizing accurate positioning of the surgical area, and transmitting it to the spatial positioning and registration module;

[0008] The spatial positioning and registration module processes preoperative CT / MR data using a high-performance workstation, performs three-dimensional reconstruction, integrates soft tissue and bone tissue information, establishes a patient coordinate system, a surgical coordinate system, a surgeon coordinate system, and an instrument coordinate system, realizes the unification and registration between coordinate systems, locates the surgical area, forms a spatial mapping with the preoperative image planning, and provides accurate spatial information for virtual navigation;

[0009] The optical reflection navigation module includes an optical reflection reference frame, which is used to connect surgical instruments and the intraoperative fluoroscopy imaging module, realizes the spatial positioning of the mixed reality device under the premise of non-invasiveness and the real-time monitoring of instrument operation. Through the rigid connection of the optical reflection reference frame, a fixed spatial position relationship with the carrier is achieved, providing accurate position information for the mixed reality display module and realizing non-invasive virtual navigation;

[0010] The mixed reality display module includes a high-precision mixed reality display device, which has high-precision optical recognition ability and spatial position calculation ability, recognizes the optical reflection reference frame of the optical reflection navigation module, is used to obtain the spatial position information of the surgical area, preoperative planning, and the real-time position of the instrument, gives visual feedback to the surgeon, realizes the full visualization of the surgical operation process, and improves the surgical accuracy.

[0011] A further improvement of the technical solution of the present invention lies in that in the intraoperative fluoroscopy positioning module, the process of accurately positioning the surgical area is as follows:

[0012] Before the operation, the positioning plate is accurately installed on the surgical area. The positioning plate is used to provide a stable reference plane and contains marking points for registration. Then, the intraoperative fluoroscopy device is connected to the relevant systems in the operating room to ensure its good working state. Among them, the intraoperative fluoroscopy device is a C-arm X-ray machine or a CBCT machine;

[0013] Under sterile conditions, the intraoperative fluoroscopy device is used to perform fluoroscopy imaging on the surgical area with the positioning plate, avoiding the contamination of the surgical area, and obtaining the two-dimensional fluoroscopy image of the surgical area from the intraoperative fluoroscopy device. The two-dimensional fluoroscopy image of the surgical area is used for subsequent registration and positioning processes;

[0014] The two-dimensional fluoroscopy image obtained during the operation is registered with the preoperative three-dimensional images (CT, MRI), the final registration accuracy value is analyzed, so that the actual position of the surgical area corresponds to the position planned before the operation, and through image registration technology, the instant position information of the surgical area is calculated;

[0015] The calculated position information is transmitted to the spatial positioning and registration module. After receiving the position information, the spatial positioning and registration module updates the surgical navigation system and displays the position and status of the surgical area to the doctor in real time during the operation.

[0016] A further improvement of the technical solution of the present invention lies in that: the calculation formula of the final registration accuracy value is as follows:

[0017]

[0018] where P is the final registration accuracy value, representing the error magnitude during the registration process. The smaller the value, the higher the registration accuracy, that is, the smaller the image registration error between pre-operative and intra-operative images. A i is the position coordinate of the i-th feature point in the pre-operative three-dimensional image, and B i is the position coordinate of the i-th corresponding feature point in the intra-operative fluoroscopic image, and C i is the weight of the i-th feature point. Different weight values are assigned according to the importance or reliability of the feature point. The larger the weight, the more important the feature point is in the registration process. n is the total number of feature points considered in the registration;

[0019] The calculation formula of the real-time position information of the surgical area is as follows:

[0020]

[0021] where X is the real-time position information of the surgical area, representing the position deviation of the surgical area during the operation relative to the pre-operative plan. P is the final registration accuracy value, representing the error magnitude during the registration process. P 0 is a preset registration accuracy threshold. When P is less than P 0 , it is considered that the registration accuracy meets the requirements. k is the coefficient of the exponential function, used to adjust the influence degree of P on X. A i is the position coordinate of the i-th feature point in the pre-operative three-dimensional image, and B i is the position coordinate of the i-th corresponding feature point in the intra-operative fluoroscopic image, and C i is the weight of the i-th feature point.

[0022] A further improvement of the technical solution of the present invention lies in that: in the spatial positioning and registration module, the process of positioning the surgical area and forming a spatial mapping with the pre-operative image plan is as follows:

[0023] Obtain the pre-operative image data of the patient from the CT / MRI device, import the patient's pre-operative CT / MR data into a high-performance workstation, and use the high-performance workstation to process the image data, including denoising and enhancing contrast, etc., to improve the image quality;

[0024] Use the workstation to perform three-dimensional reconstruction on the imported data, integrate the soft tissue and bone tissue information to form a three-dimensional model of the patient. Among them, the model contains detailed information of the soft tissue and bone tissue;

[0025] A patient coordinate system is established with a certain part of the patient's body as a reference. The patient coordinate system is a Cartesian coordinate system with a certain point on the patient's body as the origin and three mutually perpendicular axes (left - right, up - down, front - back). According to the position and direction of the surgical area, a surgical coordinate system is established to describe the position and direction changes during the surgical process. Based on the perspective and position of the surgeon, a surgeon coordinate system is established, which helps the surgeon better understand and operate during the surgical process. An instrument coordinate system is established with the surgical instrument as a reference to describe the position and posture of the instrument during the surgical process;

[0026] The patient coordinate system, surgical coordinate system, surgeon coordinate system, and instrument coordinate system are transformed. The transformation matrices between the surgical coordinate system, surgeon coordinate system, or instrument coordinate system and the patient coordinate system are analyzed to achieve the unification and registration among the patient coordinate system, surgical coordinate system, surgeon coordinate system, and instrument coordinate system, ensuring the consistency and accuracy among different coordinate systems. Furthermore, the surgical area is located and a spatial mapping is formed with the preoperative image planning to provide accurate spatial information for virtual navigation.

[0027] A further improvement of the technical solution of the present invention lies in that: the expression of the transformation matrix is:

[0028]

[0029] where T is the transformation matrix used to transform a point in one coordinate system to another coordinate system, p j is the j - th feature point in the patient coordinate system, q j is the j - th corresponding feature point in the surgical coordinate system, surgeon coordinate system, or instrument coordinate system, w j is the weight of the j - th feature point in the transformation matrix, and different weight values are assigned according to the importance or reliability of the feature points. n is the total number of feature points considered in the registration.

[0030] A further improvement of the technical solution of the present invention lies in that: in the optical reflection navigation module, the process of realizing non - invasive virtual navigation is as follows:

[0031] The optical reflection reference frame and the surgical instrument are fixedly connected through a connecting device, and the intraoperative fluoroscopy image module is calibrated with the optical reflection reference frame to ensure the accuracy and stability of the fluoroscopy image. Then, the preoperative image data of the patient is imported into the navigation system. The connecting device includes a mechanical clamp, a magnetic adsorption device, etc.;

[0032] During the operation, when the surgical instrument moves within the surgical area, the optical reflection reference frame reflects specific light (laser or LED light), and the intraoperative fluoroscopy imaging module captures the reflected light, real-time tracking the position and movement of the optical reflection reference frame, monitoring the real-time position and attitude changes of the surgical instrument, and converting them into digital signals, which are transmitted to the virtual navigation system. Among them, the digital signals contain the precise position and attitude information of the surgical instrument in the three-dimensional space;

[0033] Based on the received digital signals, the virtual navigation system analyzes the position vectors of the surgical instrument in the three-dimensional space, calculates the position and direction of the surgical instrument in the three-dimensional space. The calculation results are used to guide the surgical operation, improve the accuracy and safety of the surgery, and update the navigation information of the virtual navigation system according to the calculation results, including the position, attitude of the surgical instrument and its relationship with the surgical area. Furthermore, the updated navigation information is real-time fed back to the mixed reality display module.

[0034] A further improvement of the technical solution of the present invention lies in that: the expression of the position vector of the surgical instrument in the three-dimensional space is:

[0035]

[0036] where PW is the position vector of the surgical instrument in the three-dimensional space, p is the target position vector of the surgical instrument in the preoperative plan, q is the actual position vector of the surgical instrument during the operation, ||p - q|| is the Euclidean distance between the preoperative planned position and the actual position during the operation, d 0 is the preset distance standard value, representing the maximum acceptable deviation between the preoperative planned position and the actual position during the operation, is the coefficient of the exponential function, used to adjust the influence degree of the distance deviation on the position vector, and α is the adjustment coefficient, used to control the adjustment amplitude of the position vector of the surgical instrument.

[0037] A further improvement of the technical solution of the present invention lies in that: in the mixed reality display module, the process of giving the surgeon visual feedback is:

[0038] Through the optical reflection reference frame, the mixed reality display device real-time obtains the spatial position information of the surgical area, and superimposes the virtual image of the preoperative plan on the actual surgical scene to generate a mixed reality view, ensuring the precise alignment of the virtual model and the real surgical area, and realizing the whole-process visualization of the surgical operation process;

[0039] The surgeon wears the mixed reality display device, enabling the surgeon to observe the picture combining the real surgical area and the virtual model through the mixed reality display device;

[0040] During the operation, the mixed reality display device real-time updates the position and status of the surgical instrument, and superimposes and displays them with the virtual model, giving the surgeon intuitive visual feedback;

[0041] The operator adjusts the position and operation of the surgical instrument according to the visual feedback provided by the mixed reality display module to ensure that the surgery is precisely performed in accordance with the preoperative plan. The mixed reality display device is configured with functions of magnifying, shrinking, and rotating the surgical area to assist the operator in observing and understanding the structure and relationship of the surgical area, and records the navigation data during the surgery and the operation log of the mixed reality display module for postoperative analysis and improvement of future surgeries.

[0042] In a second aspect, a method for using a virtual navigation system based on mixed reality technology, which is implemented based on a virtual navigation system based on mixed reality technology, includes the following steps:

[0043] Step 1: Import the patient's preoperative CT / MR data into the workstation for three-dimensional modeling and planning, and import it into the workstation and the mixed reality glasses.

[0044] Step 2: After the patient's preoperative preparation is completed, install the fluoroscopy positioning module, use the C-arm to obtain two-dimensional fluoroscopy images of the surgical area, input them into the workstation, and register them with the preoperative three-dimensional imaging data and planning information.

[0045] Step 3: Connect the fluoroscopy positioning module to the optical reflection navigation module. After the mixed reality glasses optically capture its position information, unify it with the patient's coordinate system. After registration, present the preoperative CT / MR images and surgical planning information 1:1 on the patient's surgical area.

[0046] Step 4: During the operation, connect the optical reflection navigation module to the instrument, and the mixed reality glasses capture and monitor the operation in real time throughout the process.

[0047] Due to the adoption of the above technical solution, the technical progress achieved by the present invention compared with the prior art is:

[0048] 1. The present invention provides a virtual navigation system based on mixed reality technology and its usage method. By superimposing the virtual image of the preoperative plan on the actual surgical scene in real time, it provides an intuitive and accurate surgical navigation view for the operator, not only ensuring the precise alignment of the virtual model with the real surgical area, but also allowing the operator to view and adjust the position and state of the surgical instrument at any time during the surgery. In addition, through the high-precision optical reflection reference frame and the mixed reality display device, the operator can clearly see the relationship between the surgical instrument and the surrounding key structures, thereby avoiding accidental injury and improving the accuracy and safety of the surgery.

[0049] 2. The present invention provides a virtual navigation system based on mixed reality technology and its usage method. By providing intuitive visual feedback and auxiliary observation functions, it greatly improves the operation efficiency and surgical experience of surgeons. During the surgical process, surgeons can directly view the position and status of surgical instruments, as well as the structure and relationship of the surgical area, through the mixed reality display device without frequently switching perspectives or referring to other imaging data, which not only reduces the cognitive burden and operation time of surgeons, but also improves the fluency and coherence of the surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0051] Figure 1 Schematic diagram of the virtual navigation system and its usage method of the present invention;

[0052] Figure 2 Frame diagram of the virtual navigation system and its usage method of the present invention;

[0053] Figure 3 Flow chart of the virtual navigation system and its usage method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] Embodiment 1, as Figure 1 、 Figure 2 shown, the present invention provides a virtual navigation system based on mixed reality technology. The virtual navigation system includes an intraoperative fluoroscopy positioning module, a spatial positioning and registration module, an optical reflection navigation module, and a mixed reality display module. Among them, the modules are electrically connected to each other;

[0056] Intraoperative fluoroscopy positioning module, including a positioning plate and an intraoperative fluoroscopy device, is used to acquire two-dimensional fluoroscopy images of the surgical area with the positioning plate, obtain the immediate position of the surgical area for positioning and registration, achieve precise positioning of the surgical area, and convey it to the spatial positioning and registration module. Before the operation, the positioning plate is precisely installed on the surgical area. The positioning plate is used to provide a stable reference plane and contains marking points for registration. Then, the intraoperative fluoroscopy device is connected to the relevant systems in the operating room to ensure its good working condition. Among them, the intraoperative fluoroscopy device is a C-arm X-ray machine or a CBCT machine. Under aseptic conditions, the intraoperative fluoroscopy device is used to perform fluoroscopy imaging on the surgical area with the positioning plate to avoid contamination of the surgical area, and obtain two-dimensional fluoroscopy images of the surgical area from the intraoperative fluoroscopy device. The two-dimensional fluoroscopy images of the surgical area are used for subsequent registration and positioning processes. The two-dimensional fluoroscopy images obtained during the operation are registered with the preoperative three-dimensional images (CT, MRI), and the final registration accuracy value is analyzed to make the actual position of the surgical area correspond to the position planned before the operation. Through image registration technology, the immediate position information of the surgical area is calculated, and the calculated position information is transmitted to the spatial positioning and registration module. After receiving the position information, the spatial positioning and registration module updates the surgical navigation system and displays the position and status of the surgical area to the doctor in real time during the operation;

[0057] Further, the calculation formula for the final registration accuracy value is:

[0058]

[0059] Where P is the final registration accuracy value, representing the error magnitude in the registration process. The smaller the value, the higher the registration accuracy, that is, the smaller the image registration error between the preoperative and intraoperative images. A i is the position coordinate of the i-th feature point in the preoperative three-dimensional image, and B i is the position coordinate of the i-th corresponding feature point in the intraoperative fluoroscopy image, and C i is the weight of the i-th feature point. Different weight values are assigned according to the importance or reliability of the feature point. The larger the weight, the more important the feature point is in the registration process. n is the total number of feature points considered in the registration;

[0060] The calculation formula for the immediate position information of the surgical area is:

[0061]

[0062] Where X is the immediate position information of the surgical area, representing the position deviation of the surgical area during the operation relative to the preoperative plan. P is the final registration accuracy value, representing the error magnitude in the registration process. P 0 is the preset registration accuracy threshold. When P is less than P 0When it is considered that the registration accuracy meets the requirements, k is the coefficient of the exponential function, which is used to adjust the influence degree of P on X, and A i is the position coordinate of the i-th feature point in the preoperative three-dimensional image, and B i is the position coordinate of the i-th corresponding feature point in the intraoperative fluoroscopic image, and C i is the weight of the i-th feature point, and different weight values are assigned according to the importance or reliability of the feature point. is used to adjust the influence of P on X. When P is less than P 0 , the value of the exponential function is greater than 1, otherwise it is less than 1. Ensure that the value of X does not exceed 1. When P is less than P 0 , the value of the minimum function is 1, otherwise it is Perform a weighted sum of the position differences of all feature points to obtain the sum of squares of the total registration error. Take the square root of the summation result to obtain the final registration error;

[0063] Spatial positioning and registration module: Use a high-performance workstation to process preoperative CT / MR data, perform three-dimensional reconstruction and integrate soft tissue and bone tissue information, establish a patient coordinate system, a surgical coordinate system, a surgeon coordinate system, and an instrument coordinate system, realize the unification and registration between coordinate systems, locate the surgical area and form a spatial mapping with the preoperative image planning, provide accurate spatial information for virtual navigation. Obtain the patient's preoperative image data from CT / MRI equipment, import the patient's preoperative CT / MR data into a high-performance workstation, and use the high-performance workstation to process the image data, including denoising and enhancing contrast, etc., to improve the image quality. Use the workstation to perform three-dimensional reconstruction on the imported data, integrate soft tissue and bone tissue information, and form a three-dimensional model of the patient. Among them, the model contains detailed information of soft tissue and bone tissue. Establish a patient coordinate system with a certain part of the patient's body as a reference. The patient coordinate system is a Cartesian coordinate system with a certain point of the patient's body as the origin and three mutually perpendicular axes (left-right, up-down, front-back). According to the position and direction of the surgical area, establish a surgical coordinate system to describe the position and direction changes during the operation. Based on the perspective and position of the surgeon, establish a surgeon coordinate system, which helps the surgeon better understand and operate during the operation. Establish an instrument coordinate system with the surgical instrument as a reference to describe the position and posture of the instrument during the operation. Convert the patient coordinate system, surgical coordinate system, surgeon coordinate system, and instrument coordinate system, analyze the transformation matrix between the surgical coordinate system, surgeon coordinate system, or instrument coordinate system and the patient coordinate system, realize the unification and registration between the patient coordinate system, surgical coordinate system, surgeon coordinate system, and instrument coordinate system, ensure the consistency and accuracy between different coordinate systems, and then locate the surgical area and form a spatial mapping with the preoperative image planning, provide accurate spatial information for virtual navigation;

[0064] Furthermore, the expression of the transformation matrix is as follows:

[0065]

[0066] where T is the transformation matrix used to transform points in one coordinate system to another coordinate system, p j is the j-th feature point in the patient coordinate system, q j is the j-th corresponding feature point in the surgical coordinate system, the surgeon coordinate system or the instrument coordinate system, w j is the weight of the j-th feature point in the transformation matrix, and different weight values are assigned according to the importance or reliability of the feature points. n is the total number of feature points considered in the registration. ||T(p j ) - q j || 2 represents the square of the Euclidean distance between the transformed point T(p j ) and the corresponding point q j and is used to measure the registration error. The objective of the formula is to minimize the weighted sum of the registration errors of all feature points, that is, to find an optimal transformation matrix T such that is minimized;

[0067] The optical reflection navigation module includes an optical reflection reference frame for connecting the surgical instrument and the intraoperative fluoroscopy imaging module, realizing the spatial positioning of the mixed reality device and the real-time monitoring of the instrument operation under the premise of non-invasiveness. Through the rigid connection of the optical reflection reference frame, the fixed spatial position relationship with the carrier is realized, providing accurate position information for the mixed reality display module and realizing non-invasive virtual navigation;

[0068] The mixed reality display module includes a high-precision mixed reality display device with high-precision optical recognition ability and spatial position calculation ability, which recognizes the optical reflection reference frame of the optical reflection navigation module, is used to obtain the spatial position information of the surgical area, the preoperative plan and the real-time position of the instrument, gives visual feedback to the surgeon, realizes the whole-process visualization of the surgical operation process, and improves the surgical accuracy.

[0069] Embodiment 2, as Figure 1 、 Figure 2 shown, based on Embodiment 1, the present invention provides a technical solution: Preferably, in the optical reflection navigation module, the process of realizing non-invasive virtual navigation is as follows:

[0070] Fix the optical reflection reference frame and the surgical instrument through a connecting device, and calibrate the intraoperative fluoroscopy imaging module with the optical reflection reference frame to ensure the accuracy and stability of the fluoroscopy image. Then, import the preoperative image data of the patient into the navigation system. The connecting device includes a mechanical clamp, a magnetic adsorption device, etc. During the operation, when the surgical instrument moves within the surgical area, the optical reflection reference frame reflects specific light (laser or LED light), and the intraoperative fluoroscopy imaging module captures the reflected light to track the position and movement of the optical reflection reference frame in real time, monitor the real-time position and attitude changes of the surgical instrument, and convert them into digital signals, which are transmitted to the virtual navigation system. The digital signals contain the accurate position and attitude information of the surgical instrument in the three-dimensional space. The virtual navigation system analyzes the position vector of the surgical instrument in the three-dimensional space according to the received digital signals, calculates the position and direction of the surgical instrument in the three-dimensional space, and the calculation results are used to guide the surgical operation to improve the accuracy and safety of the operation. The navigation information of the virtual navigation system, including the position, attitude of the surgical instrument and its relationship with the surgical area, is updated according to the calculation results, and then the updated navigation information is fed back to the mixed reality display module in real time;

[0071] Further, the expression of the position vector of the surgical instrument in the three-dimensional space is:

[0072]

[0073] where PW is the position vector of the surgical instrument in the three-dimensional space, p is the target position vector of the surgical instrument in the preoperative plan, q is the actual position vector of the surgical instrument during the operation, ||p - q|| is the Euclidean distance between the preoperative planned position and the actual intraoperative position, d 0 is the preset distance standard value, representing the maximum acceptable deviation between the preoperative planned position and the actual intraoperative position, is the coefficient of the exponential function, used to adjust the influence degree of the distance deviation on the position vector, α is the adjustment coefficient, used to control the adjustment amplitude of the position vector of the surgical instrument, the value range of ||p - q|| is non-negative real numbers, and the smaller the value, the closer the preoperative planned position is to the actual intraoperative position, The value range of is positive real numbers, The larger the value, the more sensitive the influence of the distance deviation on the position vector. The value range of is non-negative real numbers. The larger the α value, the greater the adjustment amplitude of the position vector. The value range of PW is a real vector in the three-dimensional space, representing the accurate position of the surgical instrument in the three-dimensional space, is used to adjust the influence of the distance deviation between the preoperative planned position and the actual intraoperative position on the position vector. When ||p - q|| is less than d 0 the value of the exponential function is greater than 1, otherwise less than 1, Ensure that the adjustment amplitude of the position vector is between 0 and 1. When ||p - q|| is less than d 0 the adjustment amplitude is close to 1, otherwise it is close to 0. represents the direction vector from the preoperative planning position to the intraoperative actual position, and its length is 1. Adjust the preoperative planning position vector p according to the adjustment coefficient α and the direction vector. PW is the final position vector calculated based on the adjustment amplitude and the adjusted preoperative planning position vector;

[0074] In the mixed reality display module, the process of giving the surgeon visual feedback is as follows:

[0075] Through the optical reflection reference frame, the mixed reality display device can obtain the spatial position information of the surgical area in real time, and superimpose the preoperative planned virtual image on the actual surgical scene to generate a mixed reality view, ensuring the precise alignment of the virtual model with the real surgical area and realizing the whole-process visualization of the surgical operation process. The surgeon wears the mixed reality display device, so that the surgeon can observe the picture combining the real surgical area and the virtual model through the mixed reality display device. During the operation, the mixed reality display device can update the position and status of the surgical instruments in real time and superimpose them on the virtual model for display, giving the surgeon intuitive visual feedback. The surgeon adjusts the position and operation of the surgical instruments according to the visual feedback provided by the mixed reality display module to ensure that the operation is precisely executed according to the preoperative plan. The mixed reality display device is configured with functions of magnifying, shrinking and rotating the surgical area to assist the surgeon in observing and understanding the structure and relationship of the surgical area, and records the navigation data during the operation and the operation log of the mixed reality display module for postoperative analysis and improvement of future operations.

[0076] Example 3, as Figure 3 shown, based on Examples 1 - 2, the present invention further provides a usage method of a virtual navigation system based on mixed reality technology, which is realized based on the virtual navigation system based on mixed reality technology and includes the following steps:

[0077] Step 1: Import the patient's preoperative CT / MR data into the workstation for three-dimensional modeling and planning, and import it into the workstation and the mixed reality glasses;

[0078] Step 2: After the patient's preoperative preparation is completed, install the fluoroscopy positioning module, use the C-arm to obtain the two-dimensional fluoroscopy image of the surgical area, input it into the workstation, and register it with the preoperative three-dimensional image data and planning information;

[0079] Step 3: Connect the fluoroscopy positioning module with the optical reflection navigation module. After the mixed reality glasses optically capture its position information and unify it with the patient coordinate system, the preoperative CT / MR images and the surgical planning information will be presented 1:1 in the patient's surgical area after registration;

[0080] Step 4: During the operation, connect the optical reflection navigation module to the instrument, and the mixed reality glasses capture and monitor the operation in real time throughout the process;

[0081] First, establish the operator's coordinate system based on the head-mounted device. The head-mounted device obtains the real-time relative position relationship between the operator and the surgical area by identifying the optical reflection navigation module installed on the intraoperative fluoroscopy positioning module, calculates the spatial coordinate transformation using the real-time position relationship, and realizes the unification of the operator's coordinate system and the surgical coordinate system. Take the surgical coordinate system as the reference origin of the operator's coordinate system. During the operation, capture the position information of the optical reference frame installed on the optical capture instrument. Before the operation, upload the precise modeling of the required instrument to the mixed reality glasses system. Due to the rigid connection of the reference frame, it obtains a relative position relationship fixed to the carrier. The glasses can accurately simulate the overall position of the instrument through the spatial position information of the reference frame, achieve precise virtual navigation, and realize high-frequency refreshing based on high computing power, reducing the discomfort of wearing the MR device. Furthermore, through the unification of the patient coordinate system, surgical coordinate system, operator coordinate system, and instrument coordinate system, achieve non-invasive virtual real-time navigation.

[0082] As described above, only the specific implementation manners of the present application are provided, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A virtual navigation system based on mixed reality technology, characterized in that: The virtual navigation system includes an intraoperative perspective positioning module, a spatial positioning and registration module, an optical reflection navigation module, and a mixed reality display module, wherein the modules are connected by electrical signals; The intraoperative fluoroscopic positioning module includes a positioning plate and an intraoperative fluoroscopic device, which is used to collect a two-dimensional fluoroscopic image of the surgical area with the positioning plate, and realize the positioning of the surgical area by obtaining the real-time position of the surgical area for positioning and registration, and convey it to the spatial positioning and registration module; The spatial positioning and registration module uses a high-performance workstation to process preoperative CT / MR data, perform three-dimensional reconstruction and integrate soft tissue and bone tissue information, establish a patient coordinate system, a surgical coordinate system, a surgeon coordinate system, and an instrument coordinate system, locate the surgical area, and form a spatial mapping with the preoperative image planning; The optical reflection navigation module includes an optical reflection reference frame, which is used to connect the surgical instrument and the intraoperative perspective imaging module to achieve the spatial positioning of the mixed reality device and the real-time monitoring of the instrument operation under the premise of non-invasiveness, so as to realize non-invasive virtual navigation; The mixed reality display module includes a high-precision mixed reality display device, which is used to obtain the spatial position information of the surgical area, preoperative planning and the real-time position of the instrument, give the surgeon visual feedback, and realize the full visualization of the surgical operation process.

2. The virtual navigation system based on mixed reality technology according to claim 1, characterized in that: In the intraoperative fluoroscopic positioning module, the process of achieving surgical area positioning is as follows: Before surgery, a positioning plate is installed on the surgical area. The positioning plate is used to provide a stable reference plane and contains marking points for registration, and then the intraoperative fluoroscopic device is connected to the relevant system of the operating room, wherein the intraoperative fluoroscopic device is a C-arm X-ray machine or a CBCT machine; Under sterile conditions, an intraoperative fluoroscopic device is used to perform fluoroscopic imaging of the surgical area with the positioning board, and a two-dimensional fluoroscopic image of the surgical area is obtained from the intraoperative fluoroscopic device; The two-dimensional perspective image obtained during the operation is registered with the three-dimensional image before the operation, and the final registration accuracy value is analyzed to make the actual position of the surgical area correspond to the position planned before the operation, and the real-time position information of the surgical area is calculated through image registration technology; The calculated position information is transmitted to the spatial positioning and registration module. After receiving the position information, the spatial positioning and registration module updates the surgical navigation system and displays the position and status of the surgical area to the doctor in real time during the operation.

3. The virtual navigation system based on mixed reality technology according to claim 2, characterized in that: The calculation formula of the final registration accuracy value is: Among them, P is the final registration accuracy value, A i is the position coordinate of the i-th feature point in the preoperative 3D image, B i is the position coordinate of the i-th corresponding feature point in the intraoperative perspective image, C i is the weight of the i-th feature point, n is the total number of feature points considered in the registration; The calculation formula of the real-time position information of the surgical area is: Among them, X is the instantaneous position information of the surgical area, P is the final registration accuracy value, P0 is the preset registration accuracy threshold, k is the coefficient of the exponential function, and A i is the position coordinate of the i-th feature point in the preoperative 3D image, B i is the position coordinate of the i-th corresponding feature point in the intraoperative perspective image, C i is the weight of the i-th feature point.

4. The virtual navigation system based on mixed reality technology according to claim 3, characterized in that: In the spatial positioning and registration module, the process of positioning the surgical area and forming a spatial mapping with the preoperative image planning is as follows: Obtain the patient's preoperative imaging data from the CT / MRI device, import the patient's preoperative CT / MR data into a high-performance workstation, and use the high-performance workstation to process the imaging data, including denoising and contrast enhancement; Using a workstation to perform three-dimensional reconstruction on the imported data, integrating soft tissue and bone tissue information to form a three-dimensional model of the patient, wherein the model includes detailed information of the soft tissue and bone tissue; A patient coordinate system is established with a certain part of the patient's body as a reference. The patient coordinate system is a Cartesian coordinate system consisting of three mutually perpendicular axes with a certain point on the patient's body as the origin. A surgical coordinate system is established based on the position and direction of the surgical area to describe the position and direction changes during the surgery. A surgeon coordinate system is established based on the surgeon's perspective and position. An instrument coordinate system is established with surgical instruments as a reference to describe the position and posture of the instruments during the surgery. The patient coordinate system, surgical coordinate system, operator coordinate system and instrument coordinate system are transformed, and the transformation matrix between the surgical coordinate system, operator coordinate system or instrument coordinate system and the patient coordinate system is analyzed to achieve the unification and alignment between the patient coordinate system, surgical coordinate system, operator coordinate system and instrument coordinate system, and then locate the surgical area and form a spatial mapping with the preoperative image planning.

5. The virtual navigation system based on mixed reality technology according to claim 4, characterized in that: The expression of the transformation matrix is: Where T is the transformation matrix, which is used to transform a point in one coordinate system to another coordinate system, and p j is the jth feature point in the patient coordinate system, q j is the jth corresponding feature point in the surgical coordinate system, the operator coordinate system or the instrument coordinate system, w j is the weight of the jth feature point in the transformation matrix, and n is the total number of feature points considered in the registration.

6. The virtual navigation system based on mixed reality technology according to claim 5, characterized in that: In the optical reflection navigation module, the process of implementing non-invasive virtual navigation is as follows: The optical reflection reference frame is fixedly connected to the surgical instrument through a connecting device, and the intraoperative fluoroscopic imaging module is calibrated with the optical reflection reference frame, and then the patient's preoperative imaging data is imported into the navigation system; During the operation, when the surgical instrument moves in the operation area, the optical reflection reference frame reflects specific light, and the intraoperative perspective imaging module captures the reflected light, tracks the position and movement of the optical reflection reference frame in real time, monitors the real-time position and posture changes of the surgical instrument, and converts it into a digital signal, which is transmitted to the virtual navigation system. The digital signal contains the precise position and posture information of the surgical instrument in three-dimensional space; Based on the received digital signal, the virtual navigation system analyzes the position vector of the surgical instrument in the three-dimensional space, calculates the position and direction of the surgical instrument in the three-dimensional space, and updates the navigation information of the virtual navigation system based on the calculation results, including the position, posture and relationship of the surgical instrument to the surgical area, and then feeds the updated navigation information back to the mixed reality display module in real time.

7. The virtual navigation system based on mixed reality technology according to claim 6, characterized in that: The expression of the position vector of the surgical instrument in three-dimensional space is: Where PW is the position vector of the surgical instrument in three-dimensional space, p is the target position vector of the surgical instrument in preoperative planning, q is the actual position vector of the surgical instrument during surgery, ||pq|| is the Euclidean distance between the preoperative planning position and the actual position during surgery, and d0 is the preset distance standard value, which indicates the maximum acceptable deviation between the preoperative planning position and the actual position during surgery. is the coefficient of the exponential function, and α is the adjustment coefficient, which is used to control the adjustment amplitude of the surgical instrument position vector.

8. The virtual navigation system based on mixed reality technology according to claim 7, characterized in that: In the mixed reality display module, the process of giving visual feedback to the operator is as follows: Through the optical reflection reference frame, the mixed reality display device obtains the spatial position information of the surgical area in real time, and superimposes the virtual image planned before surgery with the actual surgical scene to generate a mixed reality view; The surgeon wears a mixed reality display device, which enables the surgeon to observe a picture combining the real surgical area and the virtual model through the mixed reality display device; During the operation, the mixed reality display device updates the position and status of the surgical instruments in real time and overlays them with the virtual model to give the surgeon intuitive visual feedback. The surgeon adjusts the position and operation of surgical instruments based on the visual feedback provided by the mixed reality display module. The mixed reality display device is configured with the functions of zooming in, zooming out, and rotating the surgical area to assist the surgeon in observing and understanding the structure and relationship of the surgical area, and to record the navigation data during the operation and the operation log of the mixed reality display module.

9. A method for using a virtual navigation system based on mixed reality technology, implemented based on the virtual navigation system based on mixed reality technology according to any one of claims 1 to 8, characterized in that: The following steps are involved: Step 1: Import the patient's preoperative CT / MR data into the workstation for 3D modeling and planning, and import it into the workstation and mixed reality glasses; Step 2: After the patient's preoperative preparation is complete, the fluoroscopic positioning module is installed, and the C-arm is used to obtain a two-dimensional fluoroscopic image of the surgical area, which is input into the workstation and registered with the preoperative three-dimensional imaging data and planning information; Step 3: The perspective positioning module is connected to the optical reflection navigation module. The mixed reality glasses optically capture its position information and align it with the patient's coordinate system. After registration, the preoperative CT / MR images and surgical planning information are presented in a 1:1 ratio in the patient's surgical area. Step 4: During the operation, the optical reflective navigation module is connected to the instrument, and the mixed reality glasses capture and monitor the operation in real time throughout the entire process.

Images