Auxiliary skull base tumor operation treatment system based on mixed reality technology

Abstract

The invention discloses a cranial base tumor surgical treatment auxiliary system based on a mixed reality technology. The cranial base tumor surgical treatment auxiliary system comprises a presetting unit, an image processing unit, a data acquisition unit, a data transmission unit, a data storage unit, a data processing unit and an optical positioning unit, the presetting unit is used for setting markers, check points and feature points on the body of the patient in the real scene; the image processing unit is used for carrying out image scanning and projection on the patient in the real scene; and the data acquisition unit is used for acquiring information data generated in a mixed reality process in real time, and belongs to the technical field of medical images. According to the registration method based on point and surface combined application, an accurate space conversion relation is established between a three-dimensional virtual image coordinate system and a real scene coordinate system, the three-dimensional virtual image coordinate system and the real scene coordinate system are quickly registered, and a registration error value of a virtual model image and a patient entity is controlled within a certain range.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and more specifically discloses a skull base tumor surgical treatment auxiliary system based on mixed reality technology. Background Art

[0002] Intracranial tumors are a type of neurosurgical disease that seriously threatens human health, and surgical treatment is the main treatment method. At present, minimally invasive and precise treatment is the direction and trend of the development of intracranial tumor surgery in neurosurgery. In order to achieve minimally invasive and precise treatment of intracranial tumors, it is crucial to reconstruct and present the tumor and surgical-related anatomical structures in three dimensions. The emergence of neuronavigation systems has partially solved this problem, but due to its limited medical model reconstruction effect, single presentation form, high price, numerous supporting equipment, and complex operation process, it is not easy for clinicians to learn and master it quickly, and it is difficult to promote and apply it to the vast number of prefecture-level cities and county-level hospitals. With the application of electronic information technology and image post-processing software, new electronic digital information technology has been used to replace the neuronavigation system to play the role of three-dimensional reconstruction and tumor localization to assist neurosurgery.

[0003] Mixed reality technology is a mediated reality proposed by Steve Mann, a professor at the University of Toronto who is known as the "father of smart hardware". It is the most advanced combination of modern electronic digital information technologies after virtual reality and augmented reality. This technology creates a mutual feedback information loop between the real world, the virtual world and the user by presenting virtual images or scenes in real scenes, so that users can interact with virtual images in real scenes in real time. At present, mixed reality technology has been widely used in education, medical care, tourism, construction and other fields by giving users a real experience, especially in the field of medical surgery, which has broad application prospects.

[0004] Although mixed reality technology can holographically present 3D medical model images in clinical applications, the process of fully registering and fusing 3D images with the patient's corresponding physical objects (patient's head) is very complicated and requires long-term debugging at multiple angles to complete. In addition, 3D images are easily displaced by changes in the real scene or camera angle, and cannot track the real objects in the real scene in real time, which seriously affects the efficiency of mixed reality technology during surgery and significantly limits its application in surgical operations. How to achieve fast and accurate registration of virtual 3D images with real objects and real-time tracking of real objects is a difficulty in the development of mixed reality technology. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to solve the problem that although the current mixed reality technology can holographically present three-dimensional medical model images in clinical application operations, the process of completely aligning and fusing the three-dimensional image with the patient's corresponding real object (patient's head) is very complicated, and requires long-term debugging at multiple angles to complete. In addition, the three-dimensional image is easily displaced by changes in the real scene or the camera angle, and cannot track the real object in the real scene in real time, which seriously affects the efficiency of the use of mixed reality technology during surgery and significantly limits its application scope in surgical operations.

[0006] To solve the above technical problems, according to one aspect of the present invention, more specifically, a skull base tumor surgical treatment auxiliary system based on mixed reality technology includes: a presetting unit, an image processing unit, a data acquisition unit, a data transmission unit, a data storage unit, a data processing unit and an optical positioning unit;

[0007] Pre-setting unit: used to set markers, check points and feature points on patients in real scenes;

[0008] Image processing unit: used to scan and project images of patients in real scenes;

[0009] Data acquisition unit: used to collect information data generated in the mixed reality process in real time;

[0010] Data transmission unit: used for real-time transmission of information data collected by the data collection unit;

[0011] Data storage unit: used to store the information data transmitted in real time by the data transmission unit;

[0012] Data processing unit: used to process and analyze the data stored in the data storage unit, obtain the processing and analysis results, and generate a joint registration report based on the processing and analysis results;

[0013] Real-time monitoring unit: used to display the information data obtained by the data acquisition module and the analysis results of the core analysis module in real time on the background terminal. At the same time, when the results obtained by the core analysis module are abnormal, an alarm is issued to the background terminal;

[0014] Optical positioning unit: used to position the virtual model on the physical object corresponding to the real scene.

[0015] Furthermore, the presetting unit includes: a marker setting module, a check point setting module and a feature point setting module;

[0016] Marker setting module: used to set markers on the patient's head in real scenes;

[0017] Verification point setting module: used to set a number of evenly distributed verification points on the patient's head in a real scene;

[0018] Feature point setting module: used to set several feature points on the patient's face in real scenes.

[0019] Furthermore, the data acquisition unit includes: a coordinate acquisition module, an angle acquisition module, an area acquisition module, a curvature acquisition module and a vector acquisition module;

[0020] Coordinate acquisition module: used to collect three-dimensional coordinate information data of objects in virtual three-dimensional models and real scenes;

[0021] Angle acquisition module: used to collect information data of the rotation angle of the image processing unit;

[0022] Area acquisition module: used to collect area information data of the virtual 3D model and the real scene patient's face;

[0023] Curvature acquisition module: used to collect curvature information data of the virtual 3D model and the patient's face in the real scene;

[0024] Vector acquisition module: used to collect normal vector information data of feature points on the virtual 3D model and the patient's face in real scenes.

[0025] Furthermore, the data processing unit includes: a data acquisition module, a core analysis module and a report generation module;

[0026] Data acquisition module: used to acquire the information data stored in the data storage unit in real time;

[0027] Core analysis module: used to analyze and judge the information data obtained by the data acquisition module, so as to obtain the results of the analysis and judgment;

[0028] Report generation module: used to generate a joint registration report based on the analysis results of the core analysis module.

[0029] Furthermore, the real-time monitoring unit includes: a data retrieval module, an information display module and an abnormal alarm module;

[0030] Data retrieval module: used to retrieve the information data obtained by the data acquisition module and the analysis result information of the analysis and judgment module in real time;

[0031] Information display module: used to display the information data retrieved by the data retrieval module on the background terminal display screen;

[0032] Abnormal alarm module: used to send an alarm prompt to the background terminal when the result obtained by the analysis and judgment module is abnormal.

[0033] Furthermore, the core analysis module can obtain the information data of the three-dimensional coordinates of the objects in the virtual three-dimensional model and the real scene and the rotation angle of the image processing unit from the data acquisition module, and can obtain the correction index through the analysis of the above acquired data:

[0034]

[0035] in, is the correction index, x v is the x-axis coordinate value of the virtual 3D model marker image, y v is the coordinate value of the y-axis of the virtual 3D model marker image, z v is the coordinate value of the z-axis of the virtual 3D model marker image, x r is the x-axis coordinate value of the real scene landmark, y r is the coordinate value of the y-axis of the real scene landmark, z r is the coordinate value of the z-axis of the real scene landmark, x g is the x-axis coordinate value of the real scene imaging device, y g is the coordinate value of the y-axis of the real scene imaging device, z g is the coordinate value of the z-axis of the real scene imaging device, θ x is the rotation angle of the imaging device along the x-axis, θ y is the rotation angle of the imaging device along the y-axis, θ z is the rotation angle of the imaging device along the z-axis, and β is the preset conversion coefficient.

[0036] Furthermore, the core analysis module can obtain the three-dimensional coordinate information data of the virtual three-dimensional model and the objects in the real scene from the data acquisition module, and can obtain the verification index through the analysis of the above acquired data:

[0037]

[0038] Among them, δ is the calibration index, N is the number of pre-set calibration points, and μ is the correction index. is the coordinate value of the i-th check point on the x-axis in the virtual 3D model image, is the coordinate value of the ith check point on the y-axis in the virtual 3D model image, is the coordinate value of the i-th check point on the z-axis in the virtual 3D model image, is the coordinate value of the i-th check point on the x-axis in the real scene, is the coordinate value of the ith check point on the y-axis in the real scene, is the coordinate value of the ith check point on the z-axis in the real scene, and the check points in the virtual 3D model image and the real scene correspond to each other, and α is a preset threshold parameter.

[0039] Furthermore, the core analysis module can obtain the virtual three-dimensional model and the area, curvature and normal vector information data of the facial feature points of the real scene patient's face from the data acquisition module, and can analyze the data obtained above to derive:

[0040]

[0041] The coordinate values ​​of the virtual 3D model image According to the registration index, the registration is converted into Where f is the registration index, is the correction index, S v is the facial image area of ​​the virtual 3D model, S r is the facial area of ​​the patient in the real scene, is the curvature of the left cheek image of the virtual 3D model face, is the curvature of the right cheek image of the virtual 3D model face, is the curvature of the left cheek image of the patient’s face in the real scene, is the curvature of the right cheek image of the patient’s face in the real scene, M is the number of facial feature points, and p v is the normal vector of the feature point on the facial surface in the virtual 3D model image, q r is the normal vector of the feature point on the patient's facial surface in the real scene.

[0042] The beneficial effects of the skull base tumor surgical treatment auxiliary system based on mixed reality technology of the present invention are: based on the registration method of joint application of points and surfaces, an accurate spatial conversion relationship is established between the three-dimensional virtual image coordinate system and the real scene coordinate system and the two are quickly registered, and the registration error value of the virtual model image and the patient entity is controlled within a certain range. Then, the optical positioning method and other technologies are applied to reconstruct the spatial position of the target object through the mobile terminal camera according to the principle of stereoscopic vision, and the virtual model is positioned on the real object corresponding to the real scene according to active optical positioning, so as to realize the spatial positioning and real-time tracking of the real scene object by the three-dimensional image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0044] Figure 1 This is a schematic diagram of the system principle. DETAILED DESCRIPTION

[0045] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0046] According to one aspect of the present invention, Figure 1 As shown, a skull base tumor surgical treatment auxiliary system based on mixed reality technology is provided. First, the markers, check points and feature points are set on the patient's head in the real scene respectively through the marker setting module, the check point setting module and the feature point setting module, and then the objects and patients in the real scene are scanned by the image processing unit, and the model images of the tumor and other related anatomical structures reconstructed by the Slicer software are obtained at the same time. After that, the three-dimensional coordinates of the objects in the virtual three-dimensional model and the real scene, the rotation angle of the image processing unit, the area and curvature of the face of the patient in the virtual three-dimensional model and the real scene, and the normal vector information data of the feature points on the patient's face are collected respectively through the coordinate acquisition module, the angle acquisition module, the area acquisition module, the curvature acquisition module and the vector acquisition module, and then the collected data is transmitted to the data storage unit in real time through the data transmission unit, and then the information data is obtained from the data storage unit through the data acquisition module in the data processing unit, and further, the obtained information data is processed and analyzed through the core analysis module.

[0047] Afterwards, when the results analyzed and judged by the core analysis module show abnormal conditions, the abnormal alarm module will be triggered to alarm the background terminal, and the analysis results will be further analyzed and aligned, and a joint alignment report will be generated and displayed in the information display module.

[0048] The data retrieval module in the real-time monitoring unit can retrieve the analysis results of the core analysis module and the information data stored in the data storage unit in real time, and display the retrieved information data on the background terminal display screen through the information display module. At the same time, when the result obtained by the core analysis module is an abnormal result, the abnormal alarm module will issue an alarm prompt to the background terminal.

[0049] Afterwards, optical positioning and other technologies are applied to reconstruct the spatial position of the target object based on the principle of stereoscopic vision through the mobile terminal camera, and the virtual model is positioned on the physical object corresponding to the real scene based on the active optical positioning unit, realizing the spatial positioning and real-time tracking of the three-dimensional image on the physical object in the real scene.

[0050] The core analysis module can obtain the information data of the three-dimensional coordinates of the objects in the virtual three-dimensional model and the real scene and the rotation angle of the image processing unit from the data acquisition module, and can obtain the correction index through the analysis of the above acquired data:

[0051]

[0052] in, is the correction index, x v is the x-axis coordinate value of the virtual 3D model marker image, y v is the coordinate value of the y-axis of the virtual 3D model marker image, z v is the coordinate value of the z-axis of the virtual 3D model marker image, x r is the x-axis coordinate value of the real scene landmark, y r is the coordinate value of the y-axis of the real scene landmark, z r is the coordinate value of the z-axis of the real scene landmark, x g is the x-axis coordinate value of the real scene imaging device, y g is the coordinate value of the y-axis of the real scene imaging device, z g is the coordinate value of the z-axis of the real scene imaging device, θ x is the rotation angle of the imaging device along the x-axis, θ y is the rotation angle of the imaging device along the y-axis, θ z is the rotation angle of the imaging device along the z-axis, and β is the preset conversion coefficient.

[0053] The core analysis module can obtain the virtual 3D model and the 3D coordinate information data of the objects in the real scene from the data acquisition module, and can obtain the verification index through the above acquired data analysis, if:

[0054]

[0055] It means that after the first correction, the virtual 3D coordinate error is large, which will trigger the abnormal alarm module to send an alarm to the background terminal. At the same time, the virtual 3D coordinates will be further aligned through the core analysis module. Otherwise, it means that the error is small. Among them, δ is the verification index, N is the number of pre-set verification points, and μ is the correction index. is the coordinate value of the i-th check point on the x-axis in the virtual 3D model image, is the coordinate value of the ith check point on the y-axis in the virtual 3D model image, is the coordinate value of the i-th check point on the z-axis in the virtual 3D model image, is the coordinate value of the i-th check point on the x-axis in the real scene, is the coordinate value of the ith check point on the y-axis in the real scene, is the coordinate value of the ith check point on the z-axis in the real scene, and the check points in the virtual 3D model image and the real scene correspond to each other, and α is a preset threshold parameter, whose value is 0.05.

[0056] The core analysis module can obtain the virtual 3D model and the area, curvature and normal vector information of the facial feature points of the real scene patient's face from the data acquisition module, and can analyze the data obtained above to derive:

[0057]

[0058] The coordinate values ​​of the virtual 3D model image According to the registration index, the registration is converted into Where f is the registration index, is the correction index, S v is the facial image area of ​​the virtual 3D model, S r is the facial area of ​​the patient in the real scene, is the curvature of the left cheek image of the virtual 3D model face, is the curvature of the right cheek image of the virtual 3D model face, is the curvature of the left cheek image of the patient’s face in the real scene, is the curvature of the right cheek image of the patient’s face in the real scene, M is the number of facial feature points, and the facial feature points can be selected from the corners of the eyes, the corners of the mouth and the tip of the nose, p v is the normal vector of the feature point on the facial surface in the virtual 3D model image, q r It is the normal vector of the feature point on the patient's facial surface in the real scene. It should be noted that the normal vector of a point on the facial surface represents the direction of the surface at that point. These normal vectors can be used to analyze the orientation and inclination of the face.

[0059] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. The skull base tumor surgical treatment auxiliary system based on mixed reality technology is characterized by: include: A presetting unit, an image processing unit, a data acquisition unit, a data transmission unit, a data storage unit, a data processing unit, and an optical positioning unit; Pre-setting unit: used to set markers, check points and feature points on patients in real scenes; Image processing unit: used to scan and project images of patients in real scenes; Data acquisition unit: used to collect information data generated in the mixed reality process in real time; Data transmission unit: used for real-time transmission of information data collected by the data collection unit; Data storage unit: used to store the information data transmitted in real time by the data transmission unit; Data processing unit: used to process and analyze the data stored in the data storage unit, obtain the processing and analysis results, and generate a joint registration report based on the processing and analysis results; Real-time monitoring unit: used to display the information data obtained by the data acquisition module and the analysis results of the core analysis module in real time on the background terminal. At the same time, when the results obtained by the core analysis module are abnormal, an alarm is issued to the background terminal; Optical positioning unit: used to position the virtual model on the physical object corresponding to the real scene.

2. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 1 is characterized by: The pre-setting unit includes: a marker setting module, a check point setting module and a feature point setting module; Marker setting module: used to set markers on the patient's head in real scenes; Verification point setting module: used to set a number of evenly distributed verification points on the patient's head in a real scene; Feature point setting module: used to set several feature points on the patient's face in real scenes.

3. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 2 is characterized by: The data acquisition unit includes: a coordinate acquisition module, an angle acquisition module, an area acquisition module, a curvature acquisition module and a vector acquisition module; Coordinate acquisition module: used to collect three-dimensional coordinate information data of virtual three-dimensional models and objects in real scenes; Angle acquisition module: used to collect information data of the rotation angle of the image processing unit; Area acquisition module: used to collect area information data of the virtual 3D model and the patient's face in the real scene; Curvature acquisition module: used to collect curvature information data of the virtual 3D model and the patient's face in the real scene; Vector acquisition module: used to collect normal vector information data of feature points on the virtual 3D model and the patient's face in real scenes.

4. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 3 is characterized by: The data processing unit includes: a data acquisition module, a core analysis module and a report generation module; Data acquisition module: used to acquire the information data stored in the data storage unit in real time; Core analysis module: used to analyze and judge the information data obtained by the data acquisition module, so as to obtain the results of the analysis and judgment; Report generation module: used to generate a joint registration report based on the analysis results of the core analysis module.

5. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 4 is characterized in that: The real-time monitoring unit includes: a data retrieval module, an information display module and an abnormal alarm module; Data retrieval module: used to retrieve the information data obtained by the data acquisition module and the analysis result information of the analysis and judgment module in real time; Information display module: used to display the information data retrieved by the data retrieval module on the background terminal display screen; Abnormal alarm module: used to send an alarm prompt to the background terminal when the result obtained by the analysis and judgment module is abnormal.

6. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 5 is characterized by: The core analysis module can obtain the information data of the three-dimensional coordinates of the objects in the virtual three-dimensional model and the real scene and the rotation angle of the image processing unit from the data acquisition module, and can obtain the correction index through the analysis of the above acquired data: in, is the correction index, x v is the x-axis coordinate value of the virtual 3D model marker image, y v is the coordinate value of the y-axis of the virtual 3D model marker image, z v is the coordinate value of the z-axis of the virtual 3D model marker image, x r is the x-axis coordinate value of the real scene landmark, y r is the coordinate value of the y-axis of the real scene landmark, z r is the coordinate value of the z-axis of the real scene landmark, x g is the x-axis coordinate value of the real scene imaging device, y g is the coordinate value of the y-axis of the real scene imaging device, z g is the coordinate value of the z-axis of the real scene imaging device, θ x is the rotation angle of the imaging device along the x-axis, θ y is the rotation angle of the imaging device along the y-axis, θ z is the rotation angle of the imaging device along the z-axis, and β is the preset conversion coefficient.

7. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 6 is characterized by: The core analysis module can obtain the virtual three-dimensional model and the three-dimensional coordinate information data of the objects in the real scene from the data acquisition module, and can obtain the verification index through the above acquired data analysis: Among them, δ is the calibration index, N is the number of pre-set calibration points, and μ is the correction index. is the coordinate value of the i-th check point on the x-axis in the virtual 3D model image, is the coordinate value of the ith check point on the y-axis in the virtual 3D model image, is the coordinate value of the i-th check point on the z-axis in the virtual 3D model image, is the coordinate value of the i-th check point on the x-axis in the real scene, is the coordinate value of the ith check point on the y-axis in the real scene, is the coordinate value of the ith check point on the z-axis in the real scene, and the check points in the virtual 3D model image and the real scene correspond to each other, and α is a preset threshold parameter.

8. The skull base tumor surgical treatment auxiliary system based on mixed reality technology according to claim 7 is characterized by: The core analysis module can obtain the virtual three-dimensional model and the area, curvature and normal vector information data of the facial feature points of the real scene patient's face from the data acquisition module, and can analyze the data obtained above to derive: The coordinate values ​​of the virtual 3D model image According to the registration index, the registration is converted into Where f is the registration index, is the correction index, S v is the facial image area of ​​the virtual 3D model, S r is the facial area of ​​the patient in the real scene, is the curvature of the left cheek image of the virtual 3D model face, is the curvature of the right cheek image of the virtual 3D model face, is the curvature of the left cheek image of the patient’s face in the real scene, is the curvature of the right cheek image of the patient’s face in the real scene, M is the number of facial feature points, and the facial feature points can be selected from the corners of the eyes, the corners of the mouth and the tip of the nose, p v is the normal vector of the feature point on the facial surface in the virtual 3D model image, q r is the normal vector of the feature point on the patient's facial surface in the real scene.

Images