Neuroendoscope under-endoscope guiding method based on multiple intraoperative scenes

Abstract

The invention relates to the technical field of neuroendoscopy, in particular to a neuroendoscopy under-endoscope guiding method based on multiple intraoperative scenes, which comprises the following steps: performing three-dimensional graph reconstruction based on a plurality of three-dimensional medical images shot by medical imaging equipment, and establishing a three-dimensional reconstruction scene; determining a space conversion matrix of the tail end of the neuroendoscope and the three-dimensional reconstruction scene according to a shooting result of the optical tracking locator and the three-dimensional medical image, and determining a virtual camera according to the space conversion matrix; adjusting the virtual camera according to the moving mode of the neuroendoscope, and obtaining a corresponding virtual endoscope picture after the neuroendoscope is moved; determining the axial direction of the endoscope according to the spatial transformation matrix, and determining a virtual section image along the axial direction of the endoscope according to the axial direction of the endoscope and the three-dimensional reconstruction scene; according to the invention, dynamic and high-precision three-dimensional guidance and augmented reality display of the neuroendoscope can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neuroendoscopy, in particular to a neuroendoscopy under-scope guiding method based on intraoperative multi-scene. BACKGROUND

[0002] With the continuous development of neuroendoscopy surgery technology, it is widely used in minimally invasive operations in complex tissue structure areas. The internal structure of the nervous system usually has characteristics such as various shapes, close spatial arrangement, and rich blood vessels, and the operation space is limited, and the important structure is complex. Therefore, before implementing such surgery, it is usually necessary to establish the spatial geometric relationship of the internal key structure of the object based on its three-dimensional image data, and in the actual operation process, the three-dimensional space corresponding to the target area through the two-dimensional image is realized. The confirmation and real-time positioning of the safe movement range of the instrument.

[0003] The existing neuroguide system mainly fuses preoperative and intraoperative images to provide spatial reference information for the operator. However, due to the limited display method of the guiding information, there are differences in the axial position, dimension and viewing angle between the guiding image and the actual field of view, which makes it difficult for the operator to quickly and intuitively interpret the guiding information, affecting the continuity and safety of the operation. In addition, the endoscopic intraoperative image often has problems such as three-dimensional structure distortion and limited visual range, which easily leads to the operator's spatial disorientation.

[0004] Although SLAM technology can realize three-dimensional reconstruction and positioning through video stream, it has high requirements for constant lighting and stable pixel brightness. In real scenes, uneven lighting, reflection, texture differences on organ surfaces, occlusion, mirror reflection, discontinuity and tissue deformation caused by endoscopic end-point light sources will seriously affect the accuracy and robustness of SLAM, bringing great challenges to endoscopic guidance.

[0005] Augmented reality technology faces the problems of registration of virtual data and real images and three-dimensional depth information display. The commonly used registration method is static registration, which is difficult to dynamically monitor the target area in real time during the operation. For depth prompting, the existing display device usually cannot provide intuitive depth information for the operator, affecting the accuracy of spatial perception. Therefore, how to realize dynamic and high-precision three-dimensional guidance and augmented reality display based on the real-time environment of the endoscope is still a technical problem faced by the field. SUMMARY

[0006] In view of the above problems, the present application provides a neuroendoscopy under-scope guiding method based on intraoperative multi-scene, which solves the technical problem that the application of SLAM technology and AR technology in the actual endoscopic scene is limited and cannot solve the intraoperative endoscopic disorientation in the prior art.

[0007] The present application provides a neuroendoscopy under-scope guiding method based on intraoperative multi-scene, comprising the following steps: Step S1, based on the medical imaging device, a plurality of three-dimensional medical images are taken to perform three-dimensional graphic reconstruction on the outer edge of the guiding space, the working area and the abnormal target of the neuroendoscope, and a three-dimensional reconstruction scene is established; Step S2, the tracking frame fixed on the neuroendoscope is photographed by the optical tracking positioner, the spatial conversion matrix of the neuroendoscope tip and the three-dimensional reconstruction scene is determined according to the optical tracking positioner photographing result and the three-dimensional medical image, and the virtual camera is determined by the spatial conversion matrix; Step S3, the virtual camera is adjusted according to the moving mode of the neuroendoscope, and the corresponding virtual endoscope picture after the neuroendoscope moves is obtained; the moving mode includes line of sight transformation, axis rotation and space motion; Step S4, the endoscope axial direction is determined according to the spatial conversion matrix, and the virtual cross-section image along the endoscope axial direction is determined according to the endoscope axial direction and the three-dimensional reconstruction scene; Step S5, the image photographed by the neuroendoscope, the virtual endoscope picture and the virtual cross-section image are simultaneously displayed on the display device, and the neuroendoscope is guided according to the picture displayed by the display device.

[0008] Preferably, step S1 specifically comprises: Step S2-1, the guiding space outer edge coordinate system, the conversion matrix from the guiding space outer edge coordinate system to the image coordinate system, the neuroendoscope coordinate system, the neuroendoscope tip coordinate system and the coordinate conversion matrix from the neuroendoscope coordinate system to the guiding space outer edge coordinate system are respectively obtained, and the spatial conversion matrix is obtained through matrix transformation; Step S2-2, the parameters of the virtual camera are set based on the position vector, the line of sight direction vector and the upward direction vector of the spatial conversion matrix, and the configured virtual camera is obtained.

[0009] Preferably, step S2-1 specifically comprises: Step S2-1-1, before the neuroendoscope is guided, the medical imaging device photographs the identifiable tracking frame of the guiding space outer edge to obtain the position of the guiding space outer edge before the guiding and the corresponding guiding space outer edge coordinate system The identifiable tracking frame of the guiding space outer edge is internally provided with a plurality of metal marker balls for marking the spatial position of the guiding space outer edge; Step S2-1-2, the identifiable tracking frame on the neuroendoscope is photographed by the optical tracking positioner to obtain the position of the neuroendoscope, and the conversion matrix from the guiding space outer edge coordinate system to the image coordinate system is obtained by registration according to the position of the guiding space outer edge and the position of the neuroendoscope; Step S2-1-3, the optical tracking positioner obtains the position of the identifiable tracking frame on the neuroendoscope in real time, and further determines the neuroendoscope coordinate system​ Step S2-1-4, determining the neural endoscope coordinate system based on the perpendicular relationship between the endoscope axis and the z-axis direction of the endoscope coordinate system Step S2-1-5, when the neural endoscope is guided, the optical tracking positioner also acquires the position of the identifiable tracking frame on the outer edge of the guide space in real time, and further determines the guide space outer edge coordinate system and coordinate conversion matrix Step S2-1-6, finally obtaining the space conversion matrix of the neural endoscope tip and the three-dimensional reconstructed scene .

[0010] Preferably, in step S2-1-4, the expression of the neural endoscope coordinate system

[0011] wherein, L represents the length of the neuroendoscope; In step S2-1-4, the expression of the space conversion matrix .

[0012] Preferably, step S3 specifically comprises: Step S3-1, determining a virtual camera transformation matrix from the space conversion matrix; determining the moving mode of the neural endoscope; Step S3-2, if the moving mode is line-of-sight transformation, determining the to-be-transformed line-of-sight, and updating the updated virtual camera transformation matrix based on the to-be-transformed line-of-sight; If the moving mode is self-rotation around the axis, the line-of-sight after self-rotation is determined, and the updated virtual camera transformation matrix is updated based on the line-of-sight after self-rotation; If the moving mode is spatial motion, the displacement direction, displacement distance, rotation axis and rotation angle of the neural endoscope are determined, and the virtual camera transformation matrix is updated by the displacement direction, displacement distance, rotation axis and rotation angle; Step S3-3, adjusting the spatial motion of the virtual camera by using the updated virtual camera transformation matrix to obtain the corresponding virtual endoscope picture after the neural endoscope moves.

[0013] Preferably, in step S3-2, the step of updating the updated virtual camera transformation matrix based on the to-be-transformed line-of-sight comprises: Calculate the first rotation axis RotateLine1: ​​​​​​​

[0014] wherein, represents an upper direction vector, represents a line-of-sight direction vector; calculating a line-of-sight transformation rotation matrix ;

[0015] wherein, represents a transpose, is in matrix form, I is a unit matrix, is a line-of-sight angle; updating a virtual camera transformation matrix:

[0016] wherein, represents an updated virtual camera transformation matrix; In step S3-3, the step of updating the updated virtual camera transformation matrix based on the self-rotation line-of-sight includes: calculating a second rotation axis :

[0017] calculating an axis self-rotation rotation matrix :

[0018] wherein is in matrix form, updating a virtual camera transformation matrix .

[0019] Preferably, step S4 specifically includes: Step S4-1, determining an endoscope axial direction vector based on the space conversion matrix; Step S4-2, determining a tangent plane from the endoscope axial direction vector and the position vector of the virtual camera, removing points in the three-dimensional reconstructed scene model located on one side of the plane equation normal vector direction, and forming a virtual tangent plane image from the remaining points.

[0020] Preferably, step S4-2 specifically includes: determining a tangent plane equation from the endoscope axial direction vector and the position vector of the virtual camera as:

[0021] wherein, is a normal vector of the tangent plane, is the distance from the plane to the origin; Let the point of the three-dimensional reconstructed scene be , calculate the distance from p to the plane,

[0022] If the sign of the distance value is positive, the point is located on the side of the normal vector of the plane, and if it is negative, it is located on the side opposite to the normal vector, the points located on the side of the normal vector of the plane equation in the three-dimensional reconstructed scene model are removed, and a virtual section image is formed.

[0023] Preferably, step S1 specifically comprises: Step S1-1, before guiding by the neuroendoscope, a plurality of three-dimensional medical images of the guide space outer edge, the operation area and the abnormal target are photographed by using a medical imaging device; Step S1-2, for the guide space outer edge, the size relationship between the three-dimensional medical image voxel gray value and the preset threshold value is determined to determine the first isosurface, the first isosurface is imported by using the vtk polygon data object, the mesh is mapped to the renderer by using the vtk polygon data attribute, and the vtk visualization object of the guide space outer edge is established; Step S1-3, for the operation area, the three-dimensional medical image voxel is imported by using the vtk image data object, the intersection of the ray emitted by the pixel and the three-dimensional medical image voxel is sampled to obtain a sampling point, the color and opacity of each sampling point along the line of sight direction are accumulated, the vtk image data attribute is used for drawing, and the vtk visualization object of the operation area is established; Step S1-4, for the abnormal target, a deep learning network algorithm is used to obtain a probability value of the three-dimensional medical image voxel belonging to the abnormal target, a second isosurface is determined according to the size relationship between the probability value and the probability threshold value, the second isosurface is imported by using the vtk polygon data object, the mesh is mapped to the renderer by using the vtk polygon data attribute, and the vtk visualization object of the abnormal target is established Step S1-5, the vtk renderer is used to load and render the vtk visualization of the guide space outer edge, the vtk visualization object of the operation area and the vtk visualization object of the abnormal target in sequence, and a three-dimensional reconstructed scene is established.

[0024] Compared with the prior art, the present application has at least the following beneficial effects: (1) The present application constructs an accurate three-dimensional reconstruction model and a guide coordinate system by spatially fusing the three-dimensional information obtained by multi-modal medical imaging and the real-time positioning data of the intraoperative endoscope. The present application provides a more intuitive and comprehensive spatial operation reference for the endoscope operator. By utilizing the spatial geometric relationship of multiple cameras, the position relationship between the guide space outer edge and the neuroendoscope can be registered based on stereo matching algorithms. The three-dimensional structural distortion in the neuroendoscope image can be eliminated in the three-dimensional reconstructed scene, thereby effectively reducing the risk of losing in the device positioning and guidance under complex spatial environment, and improving the accuracy and stability of the operation.

[0025] (2) The present application realizes real-time synchronization of virtual cameras and physical endoscope movement, automatically adjusts the reconstructed scene view angle according to the actual movement of the endoscope, and generates real-time cross-sectional images along the endoscope axis. Virtual field of view and cross-sectional information can be obtained instantly, improving the analysis efficiency of the device working area space movement, reducing the understanding burden and misjudgment probability of navigation information, and facilitating efficient and accurate image-guided operation.

[0026] (2) The present application integrates the real device collection picture, virtual field of view and related cross-section view on the same display terminal, forming a multi-dimensional linked operation interface. This integrated mode provides rich information support for the operator, and through the interactive comparison of different view pictures, the consistency of real-time operation and preset navigation path can be checked in time, effectively improving the visualization and control continuity of the overall navigation, and reducing the dependence of the surgical auxiliary system on special hardware and external devices. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of this application.

[0028] Figure 1 The present application provides a neural endoscope underguide method flow chart based on intraoperative multi-scene.

[0029] Figure 2 The present application provides a three-dimensional scene reconstruction schematic diagram.

[0030] Figure 3 The present application provides a schematic diagram of each device coordinate system.

[0031] Figure 4 The present application provides a VTK camera model schematic diagram.

[0032] Figure 5 The present application provides a schematic diagram of the three-dimensional reconstructed scene and the endoscope image.

[0033] Figure 6 The present application provides a schematic diagram of a variety of neural endoscope sight lines.

[0034] Figure 7 The present application provides a schematic diagram of the spatial movement of the neural endoscope. DETAILED DESCRIPTION

[0035] In order to enable one skilled in the art to better understand the above objects, features and advantages of the present application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0036] The present application provides a method for guiding a neuroendoscope under a microscope based on multiple intraoperative scenes. First, a high-precision three-dimensional model of the outer edge of the guiding space is constructed using preoperative CT / MRI images, and then the position and attitude of the real endoscope are real-time registered with the virtual three-dimensional model through an optical tracking system. Finally, the system can simulate the perspective and motion of the real endoscope, and provide enhanced cross-sectional views, dynamic, high-precision three-dimensional guidance and augmented reality display.

[0037] In order to illustrate the effectiveness of the method of the present application, the above technical solutions of the present application are described in detail below through a specific embodiment. As shown in Figure 1 The present application discloses a method for guiding a neuroendoscope under a microscope based on multiple intraoperative scenes, which includes the following steps: Step S1, based on the three-dimensional medical images taken by the medical imaging device, the outer edge of the guiding space, the working area and the abnormal target of the neuroendoscope are three-dimensionally reconstructed, and a three-dimensional reconstruction scene is established; Before the neuroendoscope is guided under the microscope, that is, before the operation, the present application uses the three-dimensional medical images of the environment during the guiding process of the neuroendoscope taken by the medical imaging device. Specifically, the skull is the outer edge of the guiding space of the neuroendoscope, the brain tissue is the working area of the neuroendoscope, and the lesion is the abnormal target of the neuroendoscope.

[0038] The medical imaging device of the present application can be a modality image taken by a CT, MRI or the like. The CT contains the guiding space outer edge, and the MRI contains the working area and the abnormal target. After extracting the corresponding information from the image, three-dimensional reconstruction is performed.

[0039] (1) Face drawing of the outer edge of the guiding space The core algorithm for 3D reconstruction using surface rendering in this invention is based on isosurface extraction technology. Its goal is to extract the surface mesh model of the outer edge of the guide space from CT images. A target threshold is set (usually 200 for CT skeletons). By traversing each voxel in the CT 3D data, the gray values ​​of the eight vertices of the voxel are compared with the target threshold. The vertex is marked as either inside (1) or outside (0) of the isosurface, and the state codes of the eight vertices are obtained. Based on the state codes, the corresponding triangular facet connection method is determined. Finally, the surface is smoothly reconstructed through interpolation to obtain the final surface model of the outer edge of the guide space.

[0040] A state code is formed based on the combinations of these eight vertices, and the triangular patch connection method of the isosurface within the voxel is determined according to the state code. Specifically, there are 2^8 = 256 possible combinations of the eight vertices. Each of the 256 states corresponds to an index table, which is represented in hexadecimal. For example, 0x80c = 1000 0000 1100 indicates that the edges 11, 3, and 2 of the voxel intersect with the isosurface.

[0041] The final result is a navigation space outer edge mesh model composed of a set of triangular facets. To ensure a smooth model surface, the extracted surface mesh can be further smoothed.

[0042] (2) Volume drawing of the work area This invention employs a volume rendering method for the work area, directly processing scalar field information in 3D volume data to visualize the internal structure without extracting geometric surfaces. Specifically, firstly, 3D medical image data of the work area is acquired using medical imaging equipment such as MRI. Then, when generating the final display image, rays are emitted from each pixel into the 3D volume data along the viewing direction. At each sampling point where the ray intersects with a voxel, sampling is performed according to a set step size, and an interpolation algorithm is used to calculate the color and opacity of each sampling point based on the voxel's scalar value. Next, following the order from the viewpoint to the depth of the volume data, the color and opacity of each sampling point are sequentially accumulated to obtain the pixel values ​​used for display. In this way, a true 3D visualization of the entire work area and its various layers can be achieved, intuitively displaying the internal structure.

[0043] (3) Surface drawing of abnormal targets Since the features of anomalous targets in MRI or CT image data are often not obvious, this invention first employs a deep learning network to segment anomalous targets in MRI or CT image data. Specifically, the image data is input into a trained neural network model to obtain the probability value of each voxel belonging to an anomalous target. Then, a threshold (e.g., 0.5) is set, and voxels with probabilities higher than this threshold are identified as anomalous targets. Based on the obtained probability segmentation results, an isosurface extraction algorithm is used to reconstruct the three-dimensional surface of the anomalous targets.

[0044] In some embodiments, the VTK tool can be used to create a 3D reconstructed scene, and the specific steps include: (1) Extract the surface mesh model of the guide space outer edge from the CT image to obtain triangular facet mesh data. Import the facets using the vtkPolyData object in VTK, map the mesh to the renderer using vtkPolyDataMapper, and create a vtkActor object for the guide space outer edge. You can set a high opacity and a light color (such as white or light gray) for the guide space outer edge to highlight its outline.

[0045] (2) For the work area, directly load 3D volume data such as MRI images, import the volume data through the vtkImageData object, and use vtkVolumeMapper to implement volume drawing, and establish the vtkVolume object of the work area. Different structural layers can be highlighted by customizing the color and opacity transfer function to achieve a realistic display of the work area, while ensuring effective superposition with the outer edge of the guide space.

[0046] (3) For anomalous targets, the three-dimensional surface mesh of the anomalous targets is obtained based on the deep learning segmentation results and the isosurface extraction algorithm. Then, it is imported using vtkPolyData, and the mesh is mapped to the renderer using vtkPolyDataMapper to create the vtkActor object of the anomalous targets.

[0047] (4) Use vtkRender to load and render the outer edge of the guide space, the working area and the abnormal target in sequence to establish a three-dimensional reconstruction scene.

[0048] Step S2: The optical tracking and positioning device captures images of the tracking frame fixed on the neuroendoscopy. Based on the images captured by the optical tracking and positioning device and the three-dimensional medical images, the spatial transformation matrix between the endoscope tip and the three-dimensional reconstructed scene is determined. The virtual camera is then determined based on the spatial transformation matrix. The objective lens of the neuroendoscope device is located at the end of the endoscope body, and the field of view image is collected in real time and transmitted back to the device, the virtual camera of VTK is used to perform three-dimensional reconstruction on the field of view image collected by the endoscope, the focal point, position, viewing angle and other parameters of the virtual camera are set according to the parameters of the endoscope device, and the virtual camera is determined.

[0049] In this step, as shown in the figure, the neuroendoscope of the application is installed at the end of the mechanical arm, held by the mechanical arm, and guided outside the space along with the identifiable tracking frame, and the endoscope is installed with the identifiable tracking frame. Figure 2

[0050] The identifiable tracking frame outside the guiding space is provided with a plurality of metal marker balls for marking the spatial position of the guiding space outside.

[0051] The optical tracking positioner can be a binocular camera or a multi-eye camera, and the optical tracking positioner can capture a binocular or multi-eye image for the augmented reality (AR) device to display through the images captured by the multiple cameras.

[0052] In addition, when the identifiable tracking frame on the neuroendoscope is photographed by the optical tracking positioner, the positions of the plurality of markers on the identifiable tracking frame in the optical image can be obtained, and the positional relationship between the guiding space outside and the neuroendoscope can be registered based on a stereo matching algorithm or the like by using the spatial geometric relationship of the multiple cameras, so as to obtain a conversion matrix from the guiding space outside coordinate system to the image coordinate system. The image coordinate system represents the coordinate system of the neuroendoscope field of view image space.

[0053] Through the above steps, the positional relationship between the guiding space outside and the neuroendoscope is registered, the three-dimensional structural distortion of the endoscope shooting picture can be eliminated, so that the established three-dimensional reconstruction scene has no distortion, and the guiding space of the neuroendoscope is more truly and accurately reflected.

[0054] When the neuroendoscope is guided, the optical tracking positioner can obtain the position of the identifiable tracking frame on the neuroendoscope in real time, and the neuroendoscope coordinate system can be determined.

[0055] The neuroendoscope of the application can ensure that the axial direction of the neuroendoscope is perpendicular to the z-axis direction of the endoscope terminal coordinate system through the positioning device during installation, so that the endoscope terminal coordinate system can be determined.

[0056] ​​​​​​wherein, represents the length of the neuroendoscope.

[0057] When the neuroendoscope is guided, the optical tracking positioner also acquires the position of the identifiable tracking frame on the outer edge of the guiding space in real time, and then determines the coordinate system of the outer edge of the guiding space , obtains the coordinate conversion matrix of and , and finally realizes the spatial conversion relationship between the end of the endoscope and the preoperative image, and the expression is:

[0058] wherein, represents the spatial conversion matrix of the end of the endoscope and the three-dimensional reconstructed scene. The coordinate systems of various devices are as shown in Figure 3 .

[0059] The application adopts the VTK camera in the VTK tool as the virtual camera. As shown in Figure 4 , the VTK camera model is displayed, and its essence is the same as the function of the human eye.

[0060] The position vector of the virtual camera represents the optical center position, wherein, is the optical center position coordinate.

[0061] The focal point represents the imaging plane position, wherein, is the imaging plane position coordinate.

[0062] The sight direction vector .

[0063] The virtual camera transformation matrix is ViewTransform, and the expression is:

[0064] wherein:

[0065]

[0066] wherein, represents the direction vector coordinate of the virtual camera, represents the upward direction vector, represents the sight direction vector.

[0067] After the transformation matrix of the real end of the endoscope and the three-dimensional reconstructed scene is determined , the value thereof is set to the value of the virtual camera transformation matrix. And according to the position vector , the sight direction vector ​And the up direction vector sets the parameters of the VTK virtual camera, and finally a configured virtual camera is obtained.

[0068] In step S3, the virtual camera is adjusted according to the moving mode of the neuroendoscope, and a virtual endoscope picture corresponding to the neuroendoscope after moving is obtained; the moving mode includes line of sight transformation, rotation around an axis and space motion. The neuroendoscope of the present application has multiple use scenarios in actual guiding process, and is adjusted according to actual conditions, the present application adjusts the posture and position of the virtual camera, realizes simulation of multiple operation modes such as line of sight transformation, rotation around an axis and space motion, so that the angle of view of the three-dimensional reconstructed scene is consistent with the angle of the real endoscope, and the final effect is as shown in Figure 5 It can be seen that the image view angle of the real endoscope is narrow and there is distortion at the edge of the image, while the three-dimensional reconstructed scene can see the image of the entire view angle, and the image does not have distortion. The following will be described in detail.

[0069] (1) Line of sight transformation The neuroendoscope is installed with an objective lens, the endoscope angle represents the angle formed between the incident light and the horizontal line, and is used to determine the direction and width of the line of sight. The commonly used endoscope equipment in clinic mainly has three line of sight angles: 0°, 30° and 70°, as shown in Figure 6 .

[0070] The present application realizes simulation of multiple endoscope view angles in clinic by rotating the VTK camera based on the cross product of the negative vector based on the line of sight direction and the up direction of the view angle, and the specific implementation manner is as follows: Calculate the first rotation axis RotateLine1 which is perpendicular to the vector of the view angle direction and the up direction of the view angle

[0071] Calculate the line of sight transformation rotation matrix

[0072]

[0073] Wherein, represents transposition, is in the form of a matrix, which is used for cross product calculation, I is a unit matrix, is the line of sight angle.

[0074] Update the virtual camera transformation matrix

[0075] Wherein, represents the updated virtual camera transformation matrix, which is used for line of sight transformation.

[0076] (2) Rotation around an axis The camera at the end of the endoscope device can be rotated along the endoscope shaft to achieve fixed position view adjustment. The present application achieves the rotation of the endoscope camera along the endoscope shaft in the clinic by rotating the VTK camera model along the line of sight direction.

[0077] Calculate the second rotation axis :

[0078] Calculate the rotation matrix R2 of the rotation around the axis

[0079] Wherein is in matrix form, used for cross multiplication calculation; I is the unit matrix Update the virtual camera transformation matrix

[0080] Adjust the virtual camera rotation around the axis using the updated virtual camera transformation matrix.

[0081] (3) Spatial motion When guiding, the neuroendoscope needs to be fine-tuned according to the work area. In the present application, the mechanical arm holds the neuroendoscope, and through the translation and rotation operation of the three directions of the mechanical arm, the small movement of the endoscope is realized. Correspondingly, the present application realizes the simulation of the small movement of the endoscope by adjusting the position of the vtk camera, as shown in Figure 7 .

[0082] The expression for updating the virtual camera transformation matrix is: For the neuroendoscope to translate along the x-axis by a distance of :

[0083] For the neuroendoscope to translate along the y-axis by a distance of :

[0084] For the neuroendoscope to translate along the z-axis by a distance of :

[0085] For the neuroendoscope to rotate around the x-axis by an angle of :

[0086] Wherein, represents the rotation matrix.

[0087] For the neuroendoscope to rotate around the y-axis by an angle of :

[0088] For the neuroendoscopy rotated around the z-axis by an angle of... :

[0089] An updated virtual camera transformation matrix is ​​used to adjust the spatial motion of the virtual camera. Finally, the virtual endoscopic image corresponding to the movement of the neuroendoscope is obtained.

[0090] Step S4: Determine the endoscope axis according to the spatial transformation matrix, and determine the virtual cross-sectional image along the endoscope axis according to the endoscope axis and the three-dimensional reconstructed scene; When a neuroendoscopy moves within the guided space, the actual position within that space cannot be determined using the two-dimensional images transmitted back from the endoscope. To address this, this invention provides a method that displays the endoscope's cross-sectional information within the guided space in real time based on its posture, offering multi-dimensional information to aid in diagnosis.

[0091] The spatial transformation matrix between the endoscope tip and the 3D reconstructed scene has been obtained in the above steps. ,from Extract the third column as the endoscope axis vector. .

[0092] Endoscope axial vector As the normal vector of the cutting plane Using the axial vector of the endoscope and the position vector of the virtual camera Define the plane equation of a tangent in three-dimensional space.

[0093] in, It is the normal vector of the cutting plane. It is the distance between the plane and the origin.

[0094] By calculating the distance from each point on the surface of the model in the 3D reconstructed scene to the plane, it is determined which parts are located on one side of the plane and which parts are located on the other side. Assume the points in the 3D reconstructed scene are... Calculate the distance from p to the plane.

[0095] If the distance value is positive, the point is located on one side of the plane's normal vector direction; if it is negative, it is located on the other side of the plane. During rendering, points on one side of the normal vector direction in the 3D reconstructed scene model are removed, and the updated data is submitted to the renderer to obtain a virtual cross-section image.

[0096] Step S5, simultaneously display the neuroendoscope captured image, the virtual endoscope picture and the virtual section image on the display device.

[0097] Finally, the neuroendoscope captured image, the virtual endoscope picture and the virtual section image are simultaneously displayed on the display device. Through the above method, the application can simulate the real endoscope view angle, motion during the neuroendoscope guidance, and provide enhanced section view, dynamic, high-precision three-dimensional guidance and augmented reality display The specific embodiments of the present application, although depicted in a particular order, should not be construed as requiring such order or sequential order to be performed, or that all illustrated acts or steps should be performed to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, the specific implementation details described above are not to be interpreted as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in subcombination or in a variety of combinations. The above description is merely illustrative of the preferred embodiments of the present application, and the scope of the present application is not limited to the above description. Any modifications or changes within the scope of the present application disclosed herein should be included in the scope of the present application.

[0098] The above description is merely illustrative of the preferred embodiments of the present application, and the scope of the present application is not limited to the above description. Any modifications or changes within the scope of the present application disclosed herein should be included in the scope of the present application.

Claims

1. A neuroendoscopic guided method based on multiple intraoperative scenarios, characterized in that, Includes the following steps: Step S1: Based on multiple three-dimensional medical images captured by medical imaging equipment, perform three-dimensional graphic reconstruction of the outer edge of the guidance space of the neuroendoscopy, the working area and abnormal targets, and establish a three-dimensional reconstruction scene; Step S2: The tracking frame fixed on the neuroendoscopy is photographed by the optical tracking and positioning device. Based on the photographed results of the optical tracking and positioning device and the three-dimensional medical images, the spatial transformation matrix between the end of the neuroendoscopy and the three-dimensional reconstructed scene is determined. The virtual camera is determined by the spatial transformation matrix. Step S3: Adjust the virtual camera according to the movement mode of the neuroendoscopy to obtain the virtual endoscope image corresponding to the movement of the neuroendoscopy; the movement mode includes line of sight change, rotation around the axis and spatial movement; Step S4: Determine the endoscope axis according to the spatial transformation matrix, and determine the virtual cross-sectional image along the endoscope axis according to the endoscope axis and the three-dimensional reconstructed scene; Step S5: Simultaneously display the images captured by the neuroendoscopy, the virtual endoscope screen, and the virtual cross-sectional image on the display device, and guide the neuroendoscopy according to the screen displayed on the display device.

2. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 1, characterized in that, Step S1 specifically includes: Step S2-1: Obtain the coordinate transformation matrix from the outer edge of the guide space coordinate system to the image coordinate system, the neuroendoscopy coordinate system, the neuroendoscopy end coordinate system, and the coordinate transformation matrix from the neuroendoscopy coordinate system to the outer edge of the guide space coordinate system. After matrix transformation, obtain the space transformation matrix. Step S2-2: Set the parameters of the virtual camera based on the position vector, line-of-sight vector, and up-direction vector of the spatial transformation matrix to obtain the configured virtual camera.

3. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 2, characterized in that, Step S2-1 specifically includes: Step S2-1-1: Before neuroendoscopic guidance, medical imaging equipment captures an image of the identifiable tracking frame at the outer edge of the guidance space to obtain the position of the outer edge of the guidance space and the corresponding coordinate system of the outer edge of the guidance space before guidance. The identifiable tracking frame along the outer edge of the guide space contains multiple metal marker balls for marking the spatial position of the outer edge of the guide space. Step S2-1-2: The optical tracking and positioning device captures an image of the identifiable tracking frame on the neuroendoscopy to obtain the position of the neuroendoscopy. Registration is performed based on the position of the outer edge of the guide space and the position of the neuroendoscopy to obtain the coordinate system from the outer edge of the guide space. To the image coordinate system Transformation matrix ; Step S2-1-3: The optical tracking and positioning device acquires the position of the identifiable tracking frame on the neuroendoscopy in real time, thereby determining the coordinate system of the neuroendoscopy. ; Step S2-1-4: Determine the coordinate system of the neuroendoscopy tip based on the perpendicular relationship between the axis of the endoscope and the z-axis direction of the coordinate system at the endoscope tip. ; Step S2-1-5: During neuroendoscopic guidance, the optical tracking and positioning device also acquires the position of the identifiable tracking frame on the outer edge of the guidance space in real time, thereby determining the coordinate system of the outer edge of the guidance space. ,get and coordinate transformation matrix ; Step S2-1-6: Finally, the spatial transformation matrix between the neuroendoscopic endoscope and the 3D reconstructed scene is obtained. .

4. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 3, characterized in that, In step S2-1-4, the coordinate system of the neuroendoscopic endoscope The expression is: in, Indicates the length of the neuroendoscopy; In step S2-1-4, the spatial transformation matrix The expression is: .

5. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 3, characterized in that, Step S3 specifically includes: Step S3-1: Determine the virtual camera transformation matrix from the spatial transformation matrix; determine the movement mode of the neuroendoscopy; Step S3-2: If the movement method is a line-of-sight change, then determine the line of sight to be changed, and update the virtual camera transformation matrix based on the line of sight to be changed; If the movement method is rotation around an axis, then determine the line of sight after rotation, and update the virtual camera transformation matrix based on the line of sight after rotation; If the movement is spatial motion, then determine the displacement direction, displacement distance, rotation axis, and rotation angle of the neuroendoscopy, and update the virtual camera transformation matrix based on the displacement direction, displacement distance, rotation axis, and rotation angle. Step S3-3: Adjust the spatial motion of the virtual camera using the updated virtual camera transformation matrix to obtain the virtual endoscope image corresponding to the movement of the neuroendoscopy.

6. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 5, characterized in that, In step S3-2, the step of updating the virtual camera transformation matrix based on the line of sight to be transformed includes: Calculate the first rotation axis RotateLine1: in, Represents the upward direction vector. Represents the vector of the line of sight; Calculate the rotation matrix of the view transformation ; in, Indicates transpose. for In matrix form, I is the identity matrix. For the view angle; update the virtual camera transformation matrix: in, This represents the updated virtual camera transformation matrix; In step S3-3, the step of updating the virtual camera transformation matrix based on the line of sight after rotation includes: Calculate the second axis of rotation : Calculate the rotation matrix about the axis : in for Update the virtual camera transformation matrix in matrix form. .

7. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 5, characterized in that, Step S4 specifically includes: Step S4-1: Determine the endoscope axis vector based on the spatial transformation matrix; Step S4-2: Determine the cross-section using the endoscope axial vector and the virtual camera position vector, remove points in the 3D reconstructed scene model that are located on one side of the plane equation normal vector direction, and form a virtual cross-section image from the remaining points.

8. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 7, characterized in that, Step S4-2 specifically includes: The equation of the tangent plane is determined by the axial vector of the endoscope and the position vector of the virtual camera as follows: in, It is the normal vector of the cutting plane. It is the distance between the plane and the origin; Let the points of the 3D reconstructed scene be... Calculate the distance from p to the plane. If the distance value is positive, the point is located on one side of the plane's normal vector direction; if it is negative, it is located on the opposite side of the normal vector direction. Points located on one side of the plane's normal vector direction in the 3D reconstructed scene model are removed to form a virtual cross-sectional image.

9. The intraoperative multi-scenario neuroendoscopic guidance method according to claim 7, characterized in that, Step S1 specifically includes: Step S1-1: Before the neuroendoscopy is performed, multiple three-dimensional medical images of the outer edge of the navigation space, the working area, and abnormal targets are captured using medical imaging equipment. Step S1-2: For the outer edge of the navigation space, determine the relationship between the gray values ​​of the voxels in the 3D medical image and the preset threshold to determine the first isosurface. Import the first isosurface through the VTK polygon data object, use the VTK polygon data attributes to map the mesh to the renderer, and establish the VTK visualization object of the outer edge of the guide space. Steps S1-3: For the work area, import the three-dimensional medical image voxel through the VTK image data object, sample the intersection of the ray emitted by the pixel and the three-dimensional medical image voxel to obtain sampling points, accumulate the color and opacity of each sampling point along the line of sight, draw using VTK image data attributes, and establish the VTK visualization object of the work area. Steps S1-4: For abnormal targets, a deep learning network algorithm is used to obtain the probability value of the three-dimensional medical image voxel belonging to the abnormal target. The second isosurface is determined by the relationship between the probability value and the probability threshold. The second isosurface is imported through VTK polygon data object. The VTK polygon data attribute is used to map the mesh to the renderer to establish the VTK visualization object of the abnormal target. Steps S1-5: Using the VTK renderer, load and render the VTK visualization of the outer edge of the guide space, the VTK visualization of the work area, and the VTK visualization of the abnormal target in sequence to establish a 3D reconstruction scene.

Images