Three-dimensional Localization and Surgical Navigation System for Gastrointestinal Stromal Tumors with Multimodal Image Fusion

Abstract

The present invention relates to the technical field of three-dimensional modeling, and particularly to a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multi-modal image fusion. The system includes: a model initial construction module, which is used to obtain CT images, MRI images and endoscopic images of gastrointestinal stromal tumors, and generate an initial three-dimensional model from multi-layer CT images; a model correction module, which is used to increase the number of triangular meshes in the initial three-dimensional model according to the analysis of the gastrointestinal stromal tumor region in the MRI image to obtain a mesh three-dimensional model; a texture matching module, which is used to determine the best matching pixel corresponding to each triangular mesh in the mesh three-dimensional model, and the best matching pixel belongs to the endoscopic image; a texture rendering module, which is used to perform texture rendering on the mesh three-dimensional model according to the best matching pixel to obtain a three-dimensional positioning model of the gastrointestinal stromal tumor. The present invention combines various types of images to reconstruct a three-dimensional model, improving the detail rendering of the gastrointestinal stromal tumor model.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and particularly to a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion. Background Art

[0002] Gastrointestinal stromal tumors (GIST) originate from the mesenchymal tissue of the gastrointestinal wall, and the tumor can grow into the lumen, between the walls or outside the lumen. The precise diagnosis and treatment of gastrointestinal stromal tumors rely on the three-dimensional positioning of the tumor location, morphology, and relationship with surrounding organs. Traditional single imaging techniques have their limitations. For example, CT images have low sensitivity to rectal lesions, while MRI images have more advantages in three-dimensional imaging and tissue contrast. Multimodal image fusion can improve the positioning accuracy by integrating complementary information of different modalities, such as integrating complementary information of CT images, MRI images, PET images, endoscopic images, etc.

[0003] Whether the tumor has metastasis and whether it infiltrates surrounding tissues are important indicators for judging benign and malignant. In traditional three-dimensional reconstruction techniques, only the size, morphology, and anatomical relationship with organs of the tumor are concerned, and the details of gastrointestinal stromal tumors are insufficiently rendered. For example, the invasion of adjacent regions and organs by the tumor results in a large number of missing texture details in the three-dimensional model when establishing the three-dimensional model, making it difficult to determine the invasion boundary of gastrointestinal stromal tumors in the three-dimensional model. Summary of the Invention

[0004] In order to solve the technical problem of insufficient detail rendering of gastrointestinal stromal tumors when establishing a three-dimensional model using traditional three-dimensional reconstruction techniques, the purpose of the present invention is to provide a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion. The specific technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present invention provides a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion. The system includes the following modules:

[0006] A model initial construction module for obtaining CT images, MRI images, and endoscopic images of gastrointestinal stromal tumors; generating an initial three-dimensional model from multi-layer CT images; wherein, the initial three-dimensional model is in the form of a triangular patch mesh.

[0007] A model correction module for extending the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judging the internal and external invasion trends according to the internal and external growth properties of the rendering reference line, and determining the invasion risk coefficient; increasing the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a mesh three-dimensional model.

[0008] A texture matching module, which is used to slice the endoscopic image into multiple pixels in regions, obtain the texture features within each pixel; perform feature matching on the texture features corresponding to adjacent triangular meshes in the grid three-dimensional model and the texture features corresponding to the pixels, and obtain the best matching pixel corresponding to each triangular mesh.

[0009] A texture rendering module, which is used to perform texture rendering on the grid three-dimensional model according to the best matching pixels, and obtain a three-dimensional positioning model of gastrointestinal stromal tumor.

[0010] Further, the generation of the initial three-dimensional model from the multi-slice CT images includes:

[0011] Stack the multi-slice CT images in the Z-axis direction to generate a three-dimensional reconstruction model; perform smooth reconstruction on the three-dimensional model to obtain the initial three-dimensional model; wherein, the initial three-dimensional model is in the form of a triangular patch grid.

[0012] Further, the extension of the contour line of the gastrointestinal stromal tumor region in the extended MRI image to obtain a rendering reference line includes:

[0013] Based on the MRI image, segment the gastrointestinal stromal tumor region;

[0014] Use the canny detection algorithm to obtain the boundary contour line of the digestive system to which the gastrointestinal stromal tumor region adheres;

[0015] Take the two ends of the contour line segment between the two intersection points of the gastrointestinal stromal tumor region on the boundary contour line of the digestive system to which it adheres as the extension starting points, and starting from the extension starting points, extend the contour line segment to both sides by a preset length to obtain the rendering reference line.

[0016] Further, the determination of the internal and external invasion trend and the determination of the invasion risk coefficient according to the internal and external growth properties of the rendering reference line include:

[0017] Extend the rendering reference line along the direction from the centroid of the gastrointestinal stromal tumor region to each position point on the rendering reference line, wherein each position point on the rendering reference line has its own corresponding extended line segment;

[0018] Determine the invasion trend vector of each extended line segment according to the continuous TI signal intensity of each position point on the extended line segment;

[0019] Determine the invasion direction according to the extended thickness of the extended line segment of each position point on the rendering reference line;

[0020] The included angle formed by the direction of the vector sum of the invasion trend vectors of all extended line segments and the invasion direction is used as the invasion trend transfer included angle;

[0021] Determine the infringement risk coefficient by combining the size of the infringement trend transfer angle and the fluctuation of the expansion thickness of the extended line segment corresponding to the rendering reference line.

[0022] Further, increasing the number of triangular meshes in the initial three-dimensional model according to the infringement risk coefficient to obtain a mesh three-dimensional model includes:

[0023] Increase the number of triangular meshes in the initial three-dimensional model in the area corresponding to the infringement trend transfer angle, and the multiple of the increased triangular meshes is a preset multiple of the infringement risk coefficient to obtain a network three-dimensional model.

[0024] Further, the texture features in each pixel include: gray-level co-occurrence matrix features and gray-level region size matrix features.

[0025] Further, performing feature matching on the texture features corresponding to adjacent triangular meshes in the mesh three-dimensional model and the texture features corresponding to the pixels to obtain the best-matching pixel corresponding to each triangular mesh includes:

[0026] Obtain the triangular meshes within the neighborhood range of each triangular mesh in the mesh three-dimensional model as neighborhood triangular meshes;

[0027] Construct a pixel texture matrix corresponding to each pixel from the texture features of each pixel in the endoscopic image; each texture feature corresponds to a pixel texture matrix;

[0028] Taking any triangular mesh as the target triangular mesh, construct a basic texture matrix from the texture features of all the neighborhood triangular meshes of the target triangular mesh; each texture feature of each triangular mesh corresponds to a basic texture matrix;

[0029] Perform interpolation processing on the basic texture matrix, and align the interpolated basic texture matrix with the pixel texture matrix;

[0030] Calculate the difference matrix between the interpolated basic texture matrix corresponding to the same texture feature and the pixel texture matrix of each pixel to obtain the difference matrix of each texture feature of each pixel;

[0031] Analyze the difference matrix of each texture feature of each pixel respectively to determine the pixel that best matches the target triangular mesh as the best-matching pixel.

[0032] Further, analyzing the difference matrix of each texture feature of each pixel respectively to determine the pixel that best matches the target triangular mesh as the best-matching pixel includes:

[0033] Determine the fine texture interpretability according to the difference matrix corresponding to the gray-level co-occurrence matrix;

[0034] Determine the texture contrast interpretability according to the difference matrix corresponding to the feature of the gray-scale region size matrix;

[0035] Combine the fine texture interpretability and the texture contrast interpretability to determine the matching degree corresponding to each pixel and the target triangular mesh;

[0036] Take the pixel corresponding to the maximum matching degree as the best matching pixel of the target triangular mesh.

[0037] Further, performing texture rendering on the grid 3D model according to the best matching pixel to obtain a 3D localization model of gastrointestinal stromal tumor, including:

[0038] Input all texture feature data and 3D rendering sub-resources within the range of the best matching pixel corresponding to the target triangular mesh into the target triangular mesh for texture rendering, and perform smoothing processing on the texture-rendered triangular mesh to obtain a 3D model of gastrointestinal stromal tumor as the 3D localization model.

[0039] Further, performing texture rendering on the grid 3D model according to the best matching pixel further includes:

[0040] Map each vertex of each triangular mesh in the grid 3D model to the 2D texture space of the CT image to generate UV coordinates;

[0041] Perform texture sampling on the 2D CT image region corresponding to each triangular mesh according to the UV coordinates of the vertices of the triangular mesh, and perform texture rendering on the interior of the triangular mesh based on the UV coordinates of the three vertices in each triangular mesh of the grid 3D model.

[0042] In a second aspect, a method for 3D localization and surgical navigation of gastrointestinal stromal tumor with multi-modal image fusion is provided, and the method includes the following steps:

[0043] Obtain the CT image, MRI image and endoscopic image of the gastrointestinal stromal tumor; generate an initial 3D model from the multi-slice CT image; wherein, the initial 3D model is in the form of a triangular patch mesh;

[0044] Extend the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judge the internal and external invasion trend according to the internal and external growth of the rendering reference line, and determine the invasion risk coefficient; increase the number of triangular meshes in the initial 3D model according to the invasion risk coefficient to obtain a grid 3D model;

[0045] The endoscopic image is divided into multiple pixels by region, and the texture features within each pixel are obtained; the texture features corresponding to adjacent triangular meshes in the grid three-dimensional model are feature-matched with the texture features corresponding to the pixels to obtain the best-matched pixel corresponding to each triangular mesh;

[0046] According to the best-matched pixel, texture rendering is performed on the grid three-dimensional model to obtain a three-dimensional positioning model of gastrointestinal stromal tumor.

[0047] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the embodiments of all possible implementations in the first aspect are implemented.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer program product, which includes: computer program code. When the computer program code runs on a computer, the computer is caused to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0049] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the embodiments of all possible implementations in the first aspect.

[0050] The embodiments of the present invention have at least the following beneficial effects:

[0051] In this application, an initial three-dimensional model is reconstructed from multi-layer CT images, and then the exophytic and endophytic growth types of gastrointestinal stromal tumors are extracted using MRI images. The density of triangular meshes in the regions with invasion tendency is corrected to reserve and provide a larger rendering space; further, by extracting the texture features within the pixels of the endoscopic image, the corrected triangular meshes are matched, and matching relationships are constructed around different texture features. According to the matching relationships, the best-matched pixel corresponding to each triangular mesh is obtained, and different rendering sub-resources are implanted in combination with the best-matched pixels to perform texture rendering on the grid three-dimensional model, so that the gastrointestinal stromal tumor and its surrounding tissues can obtain more delicate texture rendering, and the possible invasion tendency can be more completely expressed. By reconstructing the three-dimensional model with multiple types of images, the detail rendering of the gastrointestinal stromal tumor model is improved. Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0053] Figure 1 System block diagram of a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion provided by an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of endoscopic image of gastrointestinal stromal tumor provided by an embodiment of the present invention;

[0055] Figure 3 Schematic diagram of CT image of gastrointestinal stromal tumor provided by an embodiment of the present invention;

[0056] Figure 4 Schematic diagram of MRI image of gastrointestinal stromal tumor provided by an embodiment of the present invention;

[0057] Figure 5 Schematic diagram of multi-slice spiral CT image in the portal venous phase provided by an embodiment of the present invention;

[0058] Figure 6 Partial top view of the three-dimensional reconstruction model after three-dimensional reconstruction of all layers of CT images provided by an embodiment of the present invention;

[0059] Figure 7 Schematic diagram of the initial three-dimensional model provided by an embodiment of the present invention;

[0060] Figure 8 Schematic diagram of the rendering reference line corresponding to the GIST region provided by an embodiment of the present invention;

[0061] Figure 9 Method flowchart of a three-dimensional positioning and surgical navigation method for gastrointestinal stromal tumors with multimodal image fusion provided by an embodiment of the present invention;

[0062] Figure 10 Schematic diagram of the central triangular mesh and its corresponding neighborhood triangular mesh provided by an embodiment of the present invention. Detailed implementation manners

[0063] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following combines the accompanying drawings and preferred embodiments to elaborate in detail on the specific implementation manners, structures, features and effects of the three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion proposed according to the present invention.

[0064] In the following description, different "an embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0065] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two.

[0066] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0067] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0068] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0069] The specific scheme of the gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system with multimodal image fusion provided by the present invention is described in detail below with reference to the accompanying drawings.

[0070] See also Figure 1 , which shows a system block diagram of a gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system with multimodal image fusion provided by an embodiment of the present invention, the system includes the following modules:

[0071] The model initialization module 10 is used to obtain CT images, MRI images and endoscopic images of gastrointestinal stromal tumors; generate an initial three-dimensional model from multi-slice CT images; wherein the initial three-dimensional model is in the form of a triangular mesh.

[0072] Endoscopic imaging: It can directly observe submucosal lesions and clarify the origin layer of the tumor. For example, GIST mostly originates from the muscularis propria. It can also be used to evaluate the relationship between the tumor and surrounding tissues. However, due to the limited field of view, it is difficult to observe and evaluate the overall picture of the tumor.

[0073] CT imaging (Computed Tomography): It has fast scanning speed, high spatial resolution, supports multi-angle observation and three-dimensional reconstruction, and is suitable for surgical planning. However, the soft tissue contrast is low and it is difficult to distinguish boundary information.

[0074] Magnetic Resonance Imaging (MRI): It shows a better view of the invasion range of adjacent structures of tumors than CT images, but has a low spatial resolution and is easily interfered by intestinal peristalsis artifacts, resulting in poor image quality and easy omission of small lesions. Magnetic Resonance Imaging is simply referred to as MRI image.

[0075] First, obtain the CT images, MRI images, and endoscopic images of gastrointestinal stromal tumors respectively. It should be noted that the CT images are multi-slice spiral CT images (64 rows).

[0076] Among them, CT and MRI examinations should be completed as close as possible in the same time period, and the unified respiratory gating technology is adopted, such as synchronization of breath-holding commands, to reduce the influence of intestinal peristalsis and organ displacement on MRI.

[0077] These three imaging methods, CT images, MRI images, and endoscopic images, each have their own advantages and disadvantages. In order to more precisely construct a three-dimensional model, it is necessary to combine the advantages of different imaging methods for three-dimensional reconstruction and positioning of adjacent structures, and solve the problem of difficult data registration caused by the phase and resolution differences of different imaging devices.

[0078] Please refer to Figure 2 , Figure 2 for the schematic diagram of the endoscopic image of gastrointestinal stromal tumor; please refer to Figure 3 , Figure 3 for the schematic diagram of the CT image of gastrointestinal stromal tumor; please refer to Figure 4 , Figure 4 for the schematic diagram of the MRI image of gastrointestinal stromal tumor.

[0079] Gastrointestinal stromal tumor (GIST) is the most common mesenchymal tumor in the gastrointestinal tract, independently originating from the gastrointestinal wall, with non-directional differentiation characteristics, and can occur in any part of the digestive tract.

[0080] During three-dimensional reconstruction, the multi-slice spiral CT images of GIST (with multi-angle and high spatial resolution characteristics) are used as the main body, and a high-precision anatomical framework is constructed with CT data to clarify the tumor size, calcification, and spatial relationship with surrounding organs.

[0081] The multi-slice spiral CT images (64 rows) can generate 64 layers of images in one rotation; regarding the multi-slice spiral CT images as continuous two-dimensional slices and stacking them in the Z-axis direction to transform into three-dimensional images that can display three-dimensional forms and spatial relationships.

[0082] In each layer of CT images, manual or deep learning models are used to outline and label body tissues and tumor edges; based on the segmentation results of the same marked tissues in all layers of CT images, three-dimensional reconstruction results are generated.

[0083] Repeat the annotation and segmentation for each annotated area, and then reorganize all the volume of interest (VOI) models according to the spatial information to obtain a three-dimensional reconstruction model. Please refer to Figure 5 , Figure 5 which is a schematic diagram of a multi-slice spiral CT image in the portal venous phase, Figure 5 including the digestive system and all adjacent body tissues; please refer to Figure 6 , Figure 6 which is a partial top view of the three-dimensional reconstruction model after the three-dimensional reconstruction of all CT image layers. According to Figure 6 it can be observed that there are obvious sawteeth in the Z-axis direction.

[0084] As a preferred embodiment of the present invention, resampling is performed on the three-dimensional reconstruction model using cubic interpolation to obtain an appropriate resolution. After resampling, the voxel size of the VOI becomes 3mm×3mm×3mm.

[0085] After improving the resolution, there are still sawtooth phenomena on the surface of the three-dimensional reconstruction model. This may be due to the deviation in the delineation of tissue regions in CT images at different levels, and this deviation may be caused by the tendency of the boundary of the diseased tissue to invade outward to a certain extent under CT images at different levels.

[0086] Furthermore, filter and smooth the three-dimensional model, and reconstruct it using the triangular patch method to obtain a triangular mesh-shaped volume of interest, that is, the initial three-dimensional model is constructed. The initial three-dimensional model is in the form of a triangular patch mesh, and the triangular mesh will be simply referred to as the triangular mesh in the subsequent steps.

[0087] Please refer to Figure 7 , Figure 7 which is a schematic diagram of the initial three-dimensional model.

[0088] The model correction module 20 is used to extend the contour line of the gastrointestinal stromal tumor area in the MRI image to obtain a rendering reference line; judge the internal and external invasion trends according to the internal and external growth properties of the rendering reference line, and determine the invasion risk coefficient; according to the invasion risk coefficient, increase the number of triangular meshes in the initial three-dimensional model to obtain a mesh three-dimensional model.

[0089] Extract the invasion relationship between the location of the gastrointestinal stromal tumor in the digestive system and adjacent soft tissues according to the T1 weighted image (T1WI) of the MRI image, and adjust the initial three-dimensional model in the form of a triangular mesh.

[0090] T1-weighted images highlight the longitudinal relaxation differences of tissues through short repetition time and short echo time. The T1 signals of normal digestive organs are relatively uniform, while invasive lesions may show reduced T1 signals due to increased water content or structural damage. It should be noted that there is a T1 signal intensity at each pixel position.

[0091] First, segment the gastrointestinal stromal tumor region in the MRI image to obtain the gastrointestinal stromal tumor region. Then, use the Canny detection to obtain the boundary contour line of the digestive system attached to the gastrointestinal stromal tumor region. This contour line is a spatially continuous curve. Taking the two ends of the contour segment between the two intersection points of the gastrointestinal stromal tumor region on its attached digestive system boundary contour line as the extension starting points, and using the extension starting points as the starting points, extend the contour segment to both sides by a preset length to obtain the rendering reference line. In the embodiment of the present invention, the preset length is 30% of the length of the contour segment.

[0092] Construct a coordinate system with the direction of the line connecting the center point of the rendering reference line and the centroid of the gastrointestinal stromal tumor region as the negative y-axis direction, which is called the growth plane coordinate system.

[0093] Please refer to Figure 8 , Figure 8 which is a schematic diagram of the rendering reference line corresponding to the GIST region; Figure 8 In [reference], the contour line where the gastrointestinal stromal tumor region is collinear with the digestive system is the boundary contour line of the digestive system attached to the gastrointestinal stromal tumor region. The contour line after extending the digestive system boundary contour line along the contour of the digestive system to both sides by a preset length is the rendering reference line; Figure 8 In [reference], x is the horizontal axis of the growth plane coordinate system, and y is the vertical axis of the growth plane coordinate system.

[0094] Obtain the slope at each position on the rendering reference line. According to the slope value, the invasion trend of the contour line can be judged.

[0095] In the growth plane coordinate system, if the average slope value of the rendering reference line is negative, it belongs to endophytic growth; otherwise, it is exophytic growth. The larger the absolute value of the average slope value, the greater the invasion trend outward or inward.

[0096] Exophytic growth: When the gastrointestinal stromal tumor grows outward, the contact area between the tumor and the surrounding tissues is large, the boundary is irregular and the shape is diverse, and it will bulge out and compress the surrounding organs. There is an invasion trend from the marginal area of the stomach to the surrounding organs or soft tissues. If the invasion trend is too large, adjacent structures will be invaded, such as the manifestation of metastasis to organs such as the liver.

[0097] Endophytic growth: When the gastrointestinal stromal tumor grows inward, a clear and uniform echo will be formed at the marginal area of the stomach. The tumor usually has a regular boundary, a single shape, and a uniform density.

[0098] Therefore, when creating a three-dimensional model, there are more details in exogenous growth, and more rendering resources need to be allocated to the initial three-dimensional model reconstructed from CT images. Therefore, mainly for exogenous growth, it not only has a tendency to invade outward, but this tendency will also cause unpredictable deformation and necrosis on the side walls around the gastrointestinal stromal tumor inside the digestive system. That is, its deformation is affected by external factors. Therefore, the rendering logic of this part of the structure has always been lacking, that is, the rendering logic around the tumor has always been lacking.

[0099] By analyzing the endogenous and exogenous growth of the rendering reference line, judge the invasion tendency inside and outside to determine the invasion risk coefficient in different directions. Specifically:

[0100] Along the direction from the regional centroid of the gastrointestinal stromal tumor area to each position point on the rendering reference line, use the region growing method to expand the rendering reference line in the direction corresponding to each position point. Among them, each position point on the rendering reference line has its own corresponding extended line segment; the extended line segment is a line segment perpendicular to the tangent direction of the rendering reference line with each position point on the rendering reference line as an endpoint after region growing expansion. One endpoint of the extended line segment is a point on the rendering reference line, and the other endpoint is the boundary point of the extended area after expanding the rendering reference line using the region growing method. In the embodiment of the present invention, 20 pixel points are expanded outward at each position point on the rendering reference line.

[0101] Take the length of the extended line segment at each position point on the rendering reference line after region growing expansion as the extended thickness at each position point on the rendering reference line. In the embodiment of the present invention, the length of the line segment is the number of pixel points on the line segment.

[0102] According to the continuous TI signal intensity of each position point on the extended line segment, determine the invasion tendency vector of each extended line segment. Specifically: Take the first derivative of the TI signal intensity at continuous positions on the extended line segment, and use the average value of the first derivative corresponding to each extended line segment as the magnitude of the invasion tendency vector of the extended line segment, and take the direction of the extended line segment as the direction of the invasion tendency vector. This invasion tendency vector represents the attenuation direction and gradient of the T1 signal intensity.

[0103] According to the extended thickness of the extended line segment at each position point on the rendering reference line, determine the invasion direction. Specifically: Calculate the variance of the extended thickness at all positions on the rendering reference line, and the set of invasion tendency vectors. The more uniform the extended thickness, the more obvious the gradient, indicating that the invasion tendency of the gastrointestinal stromal tumor is smaller. On the contrary, the more irregular the contour line and the more blurred the gradient, indicating that the invasion tendency of the gastrointestinal stromal tumor is larger. Then, obtain the position corresponding to the maximum gradient when the extended thickness on the rendering reference line is the thinnest, and obtain the direction of the line connecting the centroid and this position as the invasion direction.

[0104] The angle formed by the direction of the vector sum of the invasion trend vectors of all extended line segments and the invasion direction is defined as the invasion trend transfer angle. The more consistent the directions of all the vectors in the set of invasion trend vectors are with the position of the thinnest wall thickness, the more it represents the transfer of the necrotic or cystic region towards the invasion trend direction of the gastrointestinal stromal tumor. In the embodiments of the present invention, the size range of the invasion trend transfer angle is defined as 0 - 180°. If the size of the formed angle exceeds 180°, then the size of the included angle of the angle group is defined as the size of the invasion trend transfer angle.

[0105] Combining the size of the invasion trend transfer angle and the fluctuation of the extended thickness of the extended line segment corresponding to the rendering reference line, the invasion risk coefficient is determined. Specifically: obtain the cosine value of the invasion trend transfer angle, and calculate the variance of the extended thickness of the extended line segment corresponding to the rendering reference line. The variance of the extended thickness characterizes the fluctuation of the extended thickness; the product value of the normalized value of the variance and the cosine value is used as the invasion risk coefficient.

[0106] Using the cosine function to transform the invasion trend transfer angle, this cosine value represents the transfer probability, that is, the smaller the direction angle, the more likely it is that the necrotic or cystic region has transferred towards the invasion trend direction of the gastrointestinal stromal tumor. The greater the invasion trend and the fact that the necrotic or cystic region has transferred towards the invasion trend direction of the gastrointestinal stromal tumor represent a greater invasion risk.

[0107] Furthermore, according to the invasion risk coefficient, the number of triangular meshes in the initial three-dimensional model is increased to obtain a mesh three-dimensional model, including: on the initial three-dimensional model, with the Z-axis as the reference direction, in the region corresponding to the invasion trend transfer angle, the size of the triangular meshes is reduced, and the number of triangular meshes in the initial three-dimensional model is increased. The multiple of the increased triangular meshes is a preset multiple of the invasion risk coefficient, and a network three-dimensional model is obtained.

[0108] The reason for increasing the number of triangular meshes is that a higher density of triangular meshes can reserve a larger rendering space. In the embodiments of the present invention, the value of the preset multiple is 10. In other embodiments, the implementer can adjust this value according to the actual situation. For example, when the preset multiple is 10, that is, for every increase of 0.1 in the invasion risk coefficient, the size of the triangular meshes is reduced by 1 times, and the corresponding multiple of triangular meshes is increased, which can also be understood as the number of triangular meshes is increased by 1 times.

[0109] The texture matching module 30 is used to perform regional segmentation on the endoscopic image to obtain multiple pixels, and obtain the texture features within each pixel; perform feature matching on the texture features corresponding to the adjacent triangular meshes in the mesh three-dimensional model and the texture features corresponding to the pixels to obtain the best matching pixel corresponding to each triangular mesh.

[0110] Combining the texture matching module and the subsequent texture rendering module, the internal texture is rendered according to the corrected mesh three-dimensional model.

[0111] Map each vertex of each triangular mesh in the grid 3D model to the 2D texture space of the CT image to generate UV coordinates. Preferably, it can be achieved by a planar parameterization method, such as by the method of least squares conformal mapping, to ensure that each triangular mesh has a corresponding area in the texture space and reduce deformation. In the rendering pipeline, according to the UV coordinates of the vertices of the triangular mesh, texture sampling is performed on the 2D CT image area corresponding to each triangular mesh, and then based on the UV coordinates of the three vertices in each triangular mesh in the 3D model, texture rendering is performed inside the triangular mesh to ensure correct filling of the texture inside the triangle.

[0112] The texture filled with the UV coordinates of each triangular mesh is based on the texture at the corresponding position in the texture space of the 2D CT image. However, in fact, there may be more deformation and invasion features around the exophytic tumor. The embodiments of the present invention further render by means of the texture features of the endoscopic image.

[0113] For several endoscopic images of gastrointestinal stromal tumors of the same patient, use the matlab tool to perform regional segmentation on the endoscopic image and divide it into multiple pixels. The matlab tool is such as the RADIOMICS toolbox. In the embodiments of the present invention, the image is divided into several pixels of size 50*50. Then extract all first-order statistical features, corner features, edge features, color features, etc. within the pixel range; these features are all 3D rendering sub-resources. The first-order statistical features include: mean, variance, gradient, contrast, etc.

[0114] Obtain the texture features of each pixel within each pixel range: gray-level co-occurrence matrix features and gray-level region size matrix features. The texture features are all in matrix format, including all texture details of gastrointestinal stromal tumors and adjacent digestive tract tissues. The texture detail data are all statistics obtained from the endoscopic image and mostly do not contain position information.

[0115] The 3D model at this time is a grid 3D model reconstructed from the CT image and corrected by the MRI image for the triangular mesh. The perspectives of the CT image and the MRI image can be close to the same, while the perspective of the endoscopic image is not the same as the former two, so its position information is not referenceable. Therefore, it is necessary to match the texture feature statistics in the endoscopic image with different grid positions in the 3D model to achieve the purpose of rendering. The texture features within each pixel range may be the texture rendering that should be inserted at different triangular mesh positions.

[0116] Obtain the triangular meshes within the neighborhood range of each triangular mesh in the grid 3D model as neighborhood triangular meshes. There are 13 neighborhood triangular meshes within the neighborhood range of each triangular mesh. Except for the target triangular mesh itself, there are 3 edge-adjacent meshes and 9 corner-adjacent meshes. Please refer toFigure 10 , Figure 10 Schematic diagram of a triangular mesh centered on Figure 10 and its corresponding neighborhood triangular mesh Figure 10 Among them, the edge-adjacent meshes of triangular mesh a1 centered on a2~a4, and the corner-adjacent meshes are a5~a13.

[0117] Construct a pixel texture matrix corresponding to each pixel from the texture features of each pixel in the endoscopic image; each texture feature corresponds to a pixel texture matrix. For example, for pixel i, the pixel texture matrix corresponding to pixel i is constructed, including: the gray-level co-occurrence matrix and the gray-level region size matrix of pixel i.

[0118] Taking any triangular mesh as the target triangular mesh, construct a basic texture matrix from the texture features of all neighborhood triangular meshes of the target triangular mesh; each texture feature of each triangular mesh corresponds to a basic texture matrix. For example, for triangular mesh z, the basic texture matrix corresponding to triangular mesh z is constructed, including: the gray-level co-occurrence matrix and the gray-level region size matrix of triangular mesh z.

[0119] Perform interpolation processing on the basic texture matrix, and align the interpolated basic texture matrix with the pixel texture matrix.

[0120] As a preferred embodiment of the present invention, the matrix of the basic texture features needs to be interpolated with blank cells and then aligned with the matrix of the endoscopic image texture features. The method of interpolating blank cells is to insert 0 values at intervals along the horizontal and vertical axes.

[0121] As another preferred embodiment of the present invention, the gray-level segmentation of the basic texture matrix is the same as that of the pixel texture matrix, and the horizontal axis length and vertical axis length of the two matrices are the same.

[0122] Calculate the difference matrix between the interpolated basic texture matrix corresponding to the same texture feature and the pixel texture matrix of each pixel, and obtain the difference matrix of each texture feature of each pixel.

[0123] ; ; where is the difference matrix corresponding to the gray-level co-occurrence matrix of the pixel; is the difference matrix corresponding to the gray-level region size matrix of the pixel; is the gray-level co-occurrence matrix of the pixel; is the gray-level co-occurrence matrix of the triangular mesh; is the gray-level region size matrix of the pixel; is the gray-level region size matrix of the triangular mesh.

[0124] Analyze the difference matrix of each texture feature of each pixel respectively, and determine the pixel that best matches the target triangular mesh as the best matching pixel.

[0125] Calculate the contrast D, energy E, and entropy value S of the difference matrix corresponding to the gray-level co-occurrence matrix of the pixel.

[0126] Calculate the distribution density M of all elements in the gray-level co-occurrence matrix of the pixel and the average distance between the elements at all positions and the central element of the matrix. The smaller the matrix distribution density, the more concentrated the gray levels and the smaller the gray-level difference in the difference matrix.

[0127] First, determine the fine-line interpretability based on the difference matrix corresponding to the gray-level co-occurrence matrix.

[0128] Fine-line interpretability The calculation formula is: .

[0129] High contrast + low energy + high entropy may represent fine textures, such as the edges of tumors, while the larger the ratio, the more concentrated the gray levels in the difference matrix and the more fine textures with small gray-level differences, that is, the better the texture feature of the pixel interprets the fine lines in the neighborhood range of the target triangular mesh.

[0130] Then, determine the texture contrast interpretability based on the difference matrix corresponding to the characteristics of the gray-level region size matrix.

[0131] Texture contrast interpretability The calculation formula is: ; where C1 is the high gray-level repetition in the large region; C2 is the high gray-level repetition in the small region. Among them, the high gray-level repetition in the large region and the high gray-level repetition in the small region are well-known indicators of the gray-level region size matrix.

[0132] The low gray-level repetition in the small region represents the frequency of occurrence of homogeneous regions with relatively low gray levels and small sizes, such as textures of microtumors, nodules, necrotic tissues, etc.; the high gray-level repetition in the large region represents the frequency of occurrence of homogeneous regions with relatively high gray levels and large sizes in the image, such as the basic texture of the digestive inner wall; the smaller the difference between the two, the better the texture feature of the pixel interprets the low gray-level texture contrast in the neighborhood range of the target triangular mesh.

[0133] Combine the fine-line interpretability and the texture contrast interpretability to determine the matching degree of each pixel corresponding to the target triangular mesh. Among them, the fine-line interpretability is positively correlated with the matching degree; the texture contrast interpretability is negatively correlated with the matching degree.

[0134] In some embodiments, the matching degree of each pixel corresponding to the target triangular mesh is the normalized value of the ratio of the fine-line interpretability and the texture contrast interpretability of each pixel and the target triangular mesh. The fine-line interpretability is smaller, and the texture contrast interpretability The higher it is, the higher the matching degree, and the larger the ratio of the explanatory power of fine line understanding to the explanatory power of texture contrast, indicating that the texture feature group of this pixel has stronger explanatory power for the information in the target triangular mesh and higher matching degree. In the embodiments of the present invention, the normalization of the ratio of the explanatory power of fine line understanding to the explanatory power of texture contrast can be achieved through linear normalization.

[0135] The pixel corresponding to the maximum matching degree is used as the best matching pixel of the target triangular mesh.

[0136] The texture rendering module 40 is configured to perform texture rendering on the mesh three-dimensional model according to the best matching pixel to obtain a three-dimensional positioning model of gastrointestinal stromal tumor.

[0137] According to the matching relationship, the texture within each triangular mesh is rendered. Specifically: all texture feature data and three-dimensional rendering sub-resources within the range of the best matching pixel corresponding to the target triangular mesh are input into the target triangular mesh for texture rendering, and the triangular mesh after texture rendering is smoothed to obtain a three-dimensional model of gastrointestinal stromal tumor as the three-dimensional positioning model.

[0138] That is, all texture feature data within the range of the pixel corresponding to each triangular mesh and all three-dimensional rendering sub-resources are input into the triangular mesh for texture rendering; after the rendering is completed, smoothing is performed between the meshes.

[0139] Since the density of the triangular mesh is corrected, the gastrointestinal stromal tumor and its surrounding tissues can obtain more refined texture rendering, and finally a three-dimensional reconstruction result is obtained. The final three-dimensional model is the three-dimensional positioning model.

[0140] Please refer to Figure 9 , which shows the flowchart of the steps of a three-dimensional positioning and surgical navigation method for gastrointestinal stromal tumor with multimodal image fusion provided by an embodiment of the present invention. The method includes:

[0141] Step S100, obtaining CT images, MRI images, and endoscopic images of gastrointestinal stromal tumor; generating an initial three-dimensional model from multi-slice CT images; wherein, the initial three-dimensional model is in the form of a triangular patch mesh.

[0142] Step S200, extending the contour line of the gastrointestinal stromal tumor area in the MRI image to obtain a rendering reference line; judging the internal and external invasion trend according to the internal and external growth of the rendering reference line, determining the invasion risk coefficient; increasing the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a mesh three-dimensional model.

[0143] Step S300: The endoscopic image is divided into multiple pixels in regions, and the texture features within each pixel are obtained; the texture features corresponding to adjacent triangular meshes in the grid three-dimensional model are feature-matched with the texture features corresponding to the pixels to obtain the best-matching pixel for each triangular mesh.

[0144] Step S400: Based on the best-matching pixels, texture rendering is performed on the grid three-dimensional model to obtain a three-dimensional localization model of gastrointestinal stromal tumor.

[0145] Optionally, the transmission medium can be a wired link, such as but not limited to, coaxial cable, optical fiber, and digital subscriber line, etc., or a wireless link, such as but not limited to, Wireless Fidelity (WIFI), Bluetooth, and mobile device network, etc.

[0146] It should be noted that: for the device provided in the above embodiment, only the division of the above functional modules is used for illustration. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above.

[0147] A computer device provided by an embodiment of the present invention. Exemplarily, the computer device includes: a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the computer device can execute any of the multi-modal image fusion-based three-dimensional localization and surgical navigation systems for gastrointestinal stromal tumors introduced above.

[0148] In addition, an embodiment of the present invention also protects a device, which may include a memory and a processor. Among them, an executable program code is stored in the memory, and the processor is used to call and execute the executable program code to execute the multi-modal image fusion-based three-dimensional localization and surgical navigation system for gastrointestinal stromal tumors provided by an embodiment of the present invention.

[0149] An embodiment of the present invention can divide the functions of the device according to the above method examples. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0150] In the case of dividing each module according to each corresponding function, the device may further include a signal uploading module, a determination module, an adjustment module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be cited in the function description of the corresponding functional module, and will not be repeated here.

[0151] It should be understood that the device provided in the embodiments of the present invention is used to execute the above-mentioned three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion, so the same effects as the above-mentioned implementation methods can be achieved.

[0152] In the case of adopting integrated units, the device may include a processing module and a storage module. Among them, when the device is applied to a device, the processing module can be used to control and manage the actions of the device. The storage module can be used to support the device to execute mutual program codes, etc. Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure of the present invention. The processor can also be a combination that realizes computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory.

[0153] In addition, the device provided in the embodiments of the present invention can specifically be a chip, a component or a module. The chip may include a connected processor and a memory; among them, the memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute the three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion provided in the above embodiments.

[0154] The embodiments of the present invention also provide a computer-readable storage medium, in which computer program codes are stored. When the computer program codes run on a computer, the computer is enabled to execute the above-related method steps to implement the three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion provided in the above embodiments.

[0155] The embodiments of the present invention also provide a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above-related steps to implement the three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion provided in the above embodiments.

[0156] Among them, the device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here. Through the description of the above embodiments, those skilled in the art can understand that for the convenience and simplicity of description, only the above-mentioned division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed device and method can be implemented in other ways.

[0157] The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0158] It should also be noted that in this article, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or terminal device including the said element.

[0159] It should be noted that the above-mentioned sequence of the embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0160] Each embodiment in this specification is described in a progressive manner. The same or similar parts between each embodiment can be referred to each other, and the key points of each embodiment are the differences from other embodiments.

[0161] The above content is only a specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention.

Claims

1. A three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion, characterized in that, The system includes the following modules: A model initial construction module, which is used to obtain CT images, MRI images and endoscopic images of gastrointestinal stromal tumors; generate an initial three-dimensional model from multi-layer CT images; wherein, the initial three-dimensional model is in the form of a triangular patch mesh; A model correction module, which is used to extend the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judge the internal and external invasion trend according to the internal and external growth properties of the rendering reference line, and determine the invasion risk coefficient; according to the invasion risk coefficient, increase the number of triangular meshes in the initial three-dimensional model to obtain a mesh three-dimensional model; Among them, the method for obtaining the invasion risk coefficient is as follows: extend the rendering reference line along the direction from the regional centroid of the gastrointestinal stromal tumor region to each position point on the rendering reference line, wherein each position point on the rendering reference line has its own corresponding extended line segment; determine the invasion trend vector of each extended line segment according to the continuous TI signal intensity of each position point on the extended line segment; determine the invasion direction according to the extended thickness of the extended line segment of each position point on the rendering reference line; the included angle formed by the direction of the vector sum of the invasion trend vectors of all extended line segments and the invasion direction is used as the invasion trend transfer included angle; combine the size of the invasion trend transfer included angle and the fluctuation of the extended thickness of the extended line segment corresponding to the rendering reference line to determine the invasion risk coefficient; A texture matching module, which is used to divide the endoscopic image into multiple pixels by region, and obtain the texture features in each pixel; perform feature matching on the texture features corresponding to adjacent triangular meshes in the mesh three-dimensional model and the texture features corresponding to the pixels to obtain the best matching pixel corresponding to each triangular mesh; A texture rendering module, which is used to perform texture rendering on the mesh three-dimensional model according to the best matching pixel to obtain a three-dimensional positioning model of the gastrointestinal stromal tumor.

2. The three-dimensional localization and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, The generation of the initial three-dimensional model from multi-layer CT images includes: Stack multi-layer CT images in the Z-axis direction to generate a three-dimensional reconstruction model; perform smooth reconstruction on the three-dimensional model to obtain an initial three-dimensional model; wherein, the initial three-dimensional model is in the form of a triangular patch mesh.

3. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, The extension of the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line includes: Based on the MRI image, segment the gastrointestinal stromal tumor region; Use the canny detection algorithm to obtain the boundary contour line of the digestive system attached to the gastrointestinal stromal tumor region; Take the two ends of the contour line segment between the two intersection points of the gastrointestinal stromal tumor region on its attached digestive system boundary contour line as the extension starting points, and extend the contour line segment to both sides by a preset length from the extension starting points to obtain a rendering reference line.

4. The three-dimensional localization and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, The increase in the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a mesh three-dimensional model includes: Increase the number of triangular meshes in the initial three-dimensional model in the region corresponding to the invasion trend transfer included angle, and the multiple of the increased triangular meshes is the preset multiple of the invasion risk coefficient to obtain a network three-dimensional model.

5. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, The texture features in each pixel include: gray level co-occurrence matrix features and gray level region size matrix features.

6. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 5, characterized in that, Performing feature matching on the texture features corresponding to adjacent triangular meshes in the grid 3D model and the texture features corresponding to pixels, and obtaining the best-matching pixel corresponding to each triangular mesh, includes: Obtaining the triangular meshes within the neighborhood range of each triangular mesh in the grid 3D model as the neighborhood triangular meshes; Constructing a pixel texture matrix corresponding to each pixel from the texture features of each pixel in the endoscopic image; each texture feature corresponds to a pixel texture matrix; Taking any triangular mesh as the target triangular mesh, and constructing a basic texture matrix from the texture features of all the neighborhood triangular meshes of the target triangular mesh; each texture feature of each triangular mesh corresponds to a basic texture matrix; Performing interpolation processing on the basic texture matrix, and aligning the interpolated basic texture matrix with the pixel texture matrix; Calculating the difference matrix between the interpolated basic texture matrix corresponding to the same texture feature and the pixel texture matrix of each pixel, and obtaining the difference matrix of each texture feature of each pixel; Analyzing the difference matrix of each texture feature of each pixel respectively, and determining the pixel that best matches the target triangular mesh as the best-matching pixel.

7. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 6, characterized in that, The step of analyzing the difference matrix of each texture feature of each pixel respectively, and determining the pixel that best matches the target triangular mesh as the best-matching pixel, includes: Determining the fine texture interpretability according to the difference matrix corresponding to the gray-level co-occurrence matrix; Determining the texture contrast interpretability according to the difference matrix corresponding to the gray-level region size matrix feature; Combining the fine texture interpretability and the texture contrast interpretability to determine the matching degree between each pixel and the target triangular mesh; Taking the pixel corresponding to the maximum matching degree as the best-matching pixel of the target triangular mesh.

8. The three-dimensional localization and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 6, wherein Performing texture rendering on the grid 3D model according to the best-matching pixel, and obtaining a 3D localization model of gastrointestinal stromal tumor, includes: Inputting all the texture feature data and 3D rendering sub-resources within the range of the best-matching pixel corresponding to the target triangular mesh into the target triangular mesh for texture rendering, and performing smoothing processing on the texture-rendered triangular mesh to obtain a 3D model of gastrointestinal stromal tumor as the 3D localization model.

9. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, Performing texture rendering on the grid 3D model according to the best-matching pixel further includes: Mapping each vertex of each triangular mesh in the grid 3D model to the 2D texture space of the CT image to generate UV coordinates; Performing texture sampling on the 2D CT image region corresponding to each triangular mesh according to the UV coordinates of the vertices of the triangular mesh, and performing texture rendering inside the triangular mesh based on the UV coordinates of the three vertices in each triangular mesh of the grid 3D model.

Images