Multi-modal image fusion gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system

Abstract

The invention relates to the technical field of three-dimensional modeling, in particular to a gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system based on multi-modal image fusion. The system comprises a model initial building module used for obtaining a CT image, an MRI image and an endoscope image of gastrointestinal stromal tumor; generating an initial three-dimensional model by the multi-layer CT image; the model correction module is used for increasing the number of triangular meshes in the initial three-dimensional model according to analysis of a gastrointestinal stromal tumor area in the MRI image to obtain a mesh three-dimensional model; the texture matching module is used for determining an optimal matching pixel corresponding to each triangular mesh in the mesh three-dimensional model, and the optimal matching pixel belongs to the endoscope image; and the texture rendering module is used for performing texture rendering on the grid three-dimensional model according to the optimal matching pixels to obtain a three-dimensional positioning model of the gastrointestinal stromal tumor. According to the method, the three-dimensional model is reconstructed in combination with various types of images, and the detail rendering condition of the gastrointestinal stromal tumor model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and particularly to a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion. Background Art

[0002] Gastrointestinal stromal tumors (GIST) originate from the mesenchymal tissue of the gastrointestinal wall, and the tumors can grow into the lumen, intramural or extraluminal. The precise diagnosis and treatment of gastrointestinal stromal tumors rely on the three-dimensional positioning of the tumor location, morphology, and relationship with surrounding organs. Each traditional single imaging technique has limitations. For example, CT images have low sensitivity to rectal lesions, while MRI images have more advantages in three-dimensional imaging and tissue contrast. Multimodal image fusion can improve the positioning accuracy by integrating complementary information from different modalities, such as integrating complementary information of CT images, MRI images, PET images, endoscopic images, etc.

[0003] Whether the tumor has metastasis and whether it infiltrates the surrounding tissues are important indicators for judging benign and malignant. In traditional three-dimensional reconstruction techniques, only the size, morphology, and anatomical relationship with organs of the tumor are concerned, and the details of gastrointestinal stromal tumors are insufficiently rendered. For example, the invasion of adjacent regions and organs by the tumor results in a large number of missing texture details in the three-dimensional model when establishing the three-dimensional model, making it difficult to determine the invasion boundary of gastrointestinal stromal tumors in the three-dimensional model. Summary of the Invention

[0004] In order to solve the technical problem of insufficient detail rendering of gastrointestinal stromal tumors when establishing a three-dimensional model using traditional three-dimensional reconstruction techniques, the purpose of the present invention is to provide a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion. The specific technical solutions adopted are as follows: In the first aspect, an embodiment of the present invention provides a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion. The system includes the following modules: A model initial construction module, which is used to obtain CT images, MRI images, and endoscopic images of gastrointestinal stromal tumors; generate an initial three-dimensional model from multi-layer CT images; wherein, the initial three-dimensional model is in the form of a triangular patch mesh. A model correction module, which is used to extend the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judge the internal and external invasion trends according to the internal and external growth properties of the rendering reference line, and determine the invasion risk coefficient; increase the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a mesh three-dimensional model. A texture matching module for segmenting endoscopic images into multiple pixels, obtaining the texture features within each pixel; performing feature matching between the texture features corresponding to adjacent triangular meshes in the grid three-dimensional model and the texture features corresponding to the pixels to obtain the best-matching pixel corresponding to each triangular mesh; A texture rendering module for texture-rendering the grid three-dimensional model according to the best-matching pixels to obtain a three-dimensional positioning model of gastrointestinal stromal tumor.

[0005] Further, the generating of the initial three-dimensional model from the multi-slice CT images includes: Stacking the multi-slice CT images in the Z-axis direction to generate a three-dimensional reconstruction model; performing smooth reconstruction on the three-dimensional model to obtain an initial three-dimensional model; wherein, the initial three-dimensional model is in the form of a triangular patch grid.

[0006] Further, the extending of the contour line of the gastrointestinal stromal tumor region in the extended MRI image to obtain a rendering reference line includes: Segmenting the gastrointestinal stromal tumor region based on the MRI image; Using the canny detection algorithm to obtain the boundary contour line of the digestive system to which the gastrointestinal stromal tumor region adheres; Taking the two ends of the contour line segment between the two intersection points of the gastrointestinal stromal tumor region on the boundary contour line of the digestive system to which it adheres as the extension starting points, and taking the extension starting points as the starting points, extending the contour line segment to both sides by a preset length to obtain a rendering reference line.

[0007] Further, the judging of the internal and external invasion trend and the determination of the invasion risk coefficient according to the internal and external growth properties of the rendering reference line include: Extending the rendering reference line along the directions from the regional centroid of the gastrointestinal stromal tumor region to each position point on the rendering reference line, wherein each position point on the rendering reference line has its own corresponding extended line segment; Determining the invasion trend vector of each extended line segment according to the continuous TI signal intensity of each position point on the extended line segment; Determining the invasion direction according to the extended thickness of the extended line segment of each position point on the rendering reference line; Taking the included angle formed by the direction of the vector sum of the invasion trend vectors of all extended line segments and the invasion direction as the invasion trend transfer included angle; Combining the magnitude of the invasion trend transfer included angle and the fluctuation of the extended thickness of the extended line segment corresponding to the rendering reference line to determine the invasion risk coefficient.

[0008] Further, the increasing of the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a grid three-dimensional model includes: Increase the number of triangular meshes in the initial 3D model in the area corresponding to the infringement trend transfer angle. The multiple of the increased triangular meshes is the preset multiple of the infringement risk coefficient to obtain the network 3D model.

[0009] Furthermore, the texture features within each pixel include: gray-level co-occurrence matrix features and gray-level region size matrix features.

[0010] Furthermore, the step of performing feature matching on the texture features corresponding to adjacent triangular meshes in the grid 3D model and the texture features corresponding to pixels to obtain the best-matched pixel for each triangular mesh includes: Obtain the triangular meshes within the neighborhood range of each triangular mesh in the grid 3D model as neighborhood triangular meshes; Construct a pixel texture matrix corresponding to each pixel from the texture features of each pixel in the endoscopic image; each type of texture feature corresponds to a pixel texture matrix; Take any triangular mesh as the target triangular mesh, and construct a basic texture matrix from the texture features of all the neighborhood triangular meshes of the target triangular mesh; each type of texture feature of each triangular mesh corresponds to a basic texture matrix; Perform interpolation processing on the basic texture matrix, and align the interpolated basic texture matrix with the pixel texture matrix; Calculate the difference matrix between the interpolated basic texture matrix corresponding to the same type of texture feature and the pixel texture matrix of each pixel to obtain the difference matrix of each type of texture feature of each pixel; Analyze the difference matrix of each type of texture feature of each pixel respectively to determine the pixel that best matches the target triangular mesh as the best-matched pixel.

[0011] Furthermore, the step of analyzing the difference matrix of each type of texture feature of each pixel respectively to determine the pixel that best matches the target triangular mesh as the best-matched pixel includes: Determine the fine texture interpretability according to the difference matrix corresponding to the gray-level co-occurrence matrix; Determine the texture contrast interpretability according to the difference matrix corresponding to the gray-level region size matrix feature; Combine the fine texture interpretability and the texture contrast interpretability to determine the matching degree between each pixel and the target triangular mesh; Take the pixel corresponding to the maximum matching degree as the best-matched pixel of the target triangular mesh.

[0012] Furthermore, the step of performing texture rendering on the grid 3D model according to the best-matched pixel to obtain the 3D localization model of gastrointestinal stromal tumor includes: All the texture feature data and 3D rendering sub-resources within the range of the best-matching pixels corresponding to the target triangular mesh are input into the target triangular mesh for texture rendering, and the triangular mesh after texture rendering is smoothed to obtain a 3D model of gastrointestinal stromal tumor as the 3D positioning model.

[0013] Further, the texture rendering of the grid 3D model according to the best-matching pixels further includes: Map each vertex of each triangular mesh in the grid 3D model to the 2D texture space of the CT image to generate UV coordinates; Perform texture sampling on the 2D CT image area corresponding to each triangular mesh according to the UV coordinates of the vertices of the triangular mesh, and perform texture rendering on the interior of the triangular mesh based on the UV coordinates of the three vertices in each triangular mesh of the grid 3D model.

[0014] In a second aspect, a method for 3D positioning and surgical navigation of gastrointestinal stromal tumor with multimodal image fusion is provided. The method includes the following steps: Obtain the CT image, MRI image, and endoscopic image of the gastrointestinal stromal tumor; generate an initial 3D model from the multi-layer CT image; wherein, the initial 3D model is in the form of a triangular patch mesh; Extend the contour line of the gastrointestinal stromal tumor area in the MRI image to obtain a rendering reference line; judge the internal and external invasion trend according to the internal and external growth of the rendering reference line, and determine the invasion risk coefficient; increase the number of triangular meshes in the initial 3D model according to the invasion risk coefficient to obtain a grid 3D model; Perform regional segmentation on the endoscopic image to obtain multiple pixels, and obtain the texture features within each pixel; perform feature matching on the texture features corresponding to the adjacent triangular meshes in the grid 3D model and the texture features corresponding to the pixels to obtain the best-matching pixel corresponding to each triangular mesh; Perform texture rendering on the grid 3D model according to the best-matching pixels to obtain a 3D positioning model of the gastrointestinal stromal tumor.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the embodiments of all possible implementations in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer program product, which includes: computer program code. When the computer program code runs on a computer, the computer is caused to execute the method in the first aspect or any possible implementation manner of the first aspect.

[0017] Fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the embodiments of all possible implementations of the first aspect.

[0018] The embodiments of the present invention have at least the following beneficial effects: In this application, an initial three-dimensional model is reconstructed from multi-slice CT images, and then the exophytic and endophytic growth types of gastrointestinal stromal tumors are extracted using MRI images. The density of the triangular mesh is corrected for areas with an invasive tendency to reserve and provide a larger rendering space. Further, by extracting the texture features within the pixels of the endoscopic images, the corrected triangular mesh is matched, and matching relationships are constructed around different texture features. According to the matching relationships, the best-matched pixels corresponding to each triangular mesh are obtained, and different rendering sub-resources are implanted in combination with the best-matched pixels to perform texture rendering on the mesh three-dimensional model, enabling more refined texture rendering of the gastrointestinal stromal tumor and its surrounding tissues, expressing the possible invasive tendency more completely, and improving the detail rendering of the gastrointestinal stromal tumor model by reconstructing the three-dimensional model with multiple types of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0020] Figure 1 System block diagram of a three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion provided by an embodiment of the present invention; Figure 2 Schematic diagram of an endoscopic image of a gastrointestinal stromal tumor provided by an embodiment of the present invention; Figure 3 Schematic diagram of a CT image of a gastrointestinal stromal tumor provided by an embodiment of the present invention; Figure 4 Schematic diagram of an MRI image of a gastrointestinal stromal tumor provided by an embodiment of the present invention; Figure 5 Schematic diagram of a multi-slice spiral CT image in the portal venous phase provided by an embodiment of the present invention; Figure 6 Partial top view of a three-dimensional reconstruction model after three-dimensional reconstruction of all layers of CT images provided by an embodiment of the present invention; Figure 7 Schematic diagram of an initial three-dimensional model provided by an embodiment of the present invention; Figure 8 Schematic diagram of the rendering reference line corresponding to the GIST region provided by an embodiment of the present invention; Figure 9 Method flowchart of the three-dimensional positioning and surgical navigation method for gastrointestinal stromal tumors with multimodal image fusion provided by an embodiment of the present invention; Figure 10 Schematic diagram of the triangular mesh of the center and its corresponding neighborhood triangular mesh provided by an embodiment of the present invention. Detailed implementation manners

[0021] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation manners, structures, features and effects of the three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion proposed according to the present invention.

[0022] In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means "or". For example, A / B can represent A or B. The "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "a plurality" means two or more than two.

[0024] Hereinafter, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as implying or indicating relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0026] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. Those skilled in the art know that with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0027] The following specifically describes the specific solution of the three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion provided by the present invention in conjunction with the accompanying drawings.

[0028] See also Figure 1 , which shows a system block diagram of a gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system with multimodal image fusion provided by an embodiment of the present invention, the system includes the following modules: The model initialization module 10 is used to obtain CT images, MRI images and endoscopic images of gastrointestinal stromal tumors; generate an initial three-dimensional model from multi-slice CT images; wherein the initial three-dimensional model is in the form of a triangular mesh.

[0029] Endoscopic imaging: It can directly observe submucosal lesions and clarify the origin layer of the tumor. For example, GIST mostly originates from the muscularis propria. It can also be used to evaluate the relationship between the tumor and surrounding tissues. However, due to the limited field of view, it is difficult to observe and evaluate the overall picture of the tumor.

[0030] CT imaging (Computed Tomography): It has fast scanning speed, high spatial resolution, supports multi-angle observation and three-dimensional reconstruction, and is suitable for surgical planning. However, the soft tissue contrast is low, and it is difficult to distinguish boundary information.

[0031] Magnetic Resonance Imaging (MRI): It is better than CT imaging in showing the invasion range of tumor adjacent structures, but has low spatial resolution and is easily disturbed by intestinal peristalsis artifacts, which affects image quality and easily misses tiny lesions. Magnetic Resonance Imaging is referred to as MRI imaging.

[0032] First, CT images, MRI images, and endoscopic images of gastrointestinal stromal tumors were obtained. It should be noted that the CT images were multi-slice spiral CT images (64 rows).

[0033] Among them, CT and MRI examinations should be completed in similar time periods as much as possible, using unified respiratory gating technology, such as breath-hold command synchronization, to reduce the impact of intestinal peristalsis and organ displacement on MRI.

[0034] CT imaging, MRI imaging and endoscopic imaging each have their own advantages and disadvantages. In order to construct a three-dimensional model more accurately, it is necessary to combine the advantages of different imaging methods to perform three-dimensional reconstruction and locate adjacent structures, and solve the problem of difficulty in data registration caused by differences in phase and resolution of different imaging devices.

[0035] See also Figure 2 , Figure 2 is a schematic diagram of an endoscopic image of a gastrointestinal stromal tumor; see Figure 3 , Figure 3 Schematic diagram of CT images of gastrointestinal stromal tumors; see Figure 4 , Figure 4 Schematic diagram of MRI images of gastrointestinal stromal tumors.

[0036] Gastrointestinal stromal tumor (GIST) is the most common mesenchymal tumor in the gastrointestinal tract. It independently originates from the gastrointestinal wall and has the characteristics of non-directional differentiation. It can occur in any part of the digestive tract.

[0037] During three-dimensional reconstruction, taking the multi-slice spiral CT images of GIST (with the characteristics of multi-angle and high spatial resolution) as the main body, a high-precision anatomical framework is constructed with CT data to clarify the tumor size, calcification, and spatial relationship with surrounding organs.

[0038] The multi-slice spiral CT images (64-slice) can generate 64 layers of images in one rotation; regarding the multi-slice spiral CT images as continuous two-dimensional slices and stacking them in the Z-axis direction, they are transformed into three-dimensional images that can display the three-dimensional shape and spatial relationship.

[0039] In each layer of CT images, the body tissues and tumor margins are outlined and labeled using artificial or deep learning models; based on the segmentation results of the same labeled tissues in all layers of CT images, a three-dimensional reconstruction result is generated.

[0040] The annotation and segmentation are repeated for each labeled area, and then all volume of interest (VOI) models are recombined according to the spatial information to obtain a three-dimensional reconstruction model. Please refer to Figure 5 , Figure 5 which is a schematic diagram of the multi-slice spiral CT image in the portal venous phase, Figure 5 including the digestive system and all adjacent body tissues; please refer to Figure 6 , Figure 6 which is a partial top view of the three-dimensional reconstruction model after three-dimensional reconstruction of all layers of CT images. According to Figure 6 it can be observed that there are obvious sawteeth in the Z-axis direction.

[0041] As a preferred embodiment of the present invention, the three-dimensional reconstruction model is resampled using cubic interpolation to obtain an appropriate resolution. After resampling, the voxel size of the VOI becomes 3mm×3mm×3mm.

[0042] After improving the resolution, there are still sawtooth phenomena on the surface of the three-dimensional reconstruction model. This may be due to the deviation in the delineation of tissue regions in different layers of CT images, and this deviation may be caused by the tendency of the boundary of the diseased tissue to invade outward to a certain extent under different layers of CT images.

[0043] Furthermore, the three-dimensional model is filtered and smoothed, and reconstructed using the triangular patch method to obtain a triangular mesh-like volume of interest, that is, the initial three-dimensional model is constructed. The initial three-dimensional model is in the form of a triangular patch mesh, and the triangular mesh will be simply referred to as the triangular mesh in the subsequent steps.

[0044] Please refer to Figure 7 ,Figure 7 It is a schematic diagram of the initial three-dimensional model.

[0045] The model correction module 20 is used to extend the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judge the internal and external invasion trends according to the internal and external growth properties of the rendering reference line, and determine the invasion risk coefficient; increase the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a mesh three-dimensional model.

[0046] According to the T1-weighted image (T1WI) of the MRI image, extract the invasion relationship between the location of the gastrointestinal stromal tumor in the digestive system and the adjacent soft tissues, and adjust the initial three-dimensional model in the form of triangular meshes.

[0047] The T1-weighted image highlights the longitudinal relaxation differences of tissues through short repetition time and short echo time. The T1 signals of normal digestive organs are relatively uniform, while invasive lesions may cause a decrease in T1 signals due to increased water content or structural damage. It should be noted that there is a T1 signal intensity at each pixel position.

[0048] First, segment the gastrointestinal stromal tumor region in the MRI image to segment out the gastrointestinal stromal tumor region, and then use canny detection to obtain the boundary contour line of the digestive system attached to the gastrointestinal stromal tumor region. This contour line is a spatially continuous curve; take the two ends of the contour line segment between the two intersection points of the gastrointestinal stromal tumor region on its attached digestive system boundary contour line as the extension starting points, and extend this contour line segment to both sides by a preset length respectively with the extension starting points as the starting points to obtain a rendering reference line. In the embodiment of the present invention, the preset length is 30% of the length of the contour line segment.

[0049] Construct a coordinate system with the direction of the line connecting the center point of the rendering reference line and the centroid of the gastrointestinal stromal tumor region as the negative y-axis direction, which is called the growth plane coordinate system.

[0050] Please refer to Figure 8 , Figure 8 It is a schematic diagram of the rendering reference line corresponding to the GIST region; Figure 8 The contour line collinear with the digestive system in the gastrointestinal stromal tumor region is the boundary contour line of the digestive system attached to the gastrointestinal stromal tumor region, and the contour line after extending the digestive system boundary contour line along the contour of the digestive system to both sides by a preset length is the rendering reference line; Figure 8 In [reference], x is the horizontal axis of the growth plane coordinate system, and y is the vertical axis of the growth plane coordinate system.

[0051] Obtain the slope at each position on the rendering reference line. According to the slope value, the invasion trend of the contour line can be judged.

[0052] In the growth plane coordinate system, if the average slope value of the rendering reference line is negative, it belongs to endogenous growth; otherwise, it is exogenous growth. The larger the absolute value of the average slope value, the greater the tendency to invade outward or inward.

[0053] Exogenous growth: When a gastrointestinal stromal tumor grows outward, the contact area between the tumor and the surrounding tissues is relatively large, the boundary is irregular and the shape is diverse, and it will bulge out and compress the surrounding organs. There is a tendency for the marginal area of the stomach to invade the surrounding organs or soft tissues. If the invasion tendency is too large, adjacent structures will be invaded, such as the manifestation of metastasis to organs such as the liver.

[0054] Endogenous growth: When a gastrointestinal stromal tumor grows inward, a uniform echo with a clear boundary will be formed at the marginal area of the stomach. The tumor usually has a regular boundary, a single shape, and a uniform density.

[0055] Therefore, when creating a three-dimensional model, there are more details in exogenous growth, and more rendering resources need to be allocated on the initial three-dimensional model reconstructed from CT images. So mainly for exogenous growth, it not only has a tendency to invade outward, but this tendency will also cause unpredictable deformation and necrosis on the side walls around the gastrointestinal stromal tumor inside the digestive system, that is, its deformation is affected by external factors. Therefore, the rendering logic of this part of the structure has always been lacking, that is, the rendering logic around the tumor has always been lacking.

[0056] By analyzing the endogenous and exogenous growth of the rendering reference line, judging the invasion tendency inside and outside, to determine the invasion risk coefficient in different directions. Specifically: Along the direction from the regional centroid of the gastrointestinal stromal tumor area to each position point on the rendering reference line, use the region growing method to expand the rendering reference line in the direction corresponding to each position point. Among them, each position point on the rendering reference line has its own corresponding extended line segment; the extended line segment is a line segment perpendicular to the tangent direction of the rendering reference line with each position point on the rendering reference line as an endpoint after region growing expansion. One endpoint of the extended line segment is a point on the rendering reference line, and the other endpoint is the boundary point of the extended area after using the region growing method to expand the rendering reference line. In the embodiment of the present invention, 20 pixel points are expanded outward at each position point on the rendering reference line.

[0057] Take the length of the extended line segment at each position point on the rendering reference line after region growing expansion as the extended thickness at each position point on the rendering reference line. In the embodiment of the present invention, the length of the line segment is the number of pixel points on the line segment.

[0058] According to the continuous TI signal intensity of each position point on the extended line segment, determine the invasion trend vector of each extended line segment. Specifically: take the first derivative of the TI signal intensity at continuous positions on the extended line segment, and use the average value of the first derivatives corresponding to each extended line segment as the magnitude of the invasion trend vector of the extended line segment, and use the direction of the extended line segment as the direction of the invasion trend vector. This invasion trend vector represents the attenuation direction and gradient of the T1 signal intensity.

[0059] According to the extended thickness of the extended line segment at each position point of the rendering reference line, determine the invasion direction. Specifically: calculate the variance of the extended thickness at all positions on the rendering reference line, and the set of invasion trend vectors. The more uniform the extended thickness, the more obvious the gradient, indicating that the invasion trend of the gastrointestinal stromal tumor is smaller. On the contrary, the more irregular the contour line and the more blurred the gradient, the greater the invasion trend of the gastrointestinal stromal tumor. Then, obtain the position corresponding to the maximum gradient when the extended thickness on the rendering reference line is the thinnest, and obtain the direction of the line connecting the centroid and this position as the invasion direction.

[0060] The included angle formed by the direction of the vector sum of the invasion trend vectors of all extended line segments and the invasion direction is used as the invasion trend transfer included angle. The more consistent the vector directions in the set of invasion trend vectors are with the position where the wall thickness is the thinnest, the more it represents the transfer of the necrosis or cystic degeneration area to the invasion trend direction of the gastrointestinal stromal tumor. In the embodiment of the present invention, the size range of the invasion trend transfer included angle is limited to 0-180°. If the size of the formed included angle exceeds 180°, then the size of the supplementary angle of this included angle is defined as the size of the invasion trend transfer included angle.

[0061] Combined with the size of the invasion trend transfer included angle and the fluctuation of the extended thickness of the extended line segment corresponding to the rendering reference line, determine the invasion risk coefficient. Specifically: obtain the cosine value of the invasion trend transfer included angle, and calculate the variance of the extended thickness of the extended line segment corresponding to the rendering reference line. The variance of the extended thickness represents the fluctuation of the extended thickness; take the product value of the normalized value of the variance and the cosine value as the invasion risk coefficient.

[0062] Use the cosine function to transform the invasion trend transfer included angle. This cosine value represents the transfer probability, that is, the smaller the direction included angle, the more likely it is that the necrosis or cystic degeneration area has transferred to the invasion trend direction of the gastrointestinal stromal tumor. The greater the invasion trend and the necrosis or cystic degeneration area has transferred to the invasion trend direction of the gastrointestinal stromal tumor, the greater the invasion risk.

[0063] Further, according to the infringement risk coefficient, the number of triangular meshes in the initial three-dimensional model is increased to obtain a meshed three-dimensional model, including: on the initial three-dimensional model, with the Z-axis as the reference direction, the size of the triangular meshes is reduced in the area corresponding to the infringement trend transfer angle, and the number of triangular meshes in the initial three-dimensional model is increased. The multiple of the increased triangular meshes is a preset multiple of the infringement risk coefficient, so as to obtain a network three-dimensional model.

[0064] The reason for increasing the number of triangular meshes is that a higher density of triangular meshes can reserve a larger rendering space. In the embodiment of the present invention, the value of the preset multiple is 10. In other embodiments, the implementer can adjust this value according to the actual situation. For example, when the preset multiple is 10, that is, for every 0.1 increase in the infringement risk coefficient, the size of the triangular meshes is reduced by 1 time, and the corresponding multiple of triangular meshes is increased, which can also be understood as the number of triangular meshes is increased by 1 time.

[0065] The texture matching module 30 is used to divide the endoscopic image into multiple pixels by region, and obtain the texture features within each pixel; perform feature matching on the texture features corresponding to the adjacent triangular meshes in the meshed three-dimensional model and the texture features corresponding to the pixels, so as to obtain the best matching pixel corresponding to each triangular mesh.

[0066] Combined with the texture matching module and the subsequent texture rendering module, the internal texture is rendered according to the corrected meshed three-dimensional model.

[0067] Map each vertex of each triangular mesh in the meshed three-dimensional model to the two-dimensional texture space of the CT image to generate UV coordinates. Preferably, it can be achieved by a planar parameterization method, such as by the method of least squares conformal mapping, to ensure that each triangular mesh has a corresponding area in the texture space and reduce deformation. In the rendering pipeline, according to the UV coordinates of the vertices of the triangular meshes, texture sampling is performed on the two-dimensional CT image area corresponding to each position of the triangular meshes, and then based on the UV coordinates of the three vertices in each triangular mesh in the three-dimensional model, texture rendering is performed inside the triangular meshes to ensure correct filling of the texture inside the triangles.

[0068] The texture filled with the UV coordinates of each triangular mesh is based on the texture at the corresponding position in the texture space of the two-dimensional CT image. However, in fact, there may be more deformations and infringement features around the exophytic tumor. The embodiment of the present invention further renders by means of the texture features of the endoscopic image.

[0069] For several endoscopic images of the same patient, use the Matlab tool to divide the regions in the endoscopic images into multiple pixels. The Matlab tool is, for example, the RADIOMICS toolbox. In the embodiments of the present invention, the image is divided into several pixels of size 50*50. Then, extract all first-order statistical features, corner features, edge features, color features, etc. within the pixel range; these features are all three-dimensional rendering sub-resources. The first-order statistical features include: mean, variance, gradient, contrast, etc.

[0070] Obtain the texture features of each pixel within each pixel range: gray-level co-occurrence matrix features and gray-level run-length matrix features. The texture features are all in matrix format and contain all texture details of the gastrointestinal stromal tumor and the adjacent digestive tract tissues. The texture detail data are all statistics obtained from the endoscopic images and mostly do not contain position information.

[0071] The three-dimensional model at this time is a mesh three-dimensional model reconstructed from CT images and corrected by MRI images for triangular meshes. The viewing angles of the CT images and the MRI images can be approximately the same, while the viewing angle of the endoscopic image is not the same as the former two. Therefore, its position information is not referenceable. Therefore, it is necessary to match the texture feature statistics in the endoscopic image with different mesh positions in the three-dimensional model to achieve the purpose of rendering. The texture features within each pixel range may be the texture renderings that should be inserted at different triangular mesh positions.

[0072] Obtain the triangular meshes within the neighborhood range of each triangular mesh in the mesh three-dimensional model as the neighborhood triangular meshes. There are 13 neighborhood triangular meshes within the neighborhood range of each triangular mesh. Except for the target triangular mesh itself, there are 3 edge-adjacent meshes and 9 corner-adjacent meshes. Please refer to Figure 10 , Figure 10 the schematic diagram of the triangular mesh centered on Figure 10 In it, a2~a4 are the edge-adjacent meshes of the triangular mesh a1 centered on a2, and a5~a13 are the corner-adjacent meshes.

[0073] Construct a pixel texture matrix corresponding to each pixel from the texture features of each pixel in the endoscopic image; each type of texture feature corresponds to a pixel texture matrix. For example, for pixel i, construct the pixel texture matrix corresponding to pixel i, including: the gray-level co-occurrence matrix and the gray-level run-length matrix of pixel i.

[0074] Take any triangular mesh as the target triangular mesh, and construct a basic texture matrix from the texture features of all the neighborhood triangular meshes of the target triangular mesh; each type of texture feature of each triangular mesh corresponds to a basic texture matrix. For example, for triangular mesh z, construct the basic texture matrix corresponding to triangular mesh z, including: the gray-level co-occurrence matrix and the gray-level run-length matrix of triangular mesh z.

[0075] Interpolate the basic texture matrix, and align the interpolated basic texture matrix with the pixel texture matrix.

[0076] As a preferred embodiment of the present invention, the matrix of the basic texture features needs to be interpolated with blank cells and then aligned with the matrix of the endoscopic image texture features. The method of interpolating blank cells is to insert 0 values at intervals along the horizontal and vertical axes.

[0077] As another preferred embodiment of the present invention, the gray-level segmentation of the basic texture matrix is the same as that of the pixel texture matrix, and the horizontal and vertical lengths of the two matrices are the same.

[0078] Calculate the difference matrix between the interpolated basic texture matrix corresponding to the same texture feature and the pixel texture matrix of each pixel, and obtain the difference matrix of each texture feature of each pixel.

[0079] ; ; where is the difference matrix corresponding to the gray-level co-occurrence matrix of the pixel; is the difference matrix corresponding to the gray-level region size matrix of the pixel; is the gray-level co-occurrence matrix of the pixel; is the gray-level co-occurrence matrix of the triangular mesh; is the gray-level region size matrix of the pixel; is the gray-level region size matrix of the triangular mesh.

[0080] Analyze the difference matrix of each texture feature of each pixel respectively, and determine the pixel that best matches the target triangular mesh as the best matching pixel.

[0081] Calculate the contrast D, energy E, and entropy value S of the difference matrix corresponding to the gray-level co-occurrence matrix of the pixel.

[0082] Calculate the distribution density M of all elements in the gray-level co-occurrence matrix of the pixel, and the average distance between the elements at all positions and the central element of the matrix. The smaller the matrix distribution density, the more concentrated the gray levels and the smaller the gray-level difference in the difference matrix.

[0083] First, determine the fine texture interpretability according to the difference matrix corresponding to the gray-level co-occurrence matrix.

[0084] Fine texture interpretability The calculation formula of is: .

[0085] High contrast + low energy + high entropy may represent fine texture, such as the edge of a tumor, while The larger the ratio, the more concentrated the grayscale in the difference matrix and the more fine textures with small grayscale differences, that is, the better the explanatory power of the texture features of the pixel for the fine texture in the neighborhood range of the target triangular mesh.

[0086] Then, according to the difference matrix corresponding to the characteristics of the grayscale region size matrix, determine the explanatory power of texture contrast.

[0087] Explanatory power of texture contrast The calculation formula is: ; where C1 is the high grayscale repetition in the large area; C2 is the high grayscale repetition in the small area. Among them, the high grayscale repetition in the large area and the high grayscale repetition in the small area are well-known indicators of the grayscale region size matrix.

[0088] The low grayscale repetition in the small area represents the frequency of occurrence of homogeneous regions with relatively low grayscale values and small sizes, such as textures of microtumors, nodules, necrotic tissues, etc.; the high grayscale repetition in the large area represents the frequency of occurrence of homogeneous regions with relatively high grayscale values and large sizes in the image, such as the basic texture of the digestive inner wall; the smaller the difference between the two, the better the explanatory power of the texture features of the pixel for the low grayscale texture contrast in the neighborhood range of the target triangular mesh.

[0089] Combining the explanatory power of the fine texture and the explanatory power of the texture contrast, determine the matching degree corresponding to each pixel and the target triangular mesh. Among them, the explanatory power of the fine texture is positively correlated with the matching degree; the explanatory power of the texture contrast is negatively correlated with the matching degree.

[0090] In some embodiments, the matching degree corresponding to each pixel and the target triangular mesh is the normalized value of the ratio of the explanatory power of the fine texture and the explanatory power of the texture contrast of each pixel and the target triangular mesh. The smaller the explanatory power of the fine texture and the higher the explanatory power of the texture contrast the higher the matching degree, then the larger the ratio of the explanatory power of the fine texture and the explanatory power of the texture contrast, the stronger the explanatory power of the texture feature group of the pixel for the information in the target triangular mesh, and the higher the matching degree. In the embodiments of the present invention, the normalization of the ratio of the explanatory power of the fine texture and the explanatory power of the texture contrast can be achieved through linear normalization.

[0091] Take the pixel corresponding to the maximum matching degree as the best matching pixel of the target triangular mesh.

[0092] The texture rendering module 40 is used to perform texture rendering on the grid three-dimensional model according to the best matching pixel to obtain a three-dimensional localization model of gastrointestinal stromal tumor.

[0093] According to the matching relationship, render the texture within each triangular mesh. Specifically: Input all the texture feature data and 3D rendering sub-resources within the range of the best-matched pixels corresponding to the target triangular mesh into the target triangular mesh for texture rendering, and perform smoothing processing on the triangular mesh after texture rendering to obtain a 3D model of gastrointestinal stromal tumor as the 3D positioning model.

[0094] That is, input all the texture feature data within the range of the pixels corresponding to the matching of each triangular mesh, as well as all 3D rendering sub-resources, into the triangular mesh for texture rendering; after the rendering is completed, perform smoothing processing between the meshes.

[0095] Since the density of the triangular mesh is corrected, more refined texture rendering can be obtained for the gastrointestinal stromal tumor and its surrounding tissues, and finally a 3D reconstruction result is obtained. The final 3D model is the 3D positioning model.

[0096] Please refer to Figure 9 , which shows the flowchart of the steps of a 3D positioning and surgical navigation method for gastrointestinal stromal tumor with multimodal image fusion provided by an embodiment of the present invention. The method includes: Step S100, obtain the CT image, MRI image, and endoscopic image of the gastrointestinal stromal tumor; generate an initial 3D model from the multi-layer CT image; wherein, the initial 3D model is in the form of a triangular patch mesh. Step S200, extend the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judge the internal and external invasion trends according to the internal and external growth properties of the rendering reference line, and determine the invasion risk coefficient; according to the invasion risk coefficient, increase the number of triangular meshes in the initial 3D model to obtain a mesh 3D model. Step S300, perform regional segmentation on the endoscopic image to obtain multiple pixels, and obtain the texture features within each pixel; perform feature matching on the texture features corresponding to the adjacent triangular meshes in the mesh 3D model and the texture features corresponding to the pixels to obtain the best-matched pixel corresponding to each triangular mesh. Step S400, perform texture rendering on the mesh 3D model according to the best-matched pixel to obtain a 3D positioning model of the gastrointestinal stromal tumor.

[0097] Optionally, the transmission medium can be a wired link, such as but not limited to, coaxial cable, optical fiber, and digital subscriber line, etc., or a wireless link, such as but not limited to, Wireless Fidelity (WIFI), Bluetooth, and mobile device network, etc.

[0098] It should be noted that: For the device provided in the above embodiments, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above.

[0099] A computer device provided by an embodiment of the present invention. Exemplarily, the computer device includes: a memory, a processor, and a computer program stored in the memory and running on the processor. Wherein, when the processor executes the computer program, the computer device can execute any of the multi-modal image fusion gastrointestinal stromal tumor three-dimensional positioning and surgical navigation systems introduced above.

[0100] In addition, an embodiment of the present invention also protects a device, which may include a memory and a processor. Wherein, an executable program code is stored in the memory, and the processor is used to call and execute the executable program code to execute the multi-modal image fusion gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system provided by the embodiment of the present invention.

[0101] An embodiment of the present invention can divide the functions of the device according to the above method examples. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0102] In the case of dividing each module according to each corresponding function, the device may further include a signal uploading module, a determination module, an adjustment module, etc. It should be noted that all relevant contents of each step involved in the above method embodiment can be cited in the function description of the corresponding functional module, and will not be repeated here.

[0103] It should be understood that the device provided by the embodiment of the present invention is used to execute the above multi-modal image fusion gastrointestinal stromal tumor three-dimensional positioning and surgical navigation system, so the same effect as the above implementation method can be achieved.

[0104] In the case of adopting an integrated unit, the device may include a processing module and a storage module. Among them, when the device is applied to a device, the processing module may be used to control and manage the actions of the device. The storage module may be used to support the device in executing mutual program codes, etc. Among them, the processing module may be a processor or a controller, which may implement or execute various exemplary logic blocks, modules, and circuits contemplated in connection with the disclosure of the present invention. The processor may also be a combination that implements computing functions, such as including a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module may be a memory.

[0105] In addition, the device provided in the embodiments of the present invention may specifically be a chip, a component, or a module. The chip may include a connected processor and a memory. Among them, the memory is used to store instructions. When the processor calls and executes the instructions, the chip may execute the multi-modal image fusion three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors provided in the above embodiments.

[0106] The embodiments of the present invention also provide a computer-readable storage medium. Computer program code is stored in the computer-readable storage medium. When the computer program code runs on a computer, the computer is caused to execute the above-related method steps to implement the multi-modal image fusion three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors provided in the above embodiments.

[0107] The embodiments of the present invention also provide a computer program product. When the computer program product runs on a computer, the computer is caused to execute the above-related steps to implement the multi-modal image fusion three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors provided in the above embodiments.

[0108] Among them, the device, the computer-readable storage medium, the computer program product, or the chip provided in the embodiments of the present invention are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved by them can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here. Through the description of the above embodiments, those skilled in the art can understand that for the convenience and conciseness of description, only the above-mentioned division of each functional module is used as an example for illustration. In practical applications, the above functions may be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed device and method may be implemented in other ways.

[0109] The device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0110] It should also be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or terminal device. Without more limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or terminal device including the said element.

[0111] It should be noted that the above sequence of embodiments of the present invention is only for description and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Each embodiment in this specification is described in a progressive manner. The same or similar parts between each embodiment can be referred to each other, and the key point of each embodiment is to illustrate the differences from other embodiments.

[0113] The above content is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered within the protection scope of the present invention.

Claims

1. A three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion, characterized in that, The system includes the following modules: A model initial construction module, which is used to obtain CT images, MRI images and endoscopic images of gastrointestinal stromal tumors; generate an initial three-dimensional model from multi-layer CT images; wherein, the initial three-dimensional model is in the form of a triangular patch grid; A model correction module, which is used to extend the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line; judge the internal and external invasion trend according to the internal and external growth of the rendering reference line, and determine the invasion risk coefficient; according to the invasion risk coefficient, increase the number of triangular meshes in the initial three-dimensional model to obtain a grid three-dimensional model; A texture matching module, which is used to slice the endoscopic image into multiple pixels and obtain the texture features within each pixel; perform feature matching on the texture features corresponding to adjacent triangular meshes in the grid three-dimensional model and the texture features corresponding to the pixels to obtain the best matching pixel corresponding to each triangular mesh; A texture rendering module, which is used to perform texture rendering on the grid three-dimensional model according to the best matching pixel to obtain a three-dimensional positioning model of gastrointestinal stromal tumors.

2. The three-dimensional localization and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that The generation of the initial three-dimensional model from multi-layer CT images includes: Stack multi-layer CT images in the Z-axis direction to generate a three-dimensional reconstruction model; perform smooth reconstruction on the three-dimensional model to obtain an initial three-dimensional model; wherein, the initial three-dimensional model is in the form of a triangular patch grid.

3. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, wherein The extension of the contour line of the gastrointestinal stromal tumor region in the MRI image to obtain a rendering reference line includes: Based on the MRI image, segment the gastrointestinal stromal tumor region; Use the canny detection algorithm to obtain the boundary contour line of the digestive system attached to the gastrointestinal stromal tumor region; Taking the two ends of the contour line segment between the two intersection points of the gastrointestinal stromal tumor region on the boundary contour line of the attached digestive system as the extension starting points, and taking the extension starting points as the starting points, extend the contour line segment to both sides by a preset length to obtain a rendering reference line.

4. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, wherein The determination of the internal and external invasion trend and the invasion risk coefficient according to the internal and external growth of the rendering reference line includes: Expand the rendering reference line along the direction from the regional centroid of the gastrointestinal stromal tumor region to each position point on the rendering reference line, wherein each position point on the rendering reference line has its own corresponding extended line segment; Determine the invasion trend vector of each extended line segment according to the continuous TI signal intensity of each position point on the extended line segment; Determine the invasion direction according to the extended thickness of the extended line segment of each position point on the rendering reference line; Taking the angle formed by the direction of the vector sum of the invasion trend vectors of all extended line segments and the invasion direction as the invasion trend transfer angle; Combining the size of the invasion trend transfer angle and the fluctuation of the extended thickness of the extended line segment corresponding to the rendering reference line, determine the invasion risk coefficient.

5. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 4, characterized in that, The increase of the number of triangular meshes in the initial three-dimensional model according to the invasion risk coefficient to obtain a grid three-dimensional model includes: Increase the number of triangular meshes in the initial three-dimensional model in the area corresponding to the invasion trend transfer angle, and the multiple of the added triangular meshes is the preset multiple of the invasion risk coefficient to obtain a network three-dimensional model.

6. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, The texture features within each pixel include: gray-level co-occurrence matrix features and gray-level region size matrix features.

7. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 6, characterized in that, The feature matching of the texture features corresponding to adjacent triangular meshes in the grid 3D model with the texture features corresponding to the pixels to obtain the best-matching pixel corresponding to each triangular mesh includes: Obtaining the triangular meshes within the neighborhood range of each triangular mesh in the grid 3D model as the neighborhood triangular meshes; Constructing a pixel texture matrix corresponding to each pixel from the texture features of each pixel in the endoscopic image; each texture feature corresponds to a pixel texture matrix; Taking any triangular mesh as the target triangular mesh and constructing a basic texture matrix from the texture features of all the neighborhood triangular meshes of the target triangular mesh; each texture feature of each triangular mesh corresponds to a basic texture matrix; Performing interpolation processing on the basic texture matrix and aligning the interpolated basic texture matrix with the pixel texture matrix; Calculating the difference matrix between the interpolated basic texture matrix corresponding to the same texture feature and the pixel texture matrix of each pixel to obtain the difference matrix of each texture feature of each pixel; Analyzing the difference matrix of each texture feature of each pixel respectively to determine the pixel that best matches the target triangular mesh as the best-matching pixel.

8. The three-dimensional localization and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 7, characterized in that, The analyzing the difference matrix of each texture feature of each pixel respectively to determine the pixel that best matches the target triangular mesh as the best-matching pixel includes: Determining the fine texture interpretability according to the difference matrix corresponding to the gray-level co-occurrence matrix; Determining the texture contrast interpretability according to the difference matrix corresponding to the gray-level region size matrix feature; Combining the fine texture interpretability and the texture contrast interpretability to determine the matching degree between each pixel and the target triangular mesh; Taking the pixel corresponding to the maximum matching degree as the best-matching pixel of the target triangular mesh.

9. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 7, characterized in that, The texture rendering of the grid 3D model according to the best-matching pixel to obtain the 3D localization model of gastrointestinal stromal tumor includes: Inputting all the texture feature data and 3D rendering sub-resources within the range of the best-matching pixel corresponding to the target triangular mesh into the target triangular mesh for texture rendering, and performing smoothing processing on the texture-rendered triangular mesh to obtain the 3D model of gastrointestinal stromal tumor as the 3D localization model.

10. The three-dimensional positioning and surgical navigation system for gastrointestinal stromal tumors with multimodal image fusion according to claim 1, characterized in that, The texture rendering of the grid 3D model according to the best-matching pixel further includes: Mapping each vertex of each triangular mesh in the grid 3D model to the 2D texture space of the CT image to generate UV coordinates; Performing texture sampling on the 2D CT image region corresponding to each triangular mesh according to the UV coordinates of the vertices of the triangular mesh, and performing texture rendering inside the triangular mesh based on the UV coordinates of the three vertices in each triangular mesh of the grid 3D model.

Images