Digestive tract tumor lesion image segmentation method and system based on multi-modal fusion technology

Abstract

The present application relates to the technical field of image semantic segmentation, in particular to a digestive tract tumor lesion image segmentation method and system based on multi-modal fusion technology, comprising the following steps: collecting CT and EUS sequences to generate an organ surface three-dimensional triangular mesh model, mapping anatomical markers and constructing a dual-mode lumen manifold surface geodesic distance topological atlas, calculating the connection edge difference and the total amount of elastic potential energy, and generating a digestive tract tumor lesion pixel-level segmentation mask. In the present application, the organ physical form is restored by constructing a covering tube wall triangular mesh reducer, the path length is accurately quantified by using the manifold surface geodesic distance instead of the straight line distance, the potential energy gradient is calculated to drive the nodes to converge in the energy descent direction, the non-rigid mapping field conforming to the anatomical constraint is generated to offset the endoscope compression distortion, the EUS texture is accurately mapped to the CT coordinate system under the premise of maintaining the topological integrity, the multi-modal feature deep fusion is realized, and the soft tissue lesion segmentation accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of image semantic segmentation technology, and in particular to a method and system for segmenting images of digestive tract tumor lesions based on multimodal fusion technology. Background Technology

[0002] Image semantic segmentation technology mainly involves using computer vision algorithms to classify each pixel in a digital image and assign it a specific category label. Its core aspects include high-dimensional image feature extraction, contextual semantic information aggregation, and pixel-level prediction and reconstruction, serving as a key means for fine-grained understanding and analysis of the entire image. This technology constructs a deep neural network model containing encoder and decoder structures to perform hierarchical feature mapping on the input image, thereby achieving accurate separation of the target object from the background. It is widely used in scenarios such as medical image analysis, autonomous driving environmental perception, and intelligent security monitoring, aiming to establish a mapping relationship between image pixels and semantic categories. Traditional methods for segmenting images of gastrointestinal tumor lesions refer to the process of locating and extracting the contours of tumor entities in the gastrointestinal wall and adjacent tissues. These methods use single-modal equipment such as endoscopic ultrasound, computed tomography, or magnetic resonance imaging to independently acquire anatomical data of the gastrointestinal tract, primarily relying on radiologists to manually delineate layer by layer using pathological knowledge, or using threshold segmentation operators based on gray-level histograms and gradient-based edge detection operators for semi-automatic identification of lesion boundaries. When multimodal data is involved in assisted diagnosis, rigid registration algorithms based on mutual information or affine transformation geometric correction methods are generally used to spatially align images from different imaging sources. Then, the image information is fused through simple pixel-level weighted averaging or channel cascading to allow physicians to assess the tumor invasion level and metastasis status.

[0003] Existing technologies rely on independent acquisition of a single modality or manual delineation, which limits processing efficiency. Semi-automatic recognition based on grayscale histograms lacks adaptability to complex anatomical environments. When multimodal assisted diagnosis is involved, the general rigid registration or affine transformation ignores the non-rigid deformation characteristics of hollow tubular organs. Furthermore, spatial mapping based on Euclidean distance is difficult to characterize the physical path of soft tissue surfaces. Simple pixel-level weighted averaging cannot correct spatial misalignment caused by organ peristalsis or instrument compression, resulting in the failure of texture feature matching between different imaging sources. This leads to deviations in tumor boundary localization and the inability to effectively fuse anatomical structure and pathological texture information. Summary of the Invention

[0004] To address the technical problems existing in the prior art, embodiments of the present invention provide a method for segmenting digestive tract tumor lesion images based on multimodal fusion technology, comprising the following steps: S1: Collect computed tomography scan slices and endoscopic ultrasound scan sequences, extract contour data and convert them into three-dimensional point clouds, construct triangular facet indexes connecting adjacent point clouds, and generate a three-dimensional triangular mesh model of the surface of digestive organs. S2: Based on the three-dimensional triangular mesh model of the surface of the digestive tract organs, map the anatomical marker positions as nodes, perform a breadth-first search along the mesh edge and accumulate the physical edge length, obtain the connection weight, and establish a topological map of geodesic distances on the surface of the bimodal lumen manifold. S3: Compare the differences in geodesic distances between the topological map connecting edges of the dual-modal tubular manifold surface, calculate the total elastic potential energy based on the sum of squared differences, correct the node coordinates along the opposite direction of the gradient, and obtain the non-rigid deformation mapping field of the multimodal anatomical structure. S4: Based on the non-rigid deformation mapping field of the multimodal anatomical structure, calculate the corresponding coordinates of the ultrasound endoscope voxels in the tomographic scanning space and fill the echo texture data to establish a spatially aligned multimodal fusion feature data volume; S5: Screen the voxel features of the spatially aligned multimodal fusion feature data, calculate the joint probability value of the voxel belonging to the tumor lesion, and generate a pixel-level segmentation mask for the digestive tract tumor lesion.

[0005] As a further aspect of the present invention, the three-dimensional triangular mesh model of the digestive tract organ surface includes a set of vertex spatial coordinates, a list of triangular facet topological indexes, and surface normal vector data; the bimodal luminal manifold surface geodesic distance topological map includes a set of anatomical key point nodes, a list of adjacent edges of the manifold surface, and a geodesic distance weight matrix; the multimodal anatomical structure non-rigid deformation mapping field includes a set of node spatial displacement vectors, a non-rigid coordinate transformation matrix, and a minimized elastic potential energy parameter; the spatially aligned multimodal fusion feature data volume includes spatially registered voxel coordinates, CT tissue density channel data, and EUS echo texture channel data; and the digestive tract tumor lesion pixel-level segmentation mask includes a tumor region binarized marker map, a lesion edge contour index, and a set of foreground pixel positions.

[0006] As a further aspect of the present invention, the specific steps of S1 are as follows: S101: Collect computed tomography (CT) scan slice sequences and endoscopic ultrasound scan sequences of the digestive tract, scan the pixel gray-level gradient distribution of the slice plane, identify the location of gray-level abrupt changes, locate the edge pixels of the inner and outer walls of the organ, extract the two-dimensional plane coordinates of the edge pixels in the slice coordinate system, stack and arrange the multi-layer coordinate data according to the slice scan layer sequence, and generate a multimodal tomographic contour dataset. S102: Call the multimodal tomographic contour dataset, extract the scanning interlayer spacing and layer thickness parameters corresponding to the slice sequence, perform longitudinal Z-axis mapping transformation by combining the two-dimensional pixel coordinates of the contour points with the interlayer spacing values, convert the planar coordinates into spatial coordinates under a unified three-dimensional Cartesian coordinate system, perform interpolation processing on sparse regions, densify the distribution of points, and establish a discrete point cloud coordinate set. S103: For the discrete point cloud coordinate set, calculate the Euclidean distance between the spatial point cloud and the neighboring point cloud, determine the spatial topological adjacency relationship of the point cloud according to the principle of closest distance, connect the three adjacent point cloud coordinates in counterclockwise order, construct the triangular patch topological index, close and combine the triangular patches, fit the geometric surface of the organ, and generate a three-dimensional triangular mesh model of the surface of the digestive tract organ.

[0007] As a further aspect of the present invention, the specific steps of S2 are as follows: S201: Based on the three-dimensional triangular mesh model of the digestive tract organ surface, retrieve the geometric curvature feature data of the mesh surface, locate the mesh vertex index representing the position of the pyloric ring and the tumor center, map the mesh vertex index to discrete nodes in the graph theory structure, extract the Cartesian coordinate parameters of the discrete nodes in three-dimensional space, and generate a set of key anatomical feature nodes. S202: Call the set of key anatomical feature nodes and the three-dimensional triangular mesh model of the surface of digestive organs, take the nodes in the set as the starting endpoint of the search, traverse the adjacent vertices layer by layer along the common edge of the triangular facet, calculate the Euclidean distance between the coordinates of the two vertices at the two ends of the mesh edge to determine the physical edge length, accumulate the physical length of the mesh edge traversed by the search path, select the path with the smallest accumulated value between two nodes as the geodesic distance of the manifold surface, and generate the mesh surface path length matrix. S203: For the path length matrix of the mesh surface, construct a topological edge structure connecting the dissected nodes, assign the corresponding geodesic distance values ​​in the path length matrix of the mesh surface as weight parameters to the topological edges, define the adjacency relationship and distance metric constraints between nodes, and generate a topological map of geodesic distances on the surface of the bimodal tubular manifold.

[0008] As a further aspect of the present invention, the specific steps of S3 are as follows: S301: Call the geodesic distance topology map of the dual-modal lumen manifold surface, index the multiple topological edge structures connecting the map nodes, read the geodesic distance parameters of the topological edges in the computed tomography mode and the corresponding distance parameters in the endoscopic ultrasound mode, perform subtraction operation on the corresponding parameters, obtain the geometric length deviation between modes, perform square processing on the geometric length deviation, quantify the elastic deformation degree of the edge length, and obtain the topological connection edge difference measurement data between modes; S302: For the difference measurement data of the topological connection edges between the modes, the square deviation values ​​of the edge lengths in the set are accumulated and summed to establish an elastic potential energy function describing the overall deformation degree of the anatomical structure. Partial differential operation is performed on the elastic potential energy function with respect to the spatial three-dimensional coordinate variables of each graph structure node. The rate of change of the elastic potential energy function at the real-time node coordinate position is analyzed to determine the vector direction and step size parameter of the driving node to evolve towards the energy minimum state and generate a set of elastic potential energy spatial gradient vectors. S303: Based on the set of elastic potential energy spatial gradient vectors, the initial coordinates of the anatomical marker nodes in the topological map of the geodesic distance of the dual-modal tubular manifold surface are used as a reference. The coordinate displacement correction operation is performed along the opposite direction of the gradient vector. The spatial position of the node in the three-dimensional Cartesian system is iteratively updated until the potential energy reaches a stable state, thereby generating a non-rigid deformation mapping field of the multimodal anatomical structure.

[0009] As a further aspect of the present invention, the process of establishing the elastic potential energy function describing the overall deformation degree of the anatomical structure is specifically as follows: calling a preset tissue mechanics parameter library to obtain the local stiffness weighting coefficient simulating the characteristics of biological soft tissue, and performing a point-to-point multiplication operation between the square value of the geometric length deviation included in the intermodal topological connection edge difference measurement data and the local stiffness weighting coefficient to generate a weighted unilateral elastic potential energy value. Traverse all connected edge structures in the topological map of geodesic distances on the surface of the bimodal tubular manifold, perform a global accumulation operation on the weighted single-sided elastic potential energy values, and calculate the discrete Laplace coordinate vectors of the map nodes to characterize the local curvature of the mesh. Construct a composite objective function including a deformation energy accumulation term and a Laplace smoothing regularization term as the elastic potential energy function.

[0010] As a further aspect of the present invention, the specific steps of S4 are as follows: S401: Call the non-rigid deformation mapping field of the multimodal anatomical structure to obtain the original three-dimensional voxel mesh parameters of the ultrasound endoscopic scanning sequence, extract the initial spatial coordinate vector of the voxel vertices inside the mesh one by one, apply the defined nonlinear coordinate transformation rules and displacement parameters to perform geometric transformation operations on the initial spatial coordinate vectors, solve the new spatial position coordinates of the voxel points mapped to the computed tomography reference system, and generate an ultrasound endoscopic voxel remapping coordinate index table; S402: For the ultrasound endoscope voxel remapping coordinate index table, retrieve the echo texture intensity value of the pixel in the ultrasound endoscope scanning sequence, fill the echo texture intensity value into the new spatial position after conversion according to the mapping index relationship, and for spatial points that fall into the non-integer coordinate grid, calculate the distance weighted value of the echo intensity of the surrounding neighboring voxels and perform interpolation assignment operation to generate deformation-corrected echo texture data volume. S403: Based on the deformation-corrected echo texture data volume, traverse the spatial coordinate positions of the internal voxels, read the tissue radiation density values ​​corresponding to the coordinate positions in the computed tomography scan slice sequence, perform channel cascading operation on the tissue radiation density values ​​and the deformation-corrected echo texture values ​​at the same positions, construct a composite voxel unit that combines anatomical structure density information and pathological texture information, and generate a spatially aligned multimodal fusion feature data volume.

[0011] As a further aspect of the present invention, the process of calculating the distance-weighted value of the echo intensity of the surrounding neighborhood voxels and performing interpolation assignment is specifically as follows: based on the coordinate values ​​of the spatial point falling into the non-integer coordinate grid, lock the eight nearest integer grid nodes surrounding the spatial point in the three-dimensional space, and read the original ultrasonic echo intensity stored in the nearest integer grid nodes. Calculate the projection distance deviation of a spatial point relative to its nearest integer grid node in the three axes of the Cartesian coordinate system, and construct the inverse distance volume weight coefficient of the corresponding node based on the projection distance deviation. The original ultrasonic echo intensity of the nearest integer grid node is weighted and summed with the corresponding inverse distance volume weighting coefficient to output the smooth echo texture value of the spatial point.

[0012] As a further aspect of the present invention, the specific steps of S5 are as follows: S501: Based on the spatially aligned multimodal fusion feature data volume, the tissue radiation density value and echo texture intensity value stored at the voxel location are parsed. After normalizing the two sets of values, they are input into the multivariate joint probability calculation formula. For each voxel, the confidence score of the feature vector mapped to the tumor lesion category is calculated to generate the joint probability distribution matrix of voxel lesion categories. S502: For the voxel lesion category joint probability distribution matrix, set a probability judgment benchmark threshold for distinguishing lesion tissue, compare the confidence score of voxels in the voxel lesion category joint probability distribution matrix with the benchmark threshold item by item, extract the voxel spatial coordinate index of the voxel whose confidence score exceeds the benchmark threshold, mark the voxel attribute corresponding to the spatial coordinate index as a positive foreground object, mark the remaining voxels as negative background objects, and generate a set of binary labels for the tumor foreground region; S503: Call the set of binary markers for the tumor foreground region, establish a three-dimensional mask data structure with the same dimension as the original scan sequence based on the recorded coordinates of the positive foreground object, fill the voxel positions corresponding to the foreground coordinates with the highlighted label values, fill the remaining positions with the zero-value background, identify the exact distribution range and edge contour information of the tumor tissue in the three-dimensional anatomical space, and generate a pixel-level segmentation mask for digestive tract tumor lesions.

[0013] A digestive tract tumor lesion image segmentation system based on multimodal fusion technology includes: The 3D mesh construction module collects computed tomography scan slices and endoscopic ultrasound scan sequences, extracts contour data and converts it into a 3D point cloud, constructs a triangular patch index connecting adjacent point clouds, and generates a 3D triangular mesh model of the surface of digestive organs. The topology map building module, based on the three-dimensional triangular mesh model of the surface of the digestive tract organs, maps the anatomical marker positions as nodes, performs a breadth-first search along the mesh edge and accumulates the physical edge length, obtains the connection weight, and builds a topology map of geodesic distances on the surface of the bimodal lumen manifold. The deformation mapping field generation module compares the differences in geodesic distances from the topological map connecting edges of the dual-modal tubular manifold surface, calculates the total elastic potential energy based on the sum of squared differences, corrects the node coordinates along the opposite direction of the gradient, and obtains the non-rigid deformation mapping field of the multimodal anatomical structure. The multimodal feature fusion module calculates the corresponding coordinates of the ultrasound endoscopic voxels in the tomographic scanning space and fills the echo texture data based on the non-rigid deformation mapping field of the multimodal anatomical structure, and establishes a spatially aligned multimodal fusion feature data volume. The lesion segmentation mask generation module screens the voxel features of the spatially aligned multimodal fusion feature data, calls the conditional random field model to calculate the joint probability value of voxels belonging to tumor lesions, and generates a pixel-level segmentation mask for digestive tract tumor lesions.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, a triangular mesh model covering the surface of the tube wall is constructed to restore the physical geometry of the organ. A topological map is constructed by using geodesic distances along the manifold surface to replace straight-line distances in space. This allows for accurate quantification of the physical path lengths on the soft tissue surface. An elastic potential energy function describing the topological differences between modes is established. The potential energy gradient is calculated and the node coordinates are driven to converge iteratively along the energy decrease direction. This generates a non-rigid deformation mapping relationship that conforms to anatomical constraints, effectively offsetting the morphological distortion caused by endoscopic ultrasound compression. While maintaining the integrity of the topological structure, pathological texture features are accurately mapped to the anatomical coordinate system, achieving deep fusion of multimodal features and significantly improving the consistency of soft tissue lesion edge segmentation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the steps of the present invention; Figure 2This is a detailed schematic diagram of S1 of the present invention; Figure 3 This is a detailed schematic diagram of S2 of the present invention; Figure 4 This is a detailed schematic diagram of S3 of the present invention; Figure 5 This is a detailed schematic diagram of S4 of the present invention; Figure 6 This is a detailed schematic diagram of S5 of the present invention; Figure 7 This is a system module diagram of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0018] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0019] Please see Figure 1 This invention provides a method for segmenting images of gastrointestinal tumor lesions based on multimodal fusion technology, comprising the following steps: S1: Collect computed tomography (CT) slice sequences and endoscopic ultrasound (EUS) scan sequences of the digestive tract, extract the contour data of internal organs and convert them into three-dimensional spatial point cloud coordinates, construct a triangular patch index connecting adjacent point clouds, and generate a three-dimensional triangular mesh model of the surface of digestive tract organs. S2: Based on the three-dimensional triangular mesh model of the surface of digestive organs, identify the anatomical marker positions and map them as graph structure nodes. Perform a breadth-first search along the edge of the mesh triangular facets, accumulate the physical edge length of the search path mesh, obtain the inter-node pathway metric, and establish a topological map of geodesic distances on the surface of a bimodal lumen manifold. S3: Compare the differences in the measurement of the connecting edges between the two modes in the topological map of the geodesic distance of the bimodal manifold surface, calculate the total elastic potential energy based on the sum of squared differences, iteratively correct the node spatial coordinates in the opposite direction of the gradient of the total elastic potential energy, and obtain the non-rigid deformation mapping field of the multimodal anatomical structure. S4: Based on the non-rigid deformation mapping field of multimodal anatomical structures, calculate the corresponding coordinates of endoscopic ultrasound voxels in the tomographic scanning space, fill the transformed coordinates with endoscopic ultrasound echo texture data, combine the aligned texture and tomographic scanning density data, and establish a spatially aligned multimodal fusion feature data volume. S5: Screen the voxel tissue density and echo texture features in the spatially aligned multimodal fusion feature data volume, calculate the joint probability value of voxels belonging to the tumor lesion category, filter the joint probability values ​​that are higher than the preset tumor determination probability threshold, and mark the voxel position as the foreground region to generate a pixel-level segmentation mask for digestive tract tumor lesions. The three-dimensional triangular mesh model of the digestive tract organ surface includes a set of vertex spatial coordinates, a list of triangular facet topological indexes, and surface normal vector data. The bimodal luminal manifold surface geodesic distance topological map includes a set of anatomical key point nodes, a list of adjacent edges of the manifold surface, and a geodesic distance weight matrix. The multimodal anatomical structure non-rigid deformation mapping field includes a set of node spatial displacement vectors, a non-rigid coordinate transformation matrix, and a minimized elastic potential energy parameter. The spatially aligned multimodal fusion feature data volume includes spatial registration voxel coordinates, CT tissue density channel data, and EUS echo texture channel data. The pixel-level segmentation mask for digestive tract tumor lesions includes a binary marker map of the tumor region, a lesion edge contour index, and a set of foreground pixel positions.

[0020] Please see Figure 2 The specific steps of S1 are as follows: S101: Collect computed tomography (CT) scan slice sequences and endoscopic ultrasound scan sequences of the digestive tract, scan the pixel gray-level gradient distribution of the slice plane, identify the location of gray-level abrupt changes, locate the edge pixels of the inner and outer walls of the organ, extract the two-dimensional plane coordinates of the edge pixels in the slice coordinate system, stack and arrange the multi-layer coordinate data according to the slice scan layer sequence, and generate a multimodal tomographic contour dataset. High-resolution image acquisition is performed on digestive tract areas (such as the stomach, esophagus, or colon). Specifically, a computed tomography (CT) scanner with 64 rows of detectors is used, with the tube voltage set to 120 kV and the tube current set to 200 mA. Continuous volumetric scanning with a slice thickness of 0.625 mm is performed on the target area to obtain CT slice sequences. Simultaneously, a 12 MHz circular array ultrasound endoscope probe is used to acquire corresponding endoscopic ultrasound scan sequences at a retraction speed of 2 mm / s. To achieve refined identification of anatomical structures, a preprocessing logic based on image semantic segmentation is introduced, constructing an encoder module based on a fully convolutional network (FCN) or U-Net architecture. The two-dimensional image matrix in the CT slice sequence is read frame by frame. For each 512 x 512 pixel slice image, two parallel processing operations are performed: one performs grayscale gradient calculation based on the Sobel operator, and the other performs semantic feature extraction based on deep learning. In the gradient calculation path, a 3x3 horizontal convolution kernel and a 3x3 vertical convolution kernel are constructed. These two kernels are then convolved with the gray values ​​of each pixel in the slice image and its eight neighboring pixels. The horizontal convolution operation calculates the weighted difference between the gray values ​​of the right and left neighboring pixels to obtain the gray-level change rate of the pixel in the horizontal direction. The vertical convolution operation calculates the weighted difference between the gray values ​​of the lower and upper neighboring pixels to obtain the gray-level change rate of the pixel in the vertical direction. Next, the squares of the horizontal and vertical gray-level change rates are summed, and the square root of the sum is calculated to obtain the overall gradient magnitude of the pixel. After obtaining the gradient magnitudes of all pixels in the image, an edge detection baseline threshold of 150 Henle units per pixel is set. Simultaneously, the aforementioned encoder module is used to perform multi-scale convolution and pooling operations on the image to extract implicit high-dimensional semantic feature maps, aiding in determining the tissue category (e.g., tube wall, lumen, or background) of each pixel. The gradient magnitude of each pixel is compared with the baseline threshold, and a joint determination is made based on the contextual information of the semantic feature map. If the gradient magnitude of a pixel exceeds 150 Henlein units per pixel, and the semantic segmentation probability map indicates that the location is in a tissue boundary transition zone, then a gray-level abrupt change is identified at that location, classifying it as a potential organ tissue boundary point. To distinguish between the inner and outer walls, based on prior anatomical knowledge—that the interior of the digestive tract lumen is typically composed of low-density gas or liquid, while the tube wall is composed of medium- to high-density soft tissue—a radial search is performed from the center of the lumen outwards. The first high-gradient band encountered is marked as an inner wall edge pixel, and the subsequent second high-gradient band is marked as an outer wall edge pixel. The horizontal X-axis coordinates and vertical Y-axis coordinates of all marked edge pixels in the current slice plane coordinate system are extracted, and the slice sequence number in the scanning sequence is recorded.Finally, the edge pixel coordinate data of each layer are stacked and arranged in ascending order of layer number to construct a structured dataset containing multi-layer anatomical contour information, thus generating a multimodal tomographic contour dataset.

[0021] S102: Call the multimodal tomographic contour dataset, extract the scanning interlayer spacing and layer thickness parameters corresponding to the slice sequence, perform longitudinal Z-axis mapping transformation by combining the two-dimensional pixel coordinates of the contour points with the interlayer spacing values, convert the planar coordinates into spatial coordinates under a unified three-dimensional Cartesian coordinate system, perform interpolation processing on sparse regions, densify the distribution of points, and establish a discrete point cloud coordinate set. The DICOM header information is parsed to extract key spatial positioning parameters, specifically the interlayer spacing during slice scanning (e.g., 0.8 mm) and the physical layer thickness of a single slice (e.g., 1.0 mm). Then, for each 2D edge contour point in the dataset, a vertical Z-axis mapping transformation from the 2D plane to 3D space is performed. The specific logic of this transformation is as follows: the physical dimensions of the contour point's horizontal X-axis and vertical Y-axis pixel coordinates within the slice remain unchanged (i.e., the pixel coordinates are multiplied by the pixel spacing parameter, such as 0.5 mm per pixel), while simultaneously calculating the contour point's vertical Z-axis coordinate in 3D space. The Z-axis coordinate is calculated by multiplying the slice layer sequence number to which the contour point belongs by the scanning interlayer spacing parameter, and adding the absolute physical offset value of the starting scan position. This operation transforms the originally discrete 2D coordinate points distributed on various independent slice planes into spatial coordinate points (x, y, z) in a unified 3D Cartesian coordinate system. After completing the basic coordinate mapping, for sparse regions where the vertical resolution is lower than the horizontal resolution due to the large interlayer spacing, 3D spatial interpolation based on cubic spline functions is performed. Specifically, a parameterized cubic polynomial curve equation is constructed between corresponding contour points of two adjacent slice sequences. Based on a preset point density requirement (e.g., one point per 0.5 mm), new intermediate interpolation point coordinates are uniformly sampled on this curve equation. During calculation, the contour point coordinates of four adjacent slices are used as control points to solve for the polynomial coefficients, ensuring that the generated interpolation curve has second-order derivative continuity at the connection points, thus guaranteeing the smoothness of the reconstructed surface. These newly generated interpolation point coordinates are then merged with the original contour point coordinates, significantly improving the spatial density of the point cloud data and establishing a discrete point cloud coordinate set.

[0022] Table 1 shows the coordinate mapping and interpolation results of some key sequences. For example, for the original contour point A with sequence number 10, its physical coordinates within the slice are (50.0, 50.0), and the interlayer spacing is 0.8 mm. Its Z-axis coordinate is calculated as 10 * 0.8 = 8.0 mm, which is equivalent to spatial coordinates of (50.0, 50.0, 8.0). For the corresponding point B with sequence number 11, its physical coordinates within the slice are (50.2, 50.1), and its Z-axis coordinate is 11 * 0.8 = 8.8 mm, which is equivalent to spatial coordinates of (50.2, 50.1, 8.8). Interpolation is performed between points A and B. If the interpolation step size is set to 0.4 mm, the spatial coordinates of the intermediate interpolation point C, calculated using a cubic spline function, are approximately (50.1, 50.05, 8.4), thus filling the gaps between the layers.

[0023] Table 1: Example Table of Coordinate Mapping and Interpolation Parameters S103: For a set of discrete point cloud coordinates, calculate the Euclidean distance between the spatial point cloud and the neighboring point cloud, determine the spatial topological adjacency relationship of the point cloud according to the principle of closest distance, connect the three adjacent point cloud coordinates in counterclockwise order, construct the triangular patch topological index, close the combined triangular patches, fit the geometric surface of the organ, and generate a three-dimensional triangular mesh model of the surface of the digestive tract organ. A spherical rotation algorithm or Delaunay triangulation strategy is used to construct topological connections. For each target point cloud coordinate in the set, a search radius (e.g., 1.5 mm) is defined in 3D space, and all neighboring point clouds falling within this radius are retrieved. Next, the Euclidean distance between the target point cloud and each neighboring point cloud is calculated. The Euclidean distance calculation logic is as follows: obtain the squares of the differences in coordinates along the X-axis, Y-axis, and Z-axis of two points, add these three squares, and take the square root. The calculated distances are sorted in ascending order, and the two neighboring points with the smallest distance are selected as candidate connection points based on the principle of closest proximity. Subsequently, it is verified whether the triangle formed by the target point cloud and these two candidate neighboring points meets the normal vector consistency constraint, i.e., the normal vector of the triangle plane is calculated, and its direction is determined to be pointing outwards. If the constraint is met, the index numbers of these three points are recorded in a facet index list in counter-clockwise order to construct a triangular facet topological index. This process is repeated, traversing all point clouds in the set, generating a series of closely connected triangular faces. Finally, a mesh closure detection operation is performed to identify and fill the pore areas on the surface, and all triangular patches are combined to fit a continuous and smooth geometric surface of the digestive tract organs, generating a three-dimensional triangular mesh model of the digestive tract organ surface.

[0024] Please see Figure 3 The specific steps of S2 are as follows: S201: Based on the three-dimensional triangular mesh model of the surface of digestive tract organs, retrieve the geometric curvature feature data of the mesh surface, locate the mesh vertex index that represents the position of the pyloric ring and the center of the tumor, map the mesh vertex index to discrete nodes in the graph theory structure, extract the Cartesian coordinate parameters of the discrete nodes in three-dimensional space, and generate a set of key anatomical feature nodes. The geometric curvature features of each vertex on the model surface are numerically calculated. Specifically, for each vertex, the normal vector and area of ​​the surrounding adjacent triangular facets are obtained, and the curvature is calculated using the discrete Gaussian curvature formula or the average curvature formula. For example, the Gaussian curvature is estimated by calculating the ratio of the angular deficit to the area of ​​the neighborhood around the vertex. A curvature benchmark threshold for anatomical feature identification is set (e.g., the reciprocal of 0.05 mm), and the curvature calculation results of each vertex are compared one by one. Since the pyloric ring or tumor protrusion region usually has significant geometric curvature features, its curvature value will be significantly higher than that of the flat duct wall region. When the calculated curvature of a vertex is higher than the set benchmark threshold, it is marked as a candidate feature point. Further, combined with the prior constraints of anatomical location (e.g., the pylorus is located at the end of the gastric antrum), the specific grid vertex index representing the center of the pyloric ring and the geometric center of the tumor lesion is accurately located from the candidate points. Subsequently, these key mesh vertex indexes are mapped into discrete node identifiers (IDs) in the graph theory structure, and the corresponding three-dimensional Cartesian coordinate parameters (x, y, z) of these nodes are directly extracted from the data structure of the mesh model. The node IDs and coordinate parameters are stored in pairs to generate a set of dissecting key feature nodes.

[0025] S202: Call the set of key anatomical feature nodes and the three-dimensional triangular mesh model of the surface of digestive organs. Use the nodes in the set as the starting endpoints of the search. Traverse adjacent vertices layer by layer along the common edge of the triangular facets. Calculate the Euclidean distance between the coordinates of the two vertices at the two ends of the mesh edge to determine the physical edge length. Accumulate the physical length of the mesh edge traversed by the search path. Select the path with the smallest accumulated value between two nodes as the geodesic distance of the manifold surface and generate the mesh surface path length matrix. Pathfinding is performed on the triangular mesh surface using either Dijkstra's shortest path algorithm or the FastMarching Method. A node from the set of key feature nodes is designated as the source node. The distance from this source node to itself is initialized to 0, and the distances to all other nodes are set to infinity. The source node is then added to a priority queue. Next, the node with the smallest current distance is retrieved from the priority queue, and all adjacent vertices directly connected to this node in the mesh model via common triangular facet edges are traversed. For each pair of adjacencies, the Euclidean distance between the coordinates of the two vertices on the mesh edge is calculated as the physical edge length. Specifically, the magnitude of the difference between the coordinate vectors of the two endpoints is obtained. Then, a relaxation operation is performed: the known shortest distance of the current node is added to the physical edge length to obtain the total path length from the current node to the adjacent vertices. If this total path length is less than the currently recorded distance value of the adjacent vertex, the shortest distance value of the adjacent vertex is updated, and the adjacent vertex is added to the priority queue. Repeat the above expansion process until all target nodes in the set have been traversed, thereby selecting the path with the smallest cumulative value between two nodes. This path represents the geodesic distance along the surface of the manifold. Finally, fill the geodesic distance values ​​between all node pairs into a two-dimensional matrix to generate the mesh surface path length matrix.

[0026] S203: For the path length matrix of the mesh surface, construct the topological edge structure connecting the dissected nodes, assign the corresponding geodesic distance values ​​in the path length matrix of the mesh surface as weight parameters to the topological edge, define the adjacency relationship and distance metric constraints between nodes, and generate a topological map of geodesic distances on the surface of the bimodal tubular manifold. For each node in the set of key anatomical feature nodes, a graph vertex object is created. Based on the connectivity logic of the anatomical structure (e.g., the fundus of the stomach is connected to the body of the stomach, and the body of the stomach is connected to the antrum of the stomach), a topological edge structure is established between the corresponding graph vertices. For each established topological edge, the geodesic distance value connecting the two endpoint nodes of the edge is indexed from the path length matrix on the mesh surface. The found geodesic distance value is used as a weight parameter and directly assigned to the topological edge.

[0027] Through this operation, each edge in the atlas no longer represents an abstract connection, but rather precisely encodes the actual physical distance traveled on the surface of the digestive tract soft tissue. Adjacency relationships and distance metric constraints between nodes are defined, stipulating that the relative positions between nodes must be constrained by this geodesic distance weight in subsequent deformation analysis to maintain the topological consistency of the anatomical structure. Finally, all graph vertices, topological edges, and their corresponding weight data are combined to generate a bimodal geodesic distance topological atlas of the luminal manifold surface. This atlas can simultaneously characterize the topological features of anatomical structures under both CT and endoscopic ultrasound modalities. For example, the atlas contains the nodes "pylorus" and "gastric angle." The geodesic distance between them is found to be 55.4 mm in the matrix. Therefore, an edge connecting "pylorus" and "gastric angle" is established, and its attribute Weight = 55.4 is set. This weight value will serve as a reference for rigid constraints in subsequent steps.

[0028] Please see Figure 4 The specific steps of S3 are as follows: S301: Call the geodesic distance topology map of the dual-modal lumen manifold surface, index the multiple topological edge structures connecting the map nodes, read the geodesic distance parameters of the topological edges in the computed tomography mode and the corresponding distance parameters in the endoscopic ultrasound mode, perform subtraction operation on the corresponding parameters, obtain the geometric length deviation between modes, perform square processing on the geometric length deviation, quantify the elastic deformation degree of the edge length, and obtain the topological connection edge difference measurement data between modes; The topological map of geodesic distances on the surface of the bimodal lumen manifold is accessed, and all topological edge structures defined in the map are traversed. For each specific topological edge (e.g., the edge connecting node i and node j), the geodesic distance parameter calculated in the computed tomography (CT) mode, denoted as L_CT, and the corresponding geodesic distance parameter in the endoscopic ultrasound (EUS) mode, denoted as L_EUS, are retrieved from the database. Then, a subtraction operation is performed to calculate the difference between the two (L_CT minus L_EUS) to obtain the geometric length deviation between the modes. This deviation reflects the amount of soft tissue stretching or compression caused by endoscopic device pressure or organ peristalsis. Next, the geometric length deviation value is squared, i.e., the square of (L_CT - L_EUS) is calculated. The purpose of squaring is to eliminate the influence of the sign of the deviation and to impose a larger numerical penalty on larger deformation deviations, thereby effectively quantifying the degree of elastic deformation of the edge length. The squared deviation values ​​of each topological edge are recorded and summarized to construct a numerical list containing all edge length deformation information, thereby obtaining the intermodal topological connection edge difference measurement data.

[0029] Table 2 shows the calculation results of the difference metric for some topological edges. Taking the "gastric body-antrum" connection edge as an example, its geodesic distance is 45.0 mm in the CT modality, while in the EUS modality, due to the bending and compression of the endoscope, the measured distance is 42.0 mm. Subtraction yields a deviation of 3.0 mm. Squaring 3.0 gives 9.0 square millimeters. This value of 9.0 is the elastic deformation metric for this connection edge, used for subsequent potential energy calculations.

[0030] Table 2: Metrics for Differences in Topological Connection Edges Between Modalities S302: For the difference measurement data of topological connection edges between modes, the square deviation values ​​of the edge lengths in the set are accumulated and summed to establish an elastic potential energy function describing the overall deformation degree of the anatomical structure. Partial differential operation is performed on the elastic potential energy function with respect to the spatial three-dimensional coordinate variables of each graph structure node. The rate of change of the elastic potential energy function at the real-time node coordinate position is analyzed to determine the vector direction and step size parameter that drives the node to evolve towards the energy minimum state, and a set of elastic potential energy spatial gradient vectors is generated. By summing all squared deviations in the intermodal topological connection edge difference measurement data and multiplying by an elasticity coefficient (e.g., 0.5), an elastic potential energy function describing the overall deformation of the anatomical structure is established. This function represents the total virtual energy required to forcibly stretch or compress the topology in the EUS mode to match the CT mode structure. To minimize this total energy and find the most natural deformation matching state, the gradient of the energy function with respect to the node coordinates needs to be calculated. Specifically, partial derivatives of the elastic potential energy function with respect to the spatial three-dimensional coordinate variables (x, y, z) of each graph structure node are performed. The specific logic of this operation is as follows: for each node, the contribution rate of the change in the length of its connection edge with adjacent nodes to the total energy is analyzed. According to the chain rule, the partial derivatives of the energy function with respect to the X, Y, and Z coordinates of the node are calculated. These partial derivative values ​​constitute the energy gradient vector of the node at its current position. This gradient vector indicates the direction of the fastest energy increase. To reduce energy, the direction of the vector driving node evolution is determined to be the opposite direction of the gradient (i.e., the negative gradient direction). Simultaneously, a step size parameter (or learning rate, e.g., 0.01) is set to control the distance moved in each iteration. The negative gradient vectors and step size information of all nodes are then aggregated to generate a set of elastic potential energy space gradient vectors.

[0031] S303: Based on the set of elastic potential energy spatial gradient vectors, the initial coordinates of the anatomical marker nodes in the topological map of the geodesic distance of the dual-modal tubular manifold surface are used as the reference. The coordinate displacement correction operation is performed along the opposite direction of the gradient vector. The spatial position of the node in the three-dimensional Cartesian system is iteratively updated until the potential energy reaches a stable state, generating a non-rigid deformation mapping field of the multimodal anatomical structure. Using the initial coordinates of nodes in the EUS mode of the geodesic distance topology map of the bimodal tubular manifold surface as the baseline state, an iterative loop is entered: In each iteration, the corresponding set of elastic potential energy spatial gradient vectors is called to obtain the current displacement correction vector of each node (i.e., the negative gradient direction multiplied by the step size). This correction vector is directly superimposed on the current three-dimensional coordinates of the node to update the spatial position of the node in the three-dimensional Cartesian system. After each update, the length of all edges and the total elastic potential energy are recalculated. The change in total potential energy in the current iteration step is compared with the change in total potential energy in the previous iteration step. If the absolute value of the energy change is less than the preset convergence threshold (e.g., 10 to the power of -6), it is determined that the potential energy has reached a stable state, and the iteration is terminated; otherwise, the next round of gradient calculation and coordinate correction is continued. When the iteration terminates, the displacement vectors of all nodes from the initial state to the final stable state are recorded, and the displacement relationships of these sparse nodes are extended to the entire space using thin plate spline interpolation (TPS) or radial basis function (RBF) to construct a dense coordinate transformation field and generate a non-rigid deformation mapping field of the multimodal anatomical structure.

[0032] Please see Figure 5 The specific steps of S4 are as follows: S401: Call the multimodal anatomical structure non-rigid deformation mapping field to obtain the original three-dimensional voxel mesh parameters of the ultrasound endoscopic scanning sequence, extract the initial spatial coordinate vector of the voxel vertices inside the mesh one by one, apply the defined nonlinear coordinate transformation rules and displacement parameters to perform geometric transformation operations on the initial spatial coordinate vectors, solve the new spatial position coordinates of the voxel points mapped to the computed tomography reference system, and generate the ultrasound endoscopic voxel remapping coordinate index table; Obtain the dimensions (e.g., 512x512x200 voxels) of the original 3D voxel mesh constructed from the endoscopic ultrasound scan sequence. Traverse each voxel vertex within the mesh and extract its initial spatial coordinate vector (x_eus, y_eus, z_eus) in the original EUS coordinate system. Then, substitute these initial coordinates into the transformation function of the deformation mapping field. Specifically, based on thin-plate spline interpolation logic, calculate the radial distance of the point relative to all control nodes (i.e., the aforementioned anatomical marker nodes), and combine this with the displacement coefficient matrix of the control nodes to calculate the displacement (dx, dy, dz) of the point under non-rigid deformation. Perform a geometric transformation operation, adding the initial coordinates to the calculated displacement (x_new=x_eus+dx, y_new=y_eus+dy, z_new=z_eus+dz) to obtain the new spatial coordinates of the voxel point mapped to the computed tomography (CT) reference system. Establish a one-to-one mapping record between each original voxel index (i, j, k) and its calculated new floating-point coordinates (x_new, y_new, z_new) to generate an endoscopic ultrasound voxel remapping coordinate index table.

[0033] S402: For the endoscopic ultrasound voxel remapping coordinate index table, retrieve the echo texture intensity values ​​of pixels in the endoscopic ultrasound scanning sequence, fill the echo texture intensity values ​​into the new spatial position after conversion according to the mapping index relationship, and for spatial points that fall into non-integer coordinate grids, calculate the distance weighted values ​​of the echo intensity of the surrounding neighboring voxels and perform interpolation assignment to generate deformation-corrected echo texture data volume. The echo intensity value (grayscale value, e.g., 0-255) of the corresponding pixel in the original endoscopic ultrasound (EUS) scan sequence is retrieved item by item from the EUS voxel remapping coordinate index table. Based on the mapping relationship, the echo intensity value is attempted to be filled into a new position in the CT coordinate system. Since the calculated new spatial position coordinates are non-integer (floating-point numbers), they cannot be directly mapped to integer grid nodes in the CT data; therefore, inverse interpolation is required. Specifically, the target integer grid nodes in the CT space are traversed. For each target node, its corresponding floating-point position in the original EUS space is found using the inverse mapping relationship of the index table. The echo intensity values ​​of the 8 nearest original voxel nodes around this floating-point position are retrieved. The spatial distance from this floating-point position to these 8 neighboring nodes is calculated, and weight parameters are assigned based on the reciprocal of the distance. Trilinear interpolation is performed: the echo intensity values ​​of the 8 neighboring nodes are weighted and summed with the corresponding weight parameters to obtain the interpolated echo intensity of the target grid point. This operation smoothly transfers the texture information from EUS into the geometric framework of CT, generating a deformation-corrected echo texture data volume.

[0034] S403: Based on the deformation-corrected echo texture data volume, the spatial coordinate positions of the internal voxels are traversed, the tissue radiation density values ​​corresponding to the coordinate positions in the computed tomography scan slice sequence are read, and the tissue radiation density values ​​and the deformation-corrected echo texture values ​​at the same position are concatenated through a channel to construct a composite voxel unit that combines anatomical structure density information and pathological texture information, thereby generating a spatially aligned multimodal fusion feature data volume. Following a uniform spatial grid step size (e.g., 0.5 mm), the spatial coordinates of each voxel within the deformation-corrected echo texture data volume are traversed. For each coordinate point, the ultrasound echo intensity value (denoted as I_EUS) stored in the deformation-corrected echo texture data volume is first directly read. Simultaneously, the registered computed tomography slice sequence is accessed to read the tissue radiometric density value (CT value, denoted as I_CT) at the same coordinate location.

[0035] To enhance the model's semantic perception of lesion regions, not only is physical layer intensity information fused, but high-dimensional semantic features are further introduced. A pre-trained deep semantic segmentation network (such as DeepLabV3+) is invoked to extract intermediate layer feature maps (F_CT and F_EUS) at the current voxel position for the CT and EUS modalities, respectively. These feature vectors encode local texture patterns and global spatial context information. Subsequently, a channel concatenation operation is performed: an expanded vector container is constructed, with I_CT as the first channel data and I_EUS as the second channel data. The extracted semantic feature vectors F_CT and F_EUS are then concatenated to form a multi-dimensional composite voxel unit. This composite voxel unit not only contains deep tissue density and anatomical structure information provided by the CT modality, but also high-resolution mucosal texture and echo feature information provided by the EUS modality, as well as abstract semantic expressions extracted by the deep network. Perform this operation on all voxels to merge the originally independent unimodal scalar fields into a bimodal vector field rich in semantic information, generating a spatially aligned multimodal fused feature data volume.

[0036] Please see Figure 6 The specific steps of S5 are as follows: S501: Based on spatially aligned multimodal fusion feature data volume, the tissue radiation density value and echo texture intensity value stored at the voxel location are parsed. After normalizing the two sets of values, they are input into the multivariate joint probability calculation formula. For each voxel, the confidence score of the feature vector mapping to the tumor lesion category is solved, and the joint probability distribution matrix of voxel lesion category is generated. A multidimensional feature vector [I_CT, I_EUS, and their corresponding semantic feature components] is extracted for each voxel. To eliminate dimensional differences, normalization (e.g., Min-Max normalization) is performed on these two sets of values, mapping the CT value to the 0-1 interval and the EUS echo intensity to the 0-1 interval. Then, the normalized feature vector is input into the pixel classification layer of the image semantic segmentation model. This classification layer uses either the Softmax or Sigmoid activation function to calculate the joint probability of multiple variables. The specific logic of this formula is as follows: calculate the dot product of the feature vector and the lesion category weight vector, add a bias term, and then convert the result into a probability value using the Sigmoid or Softmax function. The weight vector is obtained through supervised training on a large number of labeled samples, reflecting the contribution of different modal features to tumor determination. For example, the semantic segmentation model comprehensively considers the density features of CT and the texture features of EUS, as well as their spatial semantic correlation. If a tumor exhibits a specific density on CT and low echogenicity on EUS, the corresponding weight combination will result in voxels conforming to this pattern receiving a higher probability output. For each voxel, a confidence score (between 0 and 1) for belonging to the "tumor lesion category" is calculated, and this score is filled into the matrix position corresponding to the original space to generate the joint probability distribution matrix of voxel lesion categories.

[0037] S502: For the joint probability distribution matrix of voxel lesion categories, a probability judgment benchmark threshold is set to distinguish lesion tissue. The confidence score of each voxel in the joint probability distribution matrix of voxel lesion categories is compared with the benchmark threshold. The spatial coordinate index of voxels with confidence scores exceeding the benchmark threshold is extracted. The voxel attribute corresponding to the spatial coordinate index is marked as a positive foreground object, and the remaining voxels are marked as negative background objects, generating a binary label set of tumor foreground region. A strict probability threshold is set, which is based on a balance between sensitivity and specificity (e.g., 0.65). Each element in the joint probability distribution matrix of voxel lesion categories is iterated through. For each voxel, its confidence score output by the semantic segmentation model is read and compared with 0.65. If the confidence score of the voxel is greater than or equal to 0.65, it is determined that the voxel has a very high probability of belonging to tumor tissue, its spatial coordinate index is extracted, and its attribute is marked as "1" (positive foreground object); if the confidence score is less than 0.65, it is determined to be normal tissue or background, and its attribute is marked as "0" (negative background object). This process realizes the mapping from continuous probability space to discrete semantic categories. Through this item-by-item filtering process, the continuous probability map is transformed into discrete binary mask data, establishing a list containing the coordinates of all voxels marked as "1", generating a binary label set for the tumor foreground region.

[0038] S503: Call the set of binary markers for the tumor foreground region, establish a three-dimensional mask data structure with the same dimension as the original scan sequence based on the recorded coordinates of the positive foreground object, fill the voxel position corresponding to the foreground coordinate with the highlighted label value, fill the other positions with the zero background, identify the exact distribution range and edge contour information of the tumor tissue in the three-dimensional anatomical space, and generate a pixel-level segmentation mask for digestive tract tumor lesions. A three-dimensional array (e.g., 512x512x200) with dimensions identical to the original computed tomography slice sequence is initialized in memory, with all elements initialized to 0. Next, the binarized label set for the tumor foreground region is invoked, and the coordinates of all positive foreground objects recorded in the set are traversed. For each coordinate in the set, the corresponding element value in the three-dimensional array is modified to a specific label value (e.g., 255 or 1). To optimize the connectivity and smoothness of the segmentation results, a post-processing mechanism based on Conditional Random Fields (CRF) or morphological closing operation (dilation followed by erosion) is further introduced. This mechanism utilizes semantic consistency constraints between pixels to correct isolated noise, micro-holes, and jagged boundaries generated during binarization, ensuring that the generated tumor region is topologically complete and continuous. After filling and optimizing all foreground points, this three-dimensional array constitutes a solid mask describing the tumor morphology. The final array is the pixel-level segmentation mask for gastrointestinal tumor lesions.

[0039] Please see Figure 7 A digestive tract tumor lesion image segmentation system based on multimodal fusion technology, including: The 3D mesh construction module collects computed tomography scan slices and endoscopic ultrasound scan sequences, extracts contour data and converts it into a 3D point cloud, constructs a triangular patch index connecting adjacent point clouds, and generates a 3D triangular mesh model of the surface of digestive organs. The topology map building module, based on the three-dimensional triangular mesh model of the surface of digestive tract organs, maps the anatomical marker positions as nodes, performs a breadth-first search along the mesh edge and accumulates the physical edge length, obtains the connection weight, and builds a topology map of geodesic distances on the surface of a bimodal lumen manifold. The deformation mapping field generation module compares the differences in geodesic distances between the topological map connecting edges of the dual-modal tubular manifold surface, calculates the total elastic potential energy based on the sum of squared differences, corrects the node coordinates along the opposite direction of the gradient, and obtains the non-rigid deformation mapping field of the multimodal anatomical structure. The multimodal feature fusion module calculates the corresponding coordinates of ultrasound endoscopic voxels in the tomographic scanning space and fills the echo texture data based on the non-rigid deformation mapping field of multimodal anatomical structures, and establishes a spatially aligned multimodal fusion feature data volume. The lesion segmentation mask generation module screens spatially aligned multimodal fusion feature data volume voxel features, calls a conditional random field model to calculate the joint probability value of voxels belonging to tumor lesions, and generates pixel-level segmentation masks for gastrointestinal tumor lesions.

[0040] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of protection of the technical solution.

Claims

1. A method for segmenting images of gastrointestinal tumor lesions based on multimodal fusion technology, characterized in that, Includes the following steps: S1: Collect computed tomography scan slices and endoscopic ultrasound scan sequences, extract contour data and convert them into three-dimensional point clouds, construct triangular facet indexes connecting adjacent point clouds, and generate a three-dimensional triangular mesh model of the surface of digestive organs. S2: Based on the three-dimensional triangular mesh model of the surface of the digestive tract organs, map the anatomical marker positions as nodes, perform a breadth-first search along the mesh edge and accumulate the physical edge length, obtain the connection weight, and establish a topological map of geodesic distances on the surface of the bimodal lumen manifold. S3: Compare the differences in geodesic distances between the topological map connecting edges of the dual-modal tubular manifold surface, calculate the total elastic potential energy based on the sum of squared differences, correct the node coordinates along the opposite direction of the gradient, and obtain the non-rigid deformation mapping field of the multimodal anatomical structure. S4: Based on the non-rigid deformation mapping field of the multimodal anatomical structure, calculate the corresponding coordinates of the ultrasound endoscope voxels in the tomographic scanning space and fill the echo texture data to establish a spatially aligned multimodal fusion feature data volume; S5: Screen the voxel features of the spatially aligned multimodal fusion feature data, calculate the joint probability value of the voxel belonging to the tumor lesion, and generate a pixel-level segmentation mask for the gastrointestinal tumor lesion.

2. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 1, characterized in that, The three-dimensional triangular mesh model of the digestive tract organ surface includes a set of vertex spatial coordinates, a list of triangular facet topological indexes, and surface normal vector data. The bimodal luminal manifold surface geodesic distance topological map includes a set of anatomical key point nodes, a list of adjacent edges of the manifold surface, and a geodesic distance weight matrix. The multimodal anatomical structure non-rigid deformation mapping field includes a set of node spatial displacement vectors, a non-rigid coordinate transformation matrix, and a minimized elastic potential energy parameter. The spatially aligned multimodal fusion feature data volume includes spatially registered voxel coordinates, CT tissue density channel data, and EUS echo texture channel data. The pixel-level segmentation mask of the digestive tract tumor lesion includes a tumor region binarized marker map, a lesion edge contour index, and a set of foreground pixel positions.

3. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 1, characterized in that, The specific steps of S1 are as follows: S101: Collect computed tomography (CT) scan slice sequences and endoscopic ultrasound scan sequences of the digestive tract, scan the pixel gray-level gradient distribution of the slice plane, identify the location of gray-level abrupt changes, locate the edge pixels of the inner and outer walls of the organ, extract the two-dimensional plane coordinates of the edge pixels in the slice coordinate system, stack and arrange the multi-layer coordinate data according to the slice scan layer sequence, and generate a multimodal tomographic contour dataset. S102: Call the multimodal tomographic contour dataset, extract the scanning interlayer spacing and layer thickness parameters corresponding to the slice sequence, perform longitudinal Z-axis mapping transformation by combining the two-dimensional pixel coordinates of the contour points with the interlayer spacing values, convert the planar coordinates into spatial coordinates under a unified three-dimensional Cartesian coordinate system, perform interpolation processing on sparse regions, densify the distribution of points, and establish a discrete point cloud coordinate set. S103: For the discrete point cloud coordinate set, calculate the Euclidean distance between the spatial point cloud and the neighboring point cloud, determine the spatial topological adjacency relationship of the point cloud according to the principle of closest distance, connect the three adjacent point cloud coordinates in counterclockwise order, construct the triangular patch topological index, close and combine the triangular patches, fit the geometric surface of the organ, and generate a three-dimensional triangular mesh model of the surface of the digestive tract organ.

4. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 3, characterized in that, The specific steps of S2 are as follows: S201: Based on the three-dimensional triangular mesh model of the digestive tract organ surface, retrieve the geometric curvature feature data of the mesh surface, locate the mesh vertex index representing the position of the pyloric ring and the tumor center, map the mesh vertex index to discrete nodes in the graph theory structure, extract the Cartesian coordinate parameters of the discrete nodes in three-dimensional space, and generate a set of key anatomical feature nodes. S202: Call the set of key anatomical feature nodes and the three-dimensional triangular mesh model of the surface of digestive organs, take the nodes in the set as the starting endpoint of the search, traverse the adjacent vertices layer by layer along the common edge of the triangular facet, calculate the Euclidean distance between the coordinates of the two vertices at the two ends of the mesh edge to determine the physical edge length, accumulate the physical length of the mesh edge traversed by the search path, select the path with the smallest accumulated value between two nodes as the geodesic distance of the manifold surface, and generate the mesh surface path length matrix. S203: For the path length matrix of the mesh surface, construct a topological edge structure connecting the dissected nodes, assign the corresponding geodesic distance values ​​in the path length matrix of the mesh surface as weight parameters to the topological edges, define the adjacency relationship and distance metric constraints between nodes, and generate a topological map of geodesic distances on the surface of the bimodal tubular manifold.

5. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 4, characterized in that, The specific steps for S3 are as follows: S301: Call the geodesic distance topology map of the dual-modal lumen manifold surface, index the multiple topological edge structures connecting the map nodes, read the geodesic distance parameters of the topological edges in the computed tomography mode and the corresponding distance parameters in the endoscopic ultrasound mode, perform subtraction operation on the corresponding parameters, obtain the geometric length deviation between modes, perform square processing on the geometric length deviation, quantify the elastic deformation degree of the edge length, and obtain the topological connection edge difference measurement data between modes; S302: For the difference measurement data of the topological connection edges between the modes, the square deviation values ​​of the edge lengths in the set are accumulated and summed to establish an elastic potential energy function describing the overall deformation degree of the anatomical structure. Partial differential operation is performed on the elastic potential energy function with respect to the spatial three-dimensional coordinate variables of each graph structure node. The rate of change of the elastic potential energy function at the real-time node coordinate position is analyzed to determine the vector direction and step size parameter of the driving node to evolve towards the energy minimum state and generate a set of elastic potential energy spatial gradient vectors. S303: Based on the set of elastic potential energy spatial gradient vectors, the initial coordinates of the anatomical marker nodes in the topological map of the geodesic distance of the dual-modal tubular manifold surface are used as a reference. The coordinate displacement correction operation is performed along the opposite direction of the gradient vector. The spatial position of the node in the three-dimensional Cartesian system is iteratively updated until the potential energy reaches a stable state, thereby generating a non-rigid deformation mapping field of the multimodal anatomical structure.

6. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 5, characterized in that, The process of establishing the elastic potential energy function describing the overall deformation of the anatomical structure is as follows: call the preset tissue mechanics parameter library to obtain the local stiffness weighting coefficient that simulates the characteristics of biological soft tissue, and perform point-to-point multiplication operation on the square value of the geometric length deviation included in the intermodal topological connection edge difference measurement data and the local stiffness weighting coefficient to generate the weighted unilateral elastic potential energy value. Traverse all connected edge structures in the topological map of geodesic distances on the surface of the bimodal tubular manifold, perform a global accumulation operation on the weighted single-sided elastic potential energy values, and calculate the discrete Laplace coordinate vectors of the map nodes to characterize the local curvature of the mesh. Construct a composite objective function including a deformation energy accumulation term and a Laplace smoothing regularization term as the elastic potential energy function.

7. The method for segmenting gastrointestinal tumor lesion images based on multimodal fusion technology according to claim 5, characterized in that, The specific steps of S4 are as follows: S401: Call the non-rigid deformation mapping field of the multimodal anatomical structure to obtain the original three-dimensional voxel mesh parameters of the ultrasound endoscopic scanning sequence, extract the initial spatial coordinate vector of the voxel vertices inside the mesh one by one, apply the defined nonlinear coordinate transformation rules and displacement parameters to perform geometric transformation operations on the initial spatial coordinate vectors, solve the new spatial position coordinates of the voxel points mapped to the computed tomography reference system, and generate an ultrasound endoscopic voxel remapping coordinate index table; S402: For the ultrasound endoscope voxel remapping coordinate index table, retrieve the echo texture intensity value of the pixel in the ultrasound endoscope scanning sequence, fill the echo texture intensity value into the new spatial position after conversion according to the mapping index relationship, and for spatial points that fall into the non-integer coordinate grid, calculate the distance weighted value of the echo intensity of the surrounding neighboring voxels and perform interpolation assignment operation to generate deformation-corrected echo texture data volume. S403: Based on the deformation-corrected echo texture data volume, traverse the spatial coordinate positions of the internal voxels, read the tissue radiation density values ​​corresponding to the coordinate positions in the computed tomography scan slice sequence, perform channel cascading operation on the tissue radiation density values ​​and the deformation-corrected echo texture values ​​at the same positions, construct a composite voxel unit that combines anatomical structure density information and pathological texture information, and generate a spatially aligned multimodal fusion feature data volume.

8. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 7, characterized in that, The process of calculating the distance-weighted value of the echo intensity of the surrounding neighborhood voxels and performing interpolation is as follows: based on the coordinate values ​​of the spatial point that falls into the non-integer coordinate grid, lock the eight nearest integer grid nodes surrounding the spatial point in the three-dimensional space, and read the original ultrasonic echo intensity stored in the nearest integer grid nodes. Calculate the projection distance deviation of a spatial point relative to its nearest integer grid node in the three axes of the Cartesian coordinate system, and construct the inverse distance volume weight coefficient of the corresponding node based on the projection distance deviation. The original ultrasonic echo intensity of the nearest integer grid node is weighted and summed with the corresponding inverse distance volume weighting coefficient to output the smooth echo texture value of the spatial point.

9. The method for segmenting digestive tract tumor lesion images based on multimodal fusion technology according to claim 7, characterized in that, The specific steps of S5 are as follows: S501: Based on the spatially aligned multimodal fusion feature data volume, the tissue radiation density value and echo texture intensity value stored at the voxel location are parsed. After normalizing the two sets of values, they are input into the multivariate joint probability calculation formula. For each voxel, the confidence score of the feature vector mapped to the tumor lesion category is calculated to generate the joint probability distribution matrix of voxel lesion categories. S502: For the voxel lesion category joint probability distribution matrix, set a probability judgment benchmark threshold for distinguishing lesion tissue, compare the confidence score of voxels in the voxel lesion category joint probability distribution matrix with the benchmark threshold item by item, extract the voxel spatial coordinate index of the voxel whose confidence score exceeds the benchmark threshold, mark the voxel attribute corresponding to the spatial coordinate index as a positive foreground object, mark the remaining voxels as negative background objects, and generate a set of binary labels for the tumor foreground region; S503: Call the set of binary markers for the tumor foreground region, establish a three-dimensional mask data structure with the same dimension as the original scan sequence based on the recorded coordinates of the positive foreground object, fill the voxel positions corresponding to the foreground coordinates with the highlighted label values, fill the remaining positions with the zero-value background, identify the exact distribution range and edge contour information of the tumor tissue in the three-dimensional anatomical space, and generate a pixel-level segmentation mask for digestive tract tumor lesions.

10. A digestive tract tumor lesion image segmentation system based on multimodal fusion technology, characterized in that, The system is used to implement the digestive tract tumor lesion image segmentation method based on multimodal fusion technology as described in any one of claims 1-9, and the system comprises: The 3D mesh construction module collects computed tomography scan slices and endoscopic ultrasound scan sequences, extracts contour data and converts it into a 3D point cloud, constructs a triangular patch index connecting adjacent point clouds, and generates a 3D triangular mesh model of the surface of digestive organs. The topology map building module, based on the three-dimensional triangular mesh model of the surface of the digestive tract organs, maps the anatomical marker positions as nodes, performs a breadth-first search along the mesh edge and accumulates the physical edge length, obtains the connection weight, and builds a topology map of geodesic distances on the surface of the bimodal lumen manifold. The deformation mapping field generation module compares the differences in geodesic distances from the topological map connecting edges of the dual-modal tubular manifold surface, calculates the total elastic potential energy based on the sum of squared differences, corrects the node coordinates along the opposite direction of the gradient, and obtains the non-rigid deformation mapping field of the multimodal anatomical structure. The multimodal feature fusion module calculates the corresponding coordinates of the ultrasound endoscopic voxels in the tomographic scanning space and fills the echo texture data based on the non-rigid deformation mapping field of the multimodal anatomical structure, and establishes a spatially aligned multimodal fusion feature data volume. The lesion segmentation mask generation module screens the voxel features of the spatially aligned multimodal fusion feature data, calls the conditional random field model to calculate the joint probability value of voxels belonging to tumor lesions, and generates a pixel-level segmentation mask for digestive tract tumor lesions.

Images