A gastric cancer multi-modal large model assisted staging system

Abstract

The application discloses a gastric cancer multi-modal large model assisted staging system, relates to the field of medical image intelligent analysis, and has the technical scheme that: an image acquisition module is used to acquire the expanded state two-dimensional endoscope image and the natural state three-dimensional radiographic image of a patient, a lesion extraction module is used to extract two-dimensional lesion coordinates and a two-dimensional local lesion feature matrix, a topological anchor point construction module is used to construct a spatial topological anchor point network based on two-dimensional anatomical invariant markers and three-dimensional anatomical invariant markers, a three-dimensional morphology reverse restoration module is used to output natural state three-dimensional spatial coordinates, a multi-modal alignment module is used to generate spatially aligned multi-modal features, and a large model staging prediction module is used to output gastric cancer staging prediction results. The application solves the problem that expanded state endoscopic lesion features are difficult to accurately correspond to natural state three-dimensional radiographic images.

Description

Technical Field

[0001] This application relates to the field of intelligent medical image analysis technology, and more specifically, to a multimodal large-scale model-assisted staging system for gastric cancer. Background Technology

[0002] Gastric cancer is one of the most common malignant tumors of the digestive system. Clinical diagnosis and treatment typically require a comprehensive assessment of lesions using a combination of endoscopic examination, radiological imaging, and pathological biopsy results. Endoscopic images clearly reflect the color, boundaries, elevation, ulceration, and mucosal texture of lesions within the gastric lumen, making them suitable for observing the morphology of the lesion surface. Radiological imaging such as CT and MRI can reflect gastric wall thickness, the spatial extent of the lesion, its relationship with surrounding tissues, and enhancement patterns, making them suitable for observing the distribution of lesions in three-dimensional space. Pathological biopsy results reflect histological type, degree of differentiation, and local cytological features. This multi-source information is of significant reference value for the auxiliary staging of gastric cancer.

[0003] In the prior art, patent CN113707327A, entitled "A Multi-Tumor Marker Tumor Diagnostic Model Based on Medical Big Data," discloses a technical solution that organizes and models multiple tumor marker data through modules such as data mining, model building, judgment analysis, and system development, and combines models such as logistic regression, random forest, support vector machine, and neural network to achieve tumor-assisted judgment. This solution also involves the development of prediction models for different tumors such as thyroid cancer, lung cancer, liver cancer, and gastric cancer, mainly focusing on the comprehensive analysis of medical big data and tumor marker data.

[0004] Patent CN115579094A, entitled "A Method and System for Constructing a Multimodal Medical Data Lake," discloses a technical solution for acquiring clinical data and aggregating it into a data lake, further acquiring multimodal medical data such as imaging, pathology, and genes, and integrating clinical data with multimodal data based on patient information and medical visit information to construct a multimodal medical data model. This solution is compatible with medical data in different formats, which is beneficial for the centralized storage, integration, and subsequent analysis of medical data.

[0005] However, in the context of auxiliary staging for gastric cancer, simply aggregating endoscopic images, radiographic images, or pathological biopsy results as multi-source data is still insufficient to meet the needs of refined staging. During gastroscopy, air or water is typically injected into the stomach cavity to expand the stomach wall. The location and morphology of lesions in endoscopic images correspond to the expanded surface of the stomach cavity. Radiographic images, on the other hand, usually reflect the three-dimensional structure of the stomach in its natural or near-natural state. Because the stomach wall is a flexible tissue, significant spatial deformation occurs between the expanded and natural states, making it difficult to directly correlate the location of lesions seen in endoscopic images with the actual spatial region in three-dimensional radiographic images.

[0006] Furthermore, while two-dimensional endoscopic images provide clear local surface features, they lack information on the depth direction of the gastric wall and its relationship with surrounding tissues. Three-dimensional radiographic images, while possessing spatial structural information, have limited ability to represent the subtle textures of the lesion surface. If the two types of images are not accurately spatially mapped, subsequent models may easily map local lesion features from the endoscopy to incorrect three-dimensional locations, thus affecting the judgment of invasion depth, boundary extent, and staging results. Although pathological biopsy results have high clinical value, biopsies typically only represent local sampling points. If there is a difference between the sampling location and the overall invasion area of ​​the lesion, directly fusing the pathological biopsy results with imaging information may also cause the staging basis to deviate from the actual lesion extent. Summary of the Invention

[0007] The technical problem to be solved by this application is how to handle the deformation difference between expanded two-dimensional endoscopic images and natural three-dimensional radiographic images in the auxiliary staging of gastric cancer, how to make the local lesion features of the endoscope accurately fall into the three-dimensional voxel space of the radiographic image, and how to improve the spatial consistency and reliability of multi-source medical information fusion.

[0008] To achieve the above objectives, this application adopts the following technical solution: A multimodal large-scale model-assisted staging system for gastric cancer includes: an image acquisition module for acquiring the patient's expanded two-dimensional endoscopic image and natural three-dimensional radiographic image; The lesion extraction module is used to extract two-dimensional lesion coordinates and two-dimensional local lesion feature matrix from the expanded two-dimensional endoscopic image; A topological anchor point construction module is used to extract two-dimensional and three-dimensional anatomically invariant markers from the expanded two-dimensional endoscopic image and the natural three-dimensional radiographic image, respectively. Based on the two-dimensional and three-dimensional anatomically invariant markers, a spatial coordinate registration relationship is established to construct a spatial topological anchor point network. The nodes in the spatial topological anchor point network include the two-dimensional gastric angle center point, the two-dimensional pyloric ring center point, the three-dimensional gastric angle center point, the three-dimensional pyloric ring center point, and the two-dimensional lesion point to be restored. The edges in the spatial topological anchor point network are used to express the structural connection relationship, relative position relationship, and cross-modal correspondence relationship between the nodes. A three-dimensional morphological inverse reconstruction module is used to input the spatial topological anchor point network and the two-dimensional lesion coordinates into a pre-trained gastric wall biomechanical morphological reconstruction model. The gastric wall biomechanical morphological reconstruction model performs a spatial deformation inverse mapping operation on the two-dimensional lesion coordinates and outputs the natural three-dimensional spatial coordinates corresponding to the natural three-dimensional radiographic image. The gastric wall biomechanical morphological reconstruction model is a graph neural network model. The graph neural network model uses the spatial topological anchor point network as the input graph structure. The node features of the graph neural network model include the modality identifier of the node and the relative displacement features between nodes. The graph neural network model includes a graph convolutional layer, a message passing layer, and a coordinate regression layer. The graph convolutional layer is used to summarize the positional constraints of adjacent nodes. The message passing layer is used to transmit the deformation information between anchor points to the lesion point. The coordinate regression layer is used to output the natural three-dimensional spatial coordinates. The multimodal alignment module is used to map the two-dimensional local lesion feature matrix to the voxel space region in the natural state three-dimensional radiographic image corresponding to the natural state three-dimensional spatial coordinates, thereby generating spatially aligned multimodal features; The large model staging prediction module is used to input the spatially aligned multimodal features into a pre-trained multimodal large model to perform feature aggregation and pathological staging classification calculations, and output gastric cancer staging prediction results.

[0009] Preferably, the topological anchor point construction module includes a two-dimensional marker extraction unit, a three-dimensional marker extraction unit, and an anchor point network generation unit. The two-dimensional marker extraction unit is used to input the expanded two-dimensional endoscopic image into a first-level convolutional keypoint detection network to perform structural feature extraction, and output the coordinates of the two-dimensional gastric angle center point and the two-dimensional pyloric ring center point as the two-dimensional anatomically invariant markers. The three-dimensional marker extraction unit is used to input the natural three-dimensional radiographic image into a second-level convolutional keypoint detection network to perform structural feature extraction, and output the coordinates of the three-dimensional gastric angle center point and the three-dimensional pyloric ring center point as the three-dimensional anatomically invariant markers. The anchor point network generation unit is used to establish the registration relationship between the two-dimensional gastric angle center point coordinates and the three-dimensional gastric angle center point coordinates, and to establish the registration relationship between the two-dimensional pyloric ring center point coordinates and the three-dimensional pyloric ring center point coordinates, thereby generating the spatial topological anchor point network.

[0010] Preferably, the system further includes a morphological reconstruction model training module. This module acquires a training dataset containing expanded two-dimensional endoscopic images and natural three-dimensional radiographic images of historical cases. It extracts the coordinates of the gastric angle center point and the pyloric ring center point from the training dataset as graph structure node features and obtains the corresponding real natural three-dimensional spatial coordinates to construct a training sample library. The morphological reconstruction model training module also uses a supervised training algorithm, employing the Euclidean distance between the predicted deformation coordinates output by the graph neural network model and the calibrated real natural three-dimensional spatial coordinates in the training sample library as a regression loss function, and uses a gradient descent algorithm to iteratively update the neuron weights of the graph neural network model until the graph neural network model converges.

[0011] Preferably, the three-dimensional morphology reverse reconstruction module includes an elastic contraction deformation unit. The elastic contraction deformation unit is used to apply a finite element elastic contraction deformation algorithm in the gastric wall biomechanical morphology reconstruction model to perform tensor contraction deformation operation on the two-dimensional lesion coordinates based on the anchor point positions indicated by the spatial topological anchor point network, thereby generating natural three-dimensional spatial coordinates corresponding to the natural three-dimensional radiographic image.

[0012] Preferably, the multimodal alignment module includes a three-dimensional interpolation resampling unit, which is used to apply a three-dimensional spatial interpolation resampling algorithm to fill and transform the two-dimensional local lesion feature matrix into the voxel grid structure of the natural three-dimensional radiographic image, thereby generating the spatially aligned multimodal features containing three-dimensional topological location information.

[0013] Preferably, the lesion extraction module includes a lesion segmentation unit, a lesion coordinate extraction unit, and a lesion feature extraction unit; the lesion segmentation unit is used to input the expanded state two-dimensional endoscopic image into a pre-trained lesion segmentation network and output a lesion region mask; the lesion coordinate extraction unit is used to extract the geometric center coordinates of the lesion region mask as the two-dimensional lesion coordinates; the lesion feature extraction unit is used to perform feature extraction operations on the image pixels covered by the lesion region mask using feature extraction operators to generate the two-dimensional local lesion feature matrix.

[0014] Preferably, the system further includes a microcirculation spatiotemporal dynamic alignment module, which is used to extract enhancement grayscale temporal data of the abdominal aortic region in the natural state three-dimensional radiographic image to form an individual circulatory baseline curve; the microcirculation spatiotemporal dynamic alignment module is also used to extract enhancement gradient temporal data of the gastric wall lesion region in the natural state three-dimensional radiographic image; the microcirculation spatiotemporal dynamic alignment module is also used to perform alignment mapping operation on the time axis dimension of the enhancement gradient temporal data to the individual circulatory baseline curve to generate a standardized layered enhancement tensor; the three-dimensional morphology inverse reconstruction module is also used to input the standardized layered enhancement tensor as a boundary constraint condition, together with the spatial topological anchor network and the two-dimensional lesion coordinates, into the gastric wall biomechanical morphology reconstruction model.

[0015] Preferably, the micro-circulation spatiotemporal dynamic alignment module includes a dynamic time warping unit. The dynamic time warping unit is used to construct a two-dimensional cumulative distance matrix between the reinforcement gradient time series data and the individual cyclic baseline curve using a dynamic time warping algorithm. The dynamic time warping unit is also used to find the sequence of consecutive alignment points in the two-dimensional cumulative distance matrix that minimizes the sum of element distances, and to use the sequence of consecutive alignment points as a time-warped alignment mapping path. The dynamic time warping unit is also used to perform stretching and compression transformation operations on the time axis dimension of the reinforcement gradient time series data using the time-warped alignment mapping path to generate the normalized hierarchical reinforcement tensor.

[0016] Preferably, the large-scale model staging prediction module includes a pathological text fusion unit. This unit is used to acquire pathological biopsy text associated with the expanded two-dimensional endoscopic image, and extract a word vector encoding matrix obtained by encoding the pathological biopsy text. The pathological text fusion unit is also used to acquire the two-dimensional puncture point coordinates corresponding to the pathological biopsy operation, input the two-dimensional puncture point coordinates and the spatial topological anchor network into the gastric wall biomechanical morphology restoration model to perform a spatial deformation inverse mapping operation, and output three-dimensional puncture point coordinates. Furthermore, the pathological text fusion unit is used to extract the first... The system generates a blood perfusion feature distribution and extracts a second blood perfusion feature distribution corresponding to the tumor margin extension region. The pathological text fusion unit is further used to perform a matrix difference operation on the first blood perfusion feature distribution and the second blood perfusion feature distribution to generate a relative difference matrix. The pathological text fusion unit is further used to input the relative difference matrix into a pre-trained multimodal adaptive gating network and output attention fusion weights adapted to the pathological biopsy text. The pathological text fusion unit is further used to multiply the attention fusion weights into the word vector encoding matrix, and then perform tensor concatenation and aggregation operations with the spatially aligned multimodal features within the multimodal large model.

[0017] This application also includes a gated network training module, which is used to input pre-acquired historical perfusion relative difference matrix samples into the network input layer, input manually labeled text confidence weights into the network loss layer, and use the backpropagation algorithm to adjust the weight matrix of the fully connected layer of the multimodal adaptive gated network.

[0018] The beneficial effects of this application are as follows: This application acquires expanded two-dimensional endoscopic images and natural three-dimensional radiographic images through an image acquisition module, obtains two-dimensional lesion coordinates and a two-dimensional local lesion feature matrix through a lesion extraction module, and then extracts two-dimensional and three-dimensional anatomically invariant markers through a topological anchor point construction module. Based on these two, a spatial coordinate registration relationship is established, and a spatial topological anchor point network is constructed. Since structures such as the gastric angle and pyloric ring have relatively stable anatomical reference significance under different examination states, the system can utilize these stable structures as spatial positioning references, reducing the impact of gastric dilatation, changes in imaging angle, and differences in image modality on positional correspondence.

[0019] This application further utilizes a three-dimensional morphological inverse reconstruction module to input a spatial topological anchor network and two-dimensional lesion coordinates into a gastric wall biomechanical morphological reconstruction model. This allows the lesion coordinates in the expanded two-dimensional endoscopic image to be converted into natural three-dimensional spatial coordinates corresponding to the natural three-dimensional radiographic image through spatial deformation inverse mapping. Since the gastric wall is a soft tissue with elastic deformation characteristics, the system utilizes biomechanical constraints and anatomical anchor point constraints during the gastric wall deformation process. This ensures that the reconstruction of lesion coordinates is no longer a simple planar projection or coarse positioning, but rather incorporates the morphological change patterns of the gastric wall returning from an expanded state to a natural state, improving the accuracy of the correspondence between the endoscopic lesion location and the three-dimensional image space.

[0020] This application also utilizes a multimodal alignment module to map the two-dimensional local lesion feature matrix to the voxel space region in the natural three-dimensional radiographic image corresponding to the natural three-dimensional spatial coordinates, generating spatially aligned multimodal features. Thus, the surface texture information in the endoscopic image can form a positionally consistent fusion feature with the three-dimensional spatial information in the radiographic image, avoiding the problem that different modalities, although belonging to the same patient, do not point to the same lesion region. After receiving the spatially aligned fusion features, the multimodal large model can perform staging and classification calculations based on a clearer three-dimensional positional relationship and local lesion expression, thereby improving the stability and interpretability of gastric cancer auxiliary staging results.

[0021] Furthermore, this application sets up a microcirculation spatiotemporal dynamic alignment module to extract the enhancement grayscale temporal data of the abdominal aorta region from natural-state 3D radiographic images as an individual circulatory baseline curve, and aligns the enhancement gradient temporal data of the gastric wall lesion region to this baseline curve along the time axis. Since the contrast agent circulation rate varies among different patients, directly comparing enhancement curves at absolute time can easily lead to bias. This application utilizes the technical characteristic that changes in abdominal aortic enhancement can reflect individual circulatory rhythms, adjusting the enhancement temporal sequence of the lesion region to an individual baseline for expression. This facilitates obtaining a more stable standardized layered enhancement tensor, which serves as a boundary constraint condition for gastric wall morphology reconstruction, providing a more reliable perfusion temporal basis for 3D coordinate reconstruction and lesion boundary judgment.

[0022] Furthermore, this application sets up a pathological text fusion unit and a multimodal adaptive gating network to restore the two-dimensional puncture point coordinates corresponding to the pathological biopsy operation to three-dimensional puncture point coordinates, and compares the differences in blood perfusion characteristics between the puncture point region and the tumor margin extension region. Since the pathological biopsy results originate from local sampling points, their representativeness of the overall lesion is related to the sampling location, lesion heterogeneity, and local perfusion status. This application uses the relative difference between the perfusion distribution at the puncture point and the perfusion distribution in the tumor margin extension region to generate attention fusion weights adapted to the pathological biopsy text, so that the influence of the pathological biopsy text in the multimodal large model can be adjusted according to the sampling representativeness, reducing the problem of over-application or underestimation of local biopsy information. Attached Figure Description

[0023] Figure 1 A flowchart illustrating the overall technical solution of a multimodal large-model-assisted staging system for gastric cancer provided in this application embodiment; Figure 2 This is an overall block diagram of the gastric cancer multimodal large model-assisted staging system provided in the embodiments of this application; Figure 3 A schematic diagram of two-dimensional / three-dimensional anatomical marker extraction and spatial topology anchor network construction provided in the embodiments of this application; Figure 4 This is a schematic diagram of three-dimensional morphological inverse reconstruction and deformation calculation based on topological anchor points provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the process of extracting coordinates and local feature matrices using the lesion extraction module provided in this embodiment of the application. Figure 6 This is a diagram of the supervised training process of the gastric wall biomechanical morphology reconstruction model (graph neural network) provided in the embodiments of this application. Figure 7 This is a schematic diagram of micro-circulation spatiotemporal dynamic alignment data extraction and alignment mapping provided in an embodiment of this application; Figure 8 This is a diagram showing the reverse mapping of three-dimensional puncture point coordinates and the extraction of relative differences in blood perfusion characteristics in the corresponding region, provided in the embodiments of this application. Figure 9 This is a diagram of the multimodal adaptive gated text fusion and large model phased prediction operation structure provided in the embodiments of this application. Detailed Implementation

[0024] The technical solution of this application will be described below with reference to the accompanying drawings. It should be understood that the following embodiments are used to illustrate the implementation of the system and are not intended to limit the scope of protection.

[0025] This application provides a multimodal large-model-assisted staging system for gastric cancer. Figure 1This application provides a flowchart of the overall technical solution for a multimodal large-scale model-assisted staging system for gastric cancer. The process includes S101 to S106, where S101 involves acquiring the patient's expanded two-dimensional endoscopic image and natural three-dimensional radiographic image; S102 involves extracting two-dimensional lesion coordinates and a two-dimensional local lesion feature matrix from the expanded two-dimensional endoscopic image; and S103 involves extracting two-dimensional and three-dimensional anatomically invariant markers from the expanded two-dimensional endoscopic image and the natural three-dimensional radiographic image, respectively, and establishing spatial coordinate matching based on the two types of markers. The process involves constructing a spatial topological anchor network, S104, inputting the spatial topological anchor network and two-dimensional lesion coordinates into the gastric wall biomechanical morphology restoration model, and outputting natural three-dimensional spatial coordinates through inverse spatial deformation mapping, S105, mapping the two-dimensional local lesion feature matrix to the voxel space region corresponding to the natural three-dimensional spatial coordinates in the natural three-dimensional radiographic image, generating spatially aligned multimodal features, and S106, inputting the spatially aligned multimodal features into the multimodal large model to perform feature aggregation and pathological staging classification calculations, and outputting the gastric cancer staging prediction results.

[0026] In one embodiment, the multimodal large-model-assisted staging system for gastric cancer can be deployed on a hospital imaging workstation, an endoscopy center server, or a hospital-owned private cloud computing platform. The system receives data including two-dimensional endoscopic images obtained during gastroscopy and natural-state three-dimensional radiographic images of the same patient obtained during CT or MRI examinations. The expanded-state two-dimensional endoscopic image refers to the image formed during gastroscopy when the stomach cavity and wall are in an expanded state due to inflation, water injection, or endoscopic manipulation. The natural-state three-dimensional radiographic image refers to the three-dimensional volume data formed when the stomach is in a natural or near-natural state during radiographic examinations; this can be enhanced CT volume data or enhanced MRI volume data. The system does not simply project the lesion location from the two-dimensional endoscopic image into the three-dimensional image. Instead, it first establishes the spatial correspondence between stable anatomical structures of the stomach and then reverse-engineers the lesion coordinates based on the deformation patterns of the stomach wall. This design aims to solve the spatial misalignment problem caused by the inconsistency between the endoscopic and radiographic examination states, ensuring that the local texture, spatial infiltration range, and enhancement manifestations used for subsequent staging prediction point to the same real lesion area.

[0027] Figure 2This is an overall block diagram of the gastric cancer multimodal large model-assisted staging system provided in this application embodiment. The system includes an image acquisition module, a lesion extraction module, a topological anchor construction module, a three-dimensional morphology inverse reconstruction module, a multimodal alignment module, and a large model staging prediction module. The image acquisition module receives the patient's expanded two-dimensional endoscopic images and natural three-dimensional radiographic images, and matches data from different sources by case number, examination time, and examination site. To avoid processing errors caused by inconsistent output formats from different devices, the image acquisition module can perform size normalization and brightness equalization processing on the two-dimensional endoscopic images, and voxel spacing unification, gastric region cropping, and grayscale standardization processing on the three-dimensional radiographic images. The two-dimensional endoscopic images can be white light endoscopic images, narrow-band imaging images, or chromoendoscopic images. The three-dimensional radiographic images can be enhanced CT images or enhanced MRI images. When multiple endoscopic frames exist for the same patient, the image acquisition module can select a frame containing complete lesion boundaries and whose clarity meets clinical analysis requirements as input, or it can input multiple frames into the lesion extraction module and fuse the lesion coordinates and lesion features corresponding to multiple frames.

[0028] The lesion extraction module is used to obtain two-dimensional lesion coordinates and a two-dimensional local lesion feature matrix from expanded two-dimensional endoscopic images. The two-dimensional lesion coordinates represent the location of the lesion in the expanded endoscopic field of view, while the two-dimensional local lesion feature matrix expresses local information such as color, boundary, texture, elevation morphology, and ulceration of the lesion region. This matrix does not simply store the original pixels, but rather stores the multi-channel features obtained after feature extraction, facilitating subsequent mapping to three-dimensional voxel space. By extracting the lesion coordinates first and then the local features, the location reconstruction task and the local representation task can be processed separately, reducing misjudgments caused by directly fusing multimodal data before spatial alignment is completed.

[0029] The topological anchor point construction module is used to extract anatomically invariant markers from expanded 2D endoscopic images and natural 3D radiographic images, and to establish spatial coordinate registration relationships based on these markers. In this application, anatomically invariant markers refer to structural positions in the stomach that still have relatively stable reference significance under different examination states, not points that will never shift. The center point of the gastric angle and the center point of the pyloric ring are such structural references because they have relatively clear positional characteristics in the gastric anatomy and can be identified through structural morphology in both endoscopic and 3D radiographic images. The system uses these markers to form a spatial topological anchor point network, which is equivalent to first establishing stable positioning references between the 2D field of view and 3D space, and then combining the relative positional relationship between the lesion and these references for deformation restoration. Since 2D endoscopic images themselves lack depth information, the spatial topological anchor point network is not understood as directly completing rigid 3D registration with only a few marker points, but rather as a constraint input for the subsequent gastric wall biomechanical morphology restoration model, which outputs natural 3D spatial coordinates based on the gastric wall deformation law. This avoids the instability problems that easily occur when performing cross-modal localization based solely on image similarity.

[0030] Figure 3 This diagram illustrates the two-dimensional / three-dimensional anatomical marker extraction and spatial topological anchor network construction provided in an embodiment of this application. In a further embodiment, the topological anchor construction module includes a two-dimensional marker extraction unit, a three-dimensional marker extraction unit, and an anchor network generation unit. The two-dimensional marker extraction unit inputs the expanded two-dimensional endoscopic image into a first-cascaded convolutional keypoint detection network and outputs the coordinates of the two-dimensional gastric angle center point and the two-dimensional pyloric ring center point. The first-cascaded convolutional keypoint detection network can be implemented using existing keypoint detection architectures, such as using a convolutional backbone network to extract low-level texture and high-level structural semantics, and then outputting the keypoint locations through a heatmap regression branch. The cascaded structure means that the coarse localization network first estimates the approximate areas of the gastric angle and pyloric ring, and then the fine localization network regresses the center point coordinates within the local area. This setup can reduce the impact of endoscopic perspective changes, local occlusion, and reflections on keypoint localization.

[0031] The 3D biomarker extraction unit inputs natural-state 3D radiographic images into a second-level convolutional keypoint detection network, outputting the coordinates of the 3D gastric angle center point and the 3D pyloric ring center point. The second-level convolutional keypoint detection network can be implemented using a 3D convolutional network, first identifying gastric structural regions from the 3D volumetric data, and then locating the gastric angle and pyloric ring center within these regions. The 3D keypoint output is in voxel coordinates, or spatial coordinates converted by voxel spacing. The anchor point network generation unit establishes the correspondence between the 2D and 3D gastric angle center point coordinates, as well as the correspondence between the 2D and 3D pyloric ring center point coordinates. The anchor point network generation unit also incorporates the topological relationships between lesions and the gastric angle and pyloric ring in the endoscopic images, such as relative distance, line direction, and the gastric region they are located in, forming a spatial topological anchor point network with nodes and edges. The nodes in this network can include the two-dimensional gastric angle center point, the two-dimensional pyloric ring center point, the three-dimensional gastric angle center point, the three-dimensional pyloric ring center point, and the two-dimensional lesion point to be restored. The edges are used to express the structural connection relationship, relative position relationship and cross-modal correspondence relationship between these nodes.

[0032] The 3D morphology inverse reconstruction module inputs a spatial topological anchor network and 2D lesion coordinates into a pre-trained gastric wall biomechanical morphology reconstruction model. This model performs spatial deformation inverse mapping operations, outputting natural-state 3D spatial coordinates corresponding to the natural-state 3D radiographic image. Here, spatial deformation inverse mapping refers to transforming lesion points observed under endoscopic dilation into the 3D radiographic image coordinate system according to the deformation trend of the gastric wall returning from a dilated state to a natural state. Since the gastric wall is a flexible tissue, the movement of lesion points is not a rigid translation or proportional scaling. Therefore, this module introduces biomechanical constraints to make the lesion coordinate reconstruction process more consistent with the elastic contraction characteristics of the gastric wall tissue.

[0033] Figure 4This is a schematic diagram of three-dimensional morphological inverse reconstruction and deformation calculation based on topological anchor points provided in an embodiment of this application. In a further embodiment, the gastric wall biomechanical morphological reconstruction model can be a graph neural network model. The graph neural network model uses a spatial topological anchor point network as the input graph structure. The node features in the graph can include the coordinates of the gastric angle center point, the coordinates of the pyloric ring center point, the coordinates of the lesion point, the modality identifier of the node, and the relative displacement features between nodes. Edge features can represent the anatomical topological relationship between the gastric angle and the pyloric ring, the relative positional relationship between the lesion point and the two anchor points, and the correspondence between the expanded state and the natural state. The model can include graph convolutional layers, message passing layers, nonlinear activation layers, and coordinate regression layers. The graph convolutional layers are used to summarize the positional constraints of adjacent nodes, the message passing layers are used to pass the deformation information between anchor points to the lesion point, and the coordinate regression layers are used to output the three-dimensional spatial coordinates of the natural state. The reason for using a graph neural network is that this application does not process ordinary pixel mapping on a regular grid, but rather the topological constraint mapping between multiple anatomical anchor points and lesion points. The graph structure is more suitable for expressing this irregular spatial relationship.

[0034] The system may also include a morphological reconstruction model training module. During training, the morphological reconstruction model training module acquires a training dataset containing expanded 2D endoscopic images and natural 3D radiographic images of historical cases. It extracts the coordinates of the gastric angle center point and the pyloric ring center point from the training dataset as graph structure node features. Simultaneously, it acquires the true natural 3D spatial coordinates obtained through manual calibration or clinical registration to construct a training sample library. These true natural 3D spatial coordinates can be annotated on the 3D radiographic images by experienced physicians or generated from semi-automatic registration results confirmed by physicians. During training, the graph neural network model outputs predicted deformation coordinates based on the input anchor network and 2D lesion coordinates. The morphological reconstruction model training module calculates the Euclidean distance between the predicted deformation coordinates and the true natural 3D spatial coordinates in the training sample library and uses this distance as the regression loss. Subsequently, the gradient descent algorithm is used to iteratively update the network weight parameters of the graph neural network model until the coordinate regression error of the model on the training and validation sets stabilizes. The network weight parameters are the main parameters that the model needs to learn. The node feature dimensions and edge connection relationships are determined by the spatial topological anchor network. After the model converges, it can be used to predict the natural coordinates of lesions in new patients.

[0035] In a further embodiment, the three-dimensional morphology reverse reconstruction module includes an elastic contraction deformation unit. The elastic contraction deformation unit employs a finite element elastic contraction deformation algorithm in the gastric wall biomechanical morphology reconstruction model. Based on the anchor point positions indicated by the spatial topological anchor point network, it performs tensor contraction deformation operations on the two-dimensional lesion coordinates to generate natural-state three-dimensional spatial coordinates. The finite element elastic contraction deformation algorithm can be implemented using existing soft tissue deformation simulation techniques. Specifically, the system can first extract the gastric wall region from the natural-state three-dimensional radiographic image and discretize the gastric wall region into a gastric wall deformation mesh composed of multiple mesh units; then, the center point of the gastric angle and the center point of the pyloric ring are set as boundary constraint points, so that these anchor points form a reliable displacement reference between the expanded state and the natural state; subsequently, based on the relative position between the two-dimensional lesion coordinates and the two-dimensional anchor points, the initial candidate surface position is determined in the gastric wall deformation mesh, and the candidate position is iteratively adjusted according to the anchor point constraints and the direction of elastic contraction of the gastric wall. In this application, tensor contraction deformation operation represents transforming the local position, orientation, and scale information in the expanded state into the natural voxel space according to the gastric wall contraction trend, rather than simply translating the coordinate values. Through this processing, the system can simultaneously constrain the lesion coordinates in both the gastric wall thickness direction and the gastric lumen surface direction, improving the reliability of the natural three-dimensional spatial coordinates.

[0036] A multimodal alignment module is used to map the two-dimensional local lesion feature matrix to the voxel space region corresponding to the natural three-dimensional spatial coordinates in the natural three-dimensional radiographic image, generating spatially aligned multimodal features. In a further embodiment, the multimodal alignment module includes a three-dimensional interpolation resampling unit. The three-dimensional interpolation resampling unit uses a three-dimensional spatial interpolation resampling algorithm to fill and transform the two-dimensional local lesion feature matrix into the voxel grid structure of the natural three-dimensional radiographic image, generating spatially aligned multimodal features containing three-dimensional topological location information. The three-dimensional spatial interpolation resampling algorithm can be implemented using existing techniques such as trilinear interpolation, nearest neighbor interpolation, or cubic interpolation. In actual processing, the system determines the voxel space region corresponding to the lesion with the natural three-dimensional spatial coordinates as the center, and then projects the two-dimensional local lesion feature matrix into the voxel space region according to the voxel spacing of the three-dimensional radiographic image and the orientation of the gastric wall surface, forming a feature channel attached to the corresponding voxel region. For positions in the 2D matrix that cannot be directly correlated with 3D voxels, the 3D interpolation resampling unit completes the representation based on the spatial distance and topological position of adjacent voxels, ensuring continuous representation of endoscopic local features in the 3D image. This process does not restore the 2D endoscopic image to a complete 3D texture, but rather embeds the 2D lesion features into the voxel space according to their restored 3D spatial positions, allowing them to be read by the model within the same coordinate frame as the radiographic image features. The resulting spatially aligned multimodal features simultaneously contain information from both the endoscopic surface and the radiographic image space, which can be used for staging in subsequent large-scale models.

[0037] Figure 5This is a flowchart illustrating the process of extracting coordinates and local feature matrices in the lesion extraction module provided in this application embodiment. In a further embodiment, the lesion extraction module includes a lesion segmentation unit, a lesion coordinate extraction unit, and a lesion feature extraction unit. The lesion segmentation unit inputs the expanded state two-dimensional endoscopic image into a pre-trained lesion segmentation network and outputs a lesion region mask. The lesion segmentation network can be implemented using existing segmentation networks such as U-Net, DeepLab, or Mask R-CNN, or it can be implemented using a medical image segmentation network with a Transformer encoder as its backbone. Pre-training can be completed based on labeled gastric cancer endoscopic image data, enabling the network to learn features such as lesion boundaries, mucosal texture abnormalities, and color changes. The lesion coordinate extraction unit extracts the geometric center coordinates of the lesion region mask as two-dimensional lesion coordinates. If the lesion region is irregular in shape, the geometric center coordinates can be obtained by averaging the positions of the pixels covered by the mask; if the lesion region has multiple discrete regions, the lesion coordinate extraction unit can select the connected region with the largest area to calculate the geometric center, or it can calculate the geometric center based on the main lesion region confirmed by the physician. The lesion feature extraction unit performs feature extraction operations on the image pixels covered by the lesion region mask, generating a two-dimensional local lesion feature matrix. The feature extraction operator can be a convolutional feature operator, or it can combine color histograms, texture descriptors, and edge gradient operators. Using a masked area for feature extraction can reduce the inclusion of normal mucosa, endoscopic instrument shadows, and gastric fluid reflections in the feature matrix, thereby improving the specificity of local lesion characterization.

[0038] Figure 6 This is a diagram illustrating the supervised training process of the gastric wall biomechanical morphology reconstruction model provided in this application embodiment. During training, each sample in the training sample library can include an expanded two-dimensional endoscopic image, a natural three-dimensional radiographic image, two-dimensional coordinates of the gastric angle center point, two-dimensional coordinates of the pyloric ring center point, three-dimensional coordinates of the gastric angle center point, three-dimensional coordinates of the pyloric ring center point, two-dimensional lesion coordinates, and real natural three-dimensional spatial coordinates. The morphology reconstruction model training module first constructs graph structure node features based on the gastric angle and pyloric ring, then adds the two-dimensional lesion coordinates as nodes to be predicted into the graph structure. The graph neural network outputs predicted deformation coordinates after multiple rounds of message passing. The regression loss is based on the Euclidean distance between the predicted deformation coordinates and the real natural three-dimensional spatial coordinates; the smaller the distance, the better the model can reconstruct the real spatial position. The gradient descent algorithm updates the network weight parameters in reverse based on the regression loss, enabling the model to gradually learn the coordinate change pattern of the gastric wall as it recovers from an expanded state to a natural state. To reduce bias caused by individual differences in single cases, the training dataset can cover historical cases with different gastric anatomical morphologies, different lesion locations, and different imaging acquisition devices.

[0039] In a further embodiment, the system also includes a microcirculation spatiotemporal dynamic alignment module. When the natural three-dimensional radiographic image is a multi-phase enhanced image, this module extracts the enhancement grayscale temporal data of the abdominal aortic region to construct an individual circulatory baseline curve, and simultaneously extracts the enhancement gradient temporal data of the gastric wall lesion region. Changes in the enhancement grayscale of the abdominal aorta can reflect the circulatory rhythm of the contrast agent within the patient's body. Using this curve as an individual circulatory baseline can reduce the impact of differences in cardiac output, vascular status, and contrast agent arrival time among different patients. The enhancement gradient temporal data of the gastric wall lesion region reflects the enhancement changes of the lesion and surrounding gastric wall under different enhancement phases, and is correlated with tumor invasion depth and boundary activity. The microcirculation spatiotemporal dynamic alignment module performs an alignment mapping operation on the time axis dimension of the enhancement gradient temporal data to the individual circulatory baseline curve, generating a standardized hierarchical enhancement tensor. The standardized hierarchical enhancement tensor is then used as a boundary constraint condition, and input into the gastric wall biomechanical morphology reconstruction model along with the spatial topological anchor network and the two-dimensional lesion coordinates. By incorporating perfusion timing constraints, the morphological reconstruction model not only relies on geometric anchor points but can also utilize enhancement changes at the lesion boundary to help determine the true extent of the lesion in the gastric wall.

[0040] Figure 7 This is a schematic diagram of microcirculation spatiotemporal dynamic alignment data extraction and alignment mapping provided in an embodiment of this application. In a further embodiment, the microcirculation spatiotemporal dynamic alignment module includes a dynamic time warping unit. The dynamic time warping unit uses a dynamic time warping algorithm to construct a two-dimensional cumulative distance matrix between the enhancement gradient time series data and the individual circulatory baseline curve. The elements in the two-dimensional cumulative distance matrix represent the cumulative difference between the two time series at different time points. The dynamic time warping unit searches for the continuous alignment point sequence that minimizes the sum of element distances in the two-dimensional cumulative distance matrix and uses this continuous alignment point sequence as the time warp alignment mapping path. Subsequently, the dynamic time warping unit uses the time warp alignment mapping path to perform stretching and compression transformation operations on the time axis dimension of the enhancement gradient time series data to generate a standardized hierarchical enhancement tensor. Here, the stretching and compression do not change the medical meaning of the enhancement data, but rather adjust the enhancement process of different patients or different acquisition phases to a unified individual circulatory reference, so that subsequent models compare the lesion enhancement features under the same circulatory stage.

[0041] The large-scale model staging prediction module is used to input spatially aligned multimodal features into a pre-trained multimodal large-scale model to perform feature aggregation and pathological staging classification calculations, and output gastric cancer staging prediction results. The multimodal large-scale model can employ an architecture consisting of a visual encoder, a 3D image encoder, a text encoder, and a fusion decoder. The visual encoder processes local lesion features from 2D endoscopic images, the 3D image encoder processes voxel spatial features from natural 3D radiographic images, the text encoder processes pathological biopsy text, and the fusion decoder aligns and aggregates information from different modalities. For spatially aligned multimodal features, the model can use tensor splicing, cross-attention, or multilayer perceptron fusion to complete feature aggregation. The pathological staging classification calculation in this application refers to the calculation by which the multimodal large-scale model predicts and classifies the pathological stage corresponding to the gastric cancer lesion based on the spatially aligned multimodal features; that is, it outputs the gastric cancer staging prediction result with the pathological stage as the prediction target. In embodiments where the system also includes a pathological text fusion unit, this pathological staging classification calculation further incorporates the histological information reflected in the pathological biopsy text, making the staging prediction more closely match the pathological state of the lesion. The staging and classification calculation can output TNM-related prediction results or pathological staging categories, with the specific output format selectable based on the clinical system configuration. Since the input features have already been spatially aligned, the large model can simultaneously read endoscopic surface lesion manifestations, three-dimensional spatial locations, and radiographic enhancement information during classification, thereby reducing conflicts caused by independent judgments of multimodal data.

[0042] Figure 8 This is a diagram showing the inverse mapping of three-dimensional puncture point coordinates and the extraction of relative differences in blood perfusion characteristics in the corresponding region, provided in an embodiment of this application. In a further embodiment, the large-scale model staging prediction module includes a pathological text fusion unit. The pathological text fusion unit acquires the pathological biopsy text associated with the expanded two-dimensional endoscopic image and extracts the word vector encoding matrix obtained by encoding the pathological biopsy text. The pathological biopsy text may include histological type, degree of differentiation, cancer cell morphology, and biopsy sampling description, etc. Text encoding can be implemented using existing medical text encoding models, such as a Transformer-based text encoder. The word vector encoding matrix is ​​used to convert the pathological text into a vector representation that the model can compute.

[0043] The pathology text fusion unit also obtains the two-dimensional puncture point coordinates corresponding to the pathological biopsy operation. These two-dimensional puncture point coordinates can be understood as the position coordinates of the endoscopic biopsy sampling point in the expanded two-dimensional endoscopic image, used to characterize the actual sampling location corresponding to the pathology text. The pathology text fusion unit inputs the two-dimensional puncture point coordinates and the spatial topological anchor network into the gastric wall biomechanical morphology restoration model, performs a spatial deformation inverse mapping operation, and outputs three-dimensional puncture point coordinates. This is because the pathological biopsy results originate from local sampling points under endoscopy; if the corresponding location of the sampling point in the natural three-dimensional radiographic image is unknown, the text information cannot accurately match the lesion range in the image. After obtaining the three-dimensional puncture point coordinates, the pathology text fusion unit extracts the first blood flow perfusion feature distribution of the region at the three-dimensional puncture point coordinates in the natural three-dimensional radiographic image, and extracts the second blood flow perfusion feature distribution of the tumor margin extension region. The first blood flow perfusion feature distribution is used to represent the perfusion status of the area where the biopsy sampling point is located, and the second blood flow perfusion feature distribution is used to represent the perfusion status of the lesion margin and potentially invasive extension areas. Subsequently, the pathological text fusion unit performs a matrix subtraction operation on the first and second blood flow perfusion feature distributions to generate a relative difference matrix. The relative difference matrix reflects whether the biopsy point is sufficient to represent the lesion margin extension area. If the difference between the two is large, it indicates that the local biopsy text may be less representative of the overall lesion.

[0044] The pathology text fusion unit inputs the relative difference matrix into a pre-trained multimodal adaptive gating network, outputting attention fusion weights adapted to the pathology biopsy text. These attention fusion weights are normalized to a value between 0 and 1. A value closer to 1 indicates that the pathology biopsy text is more suitable for higher-level adoption in the current case; a value closer to 0 indicates that the pathology biopsy text is less representative of the overall lesion, and the model should reduce its influence during fusion. The pathology text fusion unit multiplies the attention fusion weights into the word vector encoding matrix to form an attention-weighted text feature matrix, which is then combined with spatially aligned multimodal features and subjected to tensor concatenation and aggregation within the large multimodal model. Through this adaptive gating method, the pathology text is not added to the model with fixed weights, but rather its influence is dynamically adjusted based on the perfusion difference between the puncture point and the tumor margin extension area.

[0045] Figure 9This is a structural diagram of the multimodal adaptive gated text fusion and large-model staging prediction operation provided in the embodiments of this application. In a further embodiment, the system also includes a gated network training module. The gated network training module inputs pre-acquired historical perfusion relative difference matrix samples into the network input layer, inputs manually labeled text confidence weights into the network loss layer, and uses the backpropagation algorithm to adjust the weight matrix of the fully connected layer of the multimodal adaptive gated network. The manually labeled text confidence weights can be annotated by clinicians based on biopsy site locations, lesion heterogeneity, perfusion differences, and final pathological results in historical cases. The multimodal adaptive gated network may include a matrix flattening layer, a fully connected layer, a nonlinear activation layer, and a weight output layer. The matrix flattening layer converts the relative difference matrix into one-dimensional features, the fully connected layer learns the mapping relationship between perfusion differences and text confidence, and the weight output layer outputs attention fusion weights. During training, the backpropagation algorithm adjusts the weight matrix of the fully connected layer based on the error between the output weights and the manually labeled text confidence weights, enabling the gated network to gradually learn which perfusion differences represent more reliable biopsy texts and which perfusion differences represent biopsy texts that need to have their weights reduced.

[0046] In another embodiment, when outputting the gastric cancer staging prediction results, the system can also simultaneously output spatial positioning information related to the prediction, including natural-state 3D spatial coordinates, corresponding voxel spatial regions, sources of local lesion features, standardized hierarchical enhancement tensors, and attention fusion weights for pathological text. This output method helps physicians understand the basis of the model's judgment. The system does not replace clinical diagnosis, but rather serves as an auxiliary staging tool, providing physicians with consistent lesion locations and multi-source feature fusion results across modal spaces.

[0047] In a complete operation, the image acquisition module first acquires expanded two-dimensional endoscopic images and natural three-dimensional radiographic images of the same patient. The lesion segmentation unit identifies lesion regions in the endoscopic images and forms lesion region masks. The lesion coordinate extraction unit calculates the geometric center coordinates of the lesion region mask, and the lesion feature extraction unit extracts a two-dimensional local lesion feature matrix from the mask-covered area. The two-dimensional marker extraction unit and the three-dimensional marker extraction unit extract the center points of the gastric angle and pyloric ring, respectively. The anchor point network generation unit establishes a spatial topological anchor point network based on the correspondence between the two-dimensional and three-dimensional models. The three-dimensional morphology inverse reconstruction module inputs the spatial topological anchor point network and the two-dimensional lesion coordinates into the gastric wall biomechanical morphology reconstruction model, and introduces a standardized hierarchical enhancement tensor as a boundary constraint when necessary, outputting natural three-dimensional spatial coordinates. The multimodal alignment module then fills the two-dimensional local lesion feature matrix into the corresponding voxel space region in the natural three-dimensional radiographic image to generate spatially aligned multimodal features. The large model staging prediction module receives spatially aligned multimodal features and combines them with pathological text encoding features adjusted by attention weights for tensor concatenation and aggregation, ultimately outputting the gastric cancer staging prediction results.

[0048] Through the above implementation methods, this application can restore the location of two-dimensional lesions in expanded endoscopic images to the natural three-dimensional radiographic image space, and fuse endoscopic local features, radiographic voxel features, dynamic enhancement features, and pathological biopsy text features within the same three-dimensional coordinate framework. Because the system first resolves spatial correspondences before performing multimodal fusion and staging prediction, it can reduce information offsets caused by different examination states, different image modalities, and different sampling locations, thus improving the stability and interpretability of gastric cancer auxiliary staging results.

[0049] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and inventive concept of this application, should be included within the scope of protection of this application.

Claims

1. A multimodal large-scale model-assisted staging system for gastric cancer, characterized in that, include: The image acquisition module is used to acquire the patient's expanded two-dimensional endoscopic image and natural three-dimensional radiographic image; The lesion extraction module is used to extract two-dimensional lesion coordinates and two-dimensional local lesion feature matrix from the expanded two-dimensional endoscopic image; A topological anchor point construction module is used to extract two-dimensional and three-dimensional anatomically invariant markers from the expanded two-dimensional endoscopic image and the natural three-dimensional radiographic image, respectively. Based on the two-dimensional and three-dimensional anatomically invariant markers, a spatial coordinate registration relationship is established to construct a spatial topological anchor point network. The nodes in the spatial topological anchor point network include the two-dimensional gastric angle center point, the two-dimensional pyloric ring center point, the three-dimensional gastric angle center point, the three-dimensional pyloric ring center point, and the two-dimensional lesion point to be restored. The edges in the spatial topological anchor point network are used to express the structural connection relationship, relative position relationship, and cross-modal correspondence relationship between the nodes. A three-dimensional morphological inverse reconstruction module is used to input the spatial topological anchor point network and the two-dimensional lesion coordinates into a pre-trained gastric wall biomechanical morphological reconstruction model. The gastric wall biomechanical morphological reconstruction model performs a spatial deformation inverse mapping operation on the two-dimensional lesion coordinates and outputs the natural three-dimensional spatial coordinates corresponding to the natural three-dimensional radiographic image. The gastric wall biomechanical morphological reconstruction model is a graph neural network model. The graph neural network model uses the spatial topological anchor point network as the input graph structure. The node features of the graph neural network model include the modality identifier of the node and the relative displacement features between nodes. The graph neural network model includes a graph convolutional layer, a message passing layer, and a coordinate regression layer. The graph convolutional layer is used to summarize the positional constraints of adjacent nodes. The message passing layer is used to transmit the deformation information between anchor points to the lesion point. The coordinate regression layer is used to output the natural three-dimensional spatial coordinates. The multimodal alignment module is used to map the two-dimensional local lesion feature matrix to the voxel space region in the natural state three-dimensional radiographic image corresponding to the natural state three-dimensional spatial coordinates, thereby generating spatially aligned multimodal features; The large model staging prediction module is used to input the spatially aligned multimodal features into a pre-trained multimodal large model to perform feature aggregation and pathological staging classification calculations, and output gastric cancer staging prediction results.

2. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that, The topology anchor point construction module includes a two-dimensional marker extraction unit, a three-dimensional marker extraction unit, and an anchor point network generation unit; The two-dimensional marker extraction unit is used to input the expanded two-dimensional endoscopic image into the first cascaded convolutional key point detection network to perform structural feature extraction operations, and output the coordinates of the center point of the gastric angle and the coordinates of the center point of the pyloric ring as the two-dimensional anatomically invariant markers. The three-dimensional marker extraction unit is used to input the natural three-dimensional radiographic image into the second-level convolutional key point detection network to perform structural feature extraction operations, and output the coordinates of the three-dimensional gastric angle center point and the coordinates of the three-dimensional pyloric ring center point as the three-dimensional anatomically invariant markers. The anchor point network generation unit is used to establish the registration relationship between the coordinates of the two-dimensional gastric angle center point and the coordinates of the three-dimensional gastric angle center point, and to establish the registration relationship between the coordinates of the two-dimensional pyloric ring center point and the coordinates of the three-dimensional pyloric ring center point, thereby generating the spatial topology anchor point network.

3. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that: The system also includes a morphological model training module, which is used to acquire a training dataset containing expanded two-dimensional endoscopic images and natural three-dimensional radiographic images of historical cases, extract the coordinates of the gastric angle center point and the pyloric ring center point in the training dataset as graph structure node features, and obtain the corresponding real natural three-dimensional spatial coordinates to construct a training sample library. The morphological reconstruction model training module is further used to employ a supervised training algorithm, using the Euclidean distance between the predicted deformation coordinates output by the graph neural network model and the real natural three-dimensional spatial coordinates calibrated in the training sample library as the regression loss function, and to iteratively update the neuron weights of the graph neural network model using the gradient descent algorithm until the graph neural network model converges.

4. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that, The three-dimensional morphology reverse restoration module includes an elastic contraction deformation unit. The elastic contraction deformation unit is used to apply the finite element elastic contraction deformation algorithm in the gastric wall biomechanical morphology restoration model to perform tensor contraction deformation operation on the two-dimensional lesion coordinates based on the anchor point positions indicated by the spatial topological anchor point network, and generate natural three-dimensional spatial coordinates corresponding to the natural three-dimensional radiographic image.

5. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that, The multimodal alignment module includes a three-dimensional interpolation resampling unit, which uses a three-dimensional spatial interpolation resampling algorithm to fill and transform the two-dimensional local lesion feature matrix into the voxel grid structure of the natural three-dimensional radiographic image, thereby generating the spatially aligned multimodal features containing three-dimensional topological location information.

6. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that, The lesion extraction module includes a lesion segmentation unit, a lesion coordinate extraction unit, and a lesion feature extraction unit; The lesion segmentation unit is used to input the expanded two-dimensional endoscopic image into a pre-trained lesion segmentation network and output a lesion region mask. The lesion coordinate extraction unit is used to extract the geometric center coordinates of the lesion region mask as the two-dimensional lesion coordinates; The lesion feature extraction unit is used to perform feature extraction operations on the image pixels covered by the lesion region mask using feature extraction operators to generate the two-dimensional local lesion feature matrix.

7. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that, The system also includes a microcirculation spatiotemporal dynamic alignment module, which is used to extract enhanced grayscale temporal data of the abdominal aortic region in the natural three-dimensional radiographic image to form an individual circulatory baseline curve. The microcirculation spatiotemporal dynamic alignment module is also used to extract the enhancement gradient temporal data of the gastric wall lesion area in the natural state three-dimensional radiographic image; The micro-circulation spatiotemporal dynamic alignment module is also used to perform alignment mapping operation on the time axis dimension of the reinforcement gradient time series data to the individual circulatory baseline curve to generate a standardized hierarchical reinforcement tensor. The three-dimensional morphological inverse reconstruction module is also used to input the standardized layered enhancement tensor as a boundary constraint condition, together with the spatial topological anchor network and the two-dimensional lesion coordinates, into the gastric wall biomechanical morphological reconstruction model.

8. The gastric cancer multimodal large model-assisted staging system according to claim 7, characterized in that, The micro-circulation spatiotemporal dynamic alignment module includes a dynamic time warping unit, which is used to construct a two-dimensional cumulative distance matrix between the reinforcement gradient time series data and the individual circulatory baseline curve using a dynamic time warping algorithm. The dynamic time warping unit is also used to find the sequence of consecutive alignment points that minimizes the sum of element distances in the two-dimensional cumulative distance matrix, and to use the sequence of consecutive alignment points as the time warp alignment mapping path. The dynamic time warping unit is also used to perform time axis dimension stretching and compression transformation operations on the reinforcement gradient time series data using the time warp alignment mapping path to generate the normalized hierarchical reinforcement tensor.

9. The gastric cancer multimodal large model-assisted staging system according to claim 1, characterized in that, The large model staging prediction module includes a pathological text fusion unit, which is used to obtain pathological biopsy text associated with the expanded state two-dimensional endoscopic image and extract word vector encoding matrix obtained by encoding the pathological biopsy text. The pathological text fusion unit is also used to obtain the two-dimensional puncture point coordinates corresponding to the pathological biopsy operation, input the two-dimensional puncture point coordinates and the spatial topology anchor point network into the gastric wall biomechanical morphology restoration model to perform spatial deformation inverse mapping operation, and output the three-dimensional puncture point coordinates. The pathological text fusion unit is also used to extract a first blood flow perfusion feature distribution corresponding to the region at the coordinates of the three-dimensional puncture point in the natural three-dimensional radiographic image, and to extract a second blood flow perfusion feature distribution corresponding to the tumor edge extension region. The pathological text fusion unit is also used to perform a matrix difference operation on the first blood perfusion feature distribution and the second blood perfusion feature distribution to generate a relative difference matrix; The pathological text fusion unit is also used to input the relative difference matrix into a pre-trained multimodal adaptive gating network and output attention fusion weights that are adapted to the pathological biopsy text. The pathological text fusion unit is also used to multiply the attention fusion weights into the word vector encoding matrix, and then perform tensor splicing and aggregation operations with the spatially aligned multimodal features within the multimodal large model.

10. The gastric cancer multimodal large model-assisted staging system according to claim 9, characterized in that, The system also includes a gated network training module, which is used to input pre-acquired historical perfusion relative difference matrix samples into the network input layer, input manually labeled text confidence weights into the network loss layer, and use the backpropagation algorithm to adjust the weight matrix of the fully connected layer of the multimodal adaptive gated network.

Images