Preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on ai image recognition

Abstract

The present application relates to the field of intelligent medical treatment and artificial intelligence, and discloses a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition, which comprises: a data acquisition and preprocessing module, which acquires structured clinical data and multi-modal image data of a patient and performs preprocessing; a lesion identification and segmentation module, which performs lesion identification and segmentation processing on the preprocessed image data by using an improved nnU-Net framework; a feature extraction and screening module, which extracts image features from the segmentation result and screens out features related to clinical indicators; a multi-modal feature fusion module, which performs feature fusion on the screened image features and the preprocessed structured clinical data; a multi-task risk evaluation module, which establishes a multi-parameter risk prediction model by using a semi-structured distribution regression method and outputs multi-task prediction results; and an evaluation report generation module, which generates a structured evaluation report. The present application realizes intelligent and precise preoperative evaluation and postoperative risk prediction for gastric antrum cancer patients.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent medical treatment and artificial intelligence technology, and in particular to a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition. BACKGROUND

[0002] Gastric antrum cancer often complicates with pyloric obstruction, which seriously affects the nutritional status and quality of life of patients. Precise preoperative evaluation (such as obstruction degree, tumor infiltration range, lymph node metastasis status, and adjacent relationship with surrounding blood vessels / viscera) is crucial for formulating a reasonable surgical plan (such as resection range, anastomosis method, and whether to need combined organ resection).

[0003] Current clinical practice is challenged: strong subjectivity in image evaluation: there are subjective differences in doctors' interpretation of CT, MRI, endoscopy and other images, and the ability to identify subtle structures (such as submucosal infiltration, micro-metastatic lesions, and vascular invasion) is limited. Difficulties in multi-modal information integration: clinical decision-making needs to integrate multi-source heterogeneous information such as CT (anatomical structure), MRI (soft tissue resolution), endoscopy (intracavity morphology), pathology (molecular characteristics), and lack of efficient fusion analysis tools. Single evaluation dimension: traditional methods are difficult to simultaneously accurately quantify key indicators such as obstruction severity, tumor burden, and surgical risk. Limited intelligence: there is a lack of intelligent systems that can handle both structured and unstructured data. In addition, the existing technology does not solve the special evaluation needs of gastric antrum cancer with pyloric obstruction, and lacks deep integration with clinical indicators.

[0004] Therefore, there is an urgent need for a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition to realize multi-dimensional and full-distribution precise evaluation and risk prediction. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application aims to provide a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition, which integrates multi-modal image data such as CT, MRI, and EUS with clinical structured data to realize intelligent and precise preoperative evaluation and postoperative risk prediction for gastric antrum cancer patients.

[0006] To achieve the above-mentioned purpose, the following technical solutions are implemented:

[0007] A preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition, comprising:

[0008] A data acquisition and preprocessing module for acquiring structured clinical data and multi-modal image data of patients and performing data preprocessing;

[0009] The lesion recognition and segmentation module adopts an improved nnU-Net framework to perform lesion recognition and segmentation processing on the preprocessed multi-modal image data, and outputs pixel-level segmentation results including a tumor main body, a pyloric obstruction region, regional lymph nodes and adjacent key structures.

[0010] The feature extraction and screening module is configured to extract image features from the segmentation results and screen out features related to clinical indicators from the image features through a two-stage feature selection method based on clinical relevance driving, and output a screened image feature vector; wherein the image features include morphological features, deep features and image features.

[0011] The multi-modal feature fusion module is configured to perform feature fusion on the screened image features and the preprocessed structured clinical data, and generate a multi-modal fusion feature vector.

[0012] The multi-task risk assessment module is configured to model the multi-modal fusion feature vector using a semi-structured distribution regression method, establish a multi-parameter risk prediction model, and output a multi-task prediction result.

[0013] The evaluation report generation module is configured to generate a structured evaluation report according to the segmentation results and the multi-task prediction result.

[0014] Further, in the data acquisition and preprocessing module, the structured clinical data includes patient demographic characteristics, laboratory examination indicators, comorbidity information and tumor markers obtained from a clinical information system; and the multi-modal image data includes CT images, MRI images, EUS images and endoscopic images.

[0015] The preprocessing includes: performing spatial registration on multi-modal image data using an elastic registration algorithm based on mutual information, adjusting window width and window level, resampling to isotropic resolution and denoising processing; performing outlier processing and missing value interpolation on structured clinical data.

[0016] Further, the improved nnU-Net framework adopted by the lesion recognition and segmentation module includes:

[0017] Attention gate units and multi-scale feature pyramid structures are introduced into the encoder-decoder structure.

[0018] A composite loss function is constructed based on Dice loss, weighted cross-entropy loss and boundary loss function for optimization.

[0019] Further, the feature extraction and screening module extracts image features from the segmentation results, including:

[0020] Morphological features include tumor volume, maximum diameter, surface area, sphericity, obstructed lumen cross-sectional area, and stenosis length ratio;

[0021] Deep features are high-dimensional feature vectors extracted by nnU-Net encoder after global average pooling;

[0022] Imaging features include first-order statistics, texture features, shape features, and wavelet features extracted from tumor and lymph node ROIs.

[0023] Further, the two-stage feature selection method based on clinical relevance driving includes:

[0024] First stage: feature rough screening based on statistical significance analysis and clinical prior knowledge, retaining features significantly related to clinical endpoints;

[0025] Second stage: feature fine screening based on the combination of stability selection and LASSO regression to obtain the final feature subset.

[0026] Further, the multi-modal feature fusion module adopts a cross-modal feature fusion method based on attention mechanism to fuse the image features and the preprocessed structured clinical data, specifically including:

[0027] Project the image feature vector and the clinical data vector into the same dimension feature space respectively;

[0028] Take the clinical feature vector as Query, the image feature vector as Key and Value, and calculate the cross-attention weight;

[0029] Fuse the original features and the attention-weighted features through the gating mechanism to generate the fused feature vector.

[0030] Further, the multi-task risk assessment module adopts a semi-structured distribution regression method to model the conditional distribution of multiple target clinical indicators for the input multi-modal fusion feature vector, and each distribution parameter is expressed in the form of a structured additive predictor:

[0031]

[0032] wherein, represents the kth parameter of the conditional distribution of the target variable, is the link function, is the predictor mapping expression of the kth distribution parameter, is the jth feature effect function, and v is the multi-modal fusion feature of the patient.

[0033] Further, the predictor is decomposed into three types of feature effect functions:

[0034] Structured linear effects , for modeling linear clinical factors;

[0035] Structured nonlinear effects , for modeling nonlinear effects of physiological or laboratory indicators;

[0036] Unstructured DNN effects , for processing high-dimensional semantic features extracted by image convolution or text attention mechanisms.

[0037] Further, the structured and unstructured feature effects are separated by an orthogonalization mechanism, defining as the projection matrix of the structured feature space, the orthogonal complement projection matrix of or the orthogonal complement space of , I is the identity matrix; the hidden features of the DNN output are projected by the orthogonal complement projection matrix to obtain , ensuring that the network learns residual information that cannot be covered by the structured model; wherein is the original, unprocessed high-dimensional feature representation extracted from the deep neural network DNN; is the orthogonalized deep feature vector.

[0038] Further, the multi-task risk assessment module outputs multi-task prediction results through a multi-parameter risk prediction model, including: prediction of the degree of pyloric obstruction, probability prediction of T staging, regional lymph node metastasis risk prediction, probability prediction of key blood vessel invasion, and postoperative complication risk prediction;

[0039] The structured evaluation report output by the evaluation report generation module includes key segmentation result visualization, quantitative measurement value, predicted grade / probability, surgical difficulty prompt, and risk warning.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] 1. By introducing the attention gate (AG) mechanism and the boundary reinforcement loss function, the system has significantly better recognition and segmentation accuracy (Dice coefficient, Hausdorff distance) of subtle structures such as gastric antrum cancer lesion boundaries and micro-metastatic lymph nodes than traditional models and manual delineation by doctors, reducing subjective differences in evaluation and improving evaluation accuracy.

[0042] 2. By designing a multi-modal feature fusion module and a semi-structured distribution regression model, the present application can deeply mine complementary information between imageomic features, deep features, and structured clinical data, avoid information redundancy, and significantly improve the AUC and other indicators of the prediction model.

[0043] 3. The application adopts stability selection for feature screening, ensuring the robustness and generalization ability of the feature set of the constructed prediction model, so that it can maintain high performance on different source data sets, reducing the risk of overfitting.

[0044] 4. The system of the application not only provides static indicators such as TNM staging and obstruction degree, but also quantitatively predicts the probability of key blood vessel invasion and the risk of postoperative complications (such as pancreatic fistula), providing unprecedented decision support for surgeons to develop "individualized and risk-controllable" surgical plans (such as resection range, anastomosis method, and whether joint organ resection).

[0045] 5. The application constructs a semi-structured distribution regression model in view of the coexistence of structured (numerical, classification) and unstructured (text, image) in real clinical data, realizes joint modeling of graph structure information and multi-modal features, and improves data utilization efficiency; the semi-structured distribution regression model and the orthogonalization mechanism help doctors understand which clinical indicators and image features the prediction results are based on, and enhance the trust of clinical doctors in AI-assisted decision-making.

[0046] 6. The application liberates doctors from tedious image observation, measurement and recording work, automatically generates structured reports, shortens the evaluation time from tens of minutes to minutes, eliminates the experience difference between doctors of different levels and different hospitals, and promotes the standardization and homogeneity of diagnosis and treatment.

[0047] In summary, the application can improve the treatment effect of patients with gastric antral carcinoma with pyloric obstruction by converting multi-modal data into high-value clinical insights through AI technology.

[0048] It should be understood that the content described in the summary section is not intended to limit the key or important features of the embodiments of the application, nor to limit the scope of the application. Other features of the application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and other features, advantages, and aspects of embodiments of the application will become more apparent by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals indicate like elements in the drawings and drawings in which:

[0050] Figure 1 is a module schematic diagram of an AI image recognition-based preoperative precise evaluation system for gastric antral carcinoma with pyloric obstruction according to an embodiment of the application;

[0051] Figure 2is a system framework schematic diagram of a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition according to an embodiment of the present application;

[0052] Figure 3 is an architecture schematic diagram of the lesion identification and segmentation module 120 according to an embodiment of the present application;

[0053] Figure 4 is an architecture schematic diagram of the feature extraction and screening module 130 according to an embodiment of the present application;

[0054] Figure 5 is an architecture schematic diagram of the multi-modal feature fusion module 140 according to an embodiment of the present application

[0055] Figure 6 is an architecture schematic diagram of the multi-task risk assessment module 150 according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0057] In addition, the term "and / or" in this paper is only to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship.

[0058] Figure 1 is a module schematic diagram of a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition according to an embodiment of the present application; Figure 2 is a system framework schematic diagram of a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition according to an embodiment of the present application. As shown in Figure 1 and Figure 2 a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition 100, comprising:

[0059] a data acquisition and preprocessing module 110 for acquiring structured clinical data and multi-modal image data of a patient and performing data preprocessing;

[0060] a data acquisition unit 111: for acquiring structured clinical data and multi-modal image data of a patient.

[0061] The collected data comes from multi-center gastric antral cancer patients with pyloric obstruction (n=1580) multi-modal image data, clinical structured data, time span from January 2019 to December 2024.

[0062] Among them, the multi-modal image data includes: CT image: plain scan and enhancement (arterial phase, venous phase), layer thickness 1mm; MRI image: T1WI, T2WI, DWI sequence, layer thickness 3mm; EUS image: endoscopic ultrasound image, focusing on obtaining the five-layer structure of the gastric wall; Endoscopic image: white light endoscopy, narrow band imaging (NBI) image, etc.

[0063] Among them, the structured clinical data includes patient basic information, laboratory examination indicators and treatment history information obtained from the clinical information system, including: demographic characteristics: age, gender, BMI; Laboratory indicators: hemoglobin, albumin, prealbumin, CRP; Comorbidities: diabetes, cardiovascular disease, etc.; Tumor markers: CEA, CA19-9, CA72-4.

[0064] The data preprocessing unit 112 is used for preprocessing the multi-modal image data including standardization processing and spatial registration, specifically including: using the mutual information-based elastic registration algorithm to realize the spatial alignment of multi-phase and multi-modal images, including adjusting the window width and window level of DICOM images, resampling to isotropic resolution and denoising processing.

[0065] Among them, the preprocessing of structured clinical data includes but is not limited to: data cleaning: outlier processing, missing value interpolation (using KNN algorithm).

[0066] The lesion recognition and segmentation module 120 uses an improved nnU-Net framework to perform lesion recognition and segmentation processing on the preprocessed multi-modal image data, and outputs pixel-level segmentation results including tumor main body, pyloric obstruction area, regional lymph nodes and adjacent key structures.

[0067] Specifically, the nnU-Net segmentation model is constructed: the nnU-Net framework is used to realize automatic architecture optimization, and the nnU-Net model configuration parameters are as shown in Table 1:

[0068] Table 1

[0069]

[0070] To solve the problem of variable lesion morphology and low contrast with surrounding tissues of antral carcinoma, in further embodiments of the present application, the lesion recognition and segmentation module 120 introduces an attention gate and a multi-scale feature pyramid (Feature Pyramid Network, FPN) on the basis of the standard nnU-Net, to accurately segment the antral carcinoma lesions with variable morphology, and output a pixel-level segmentation mask including the tumor main body, the pylorus obstruction area, the regional lymph nodes, and the adjacent key structures such as blood vessels and organs. Figure 3 Fig. 1 is a schematic diagram of the architecture of the lesion recognition and segmentation module 120 of an embodiment of the present application.

[0071] Specifically, the network architecture is configured as follows: the encoder adopts 5 layers of downsampling, and the number of channels of each layer is 32, 64, 128, 256, and 512, respectively; the decoder adopts 4 layers of upsampling, and the number of channels of each layer is 256, 128, 64, and 32, respectively.

[0072] The attention gate unit is embedded at the skip connection of the encoder and the decoder, focusing on the automatic learning of the lesion area and suppressing irrelevant background information. The multi-scale feature pyramid structure is added at the end of the decoder, fusing feature maps of different scales, thereby improving the detection ability of small lesions such as micro-metastatic lymph nodes.

[0073] A composite loss function is adopted :

[0074]

[0075] Wherein: is the Dice loss, which measures the overlap between the predicted area and the true area , represents the intersection size (overlapping part) of and , and represent the area size of and , respectively, when and are completely overlapped, =0; the greater the difference, the greater the loss value, thereby promoting the consistency of the prediction result and the true result at the area level.

[0076] is the weighted cross-entropy loss, which is used to measure the uncertainty of class prediction, and is aimed at the class imbalance problem (such as tumor pixels far less than background pixels), to avoid the model biasing to the majority class;

[0077]

[0078] where S is the total number of pixels in the current batch, and are the true label and model predicted probability of the s-th pixel, respectively, is a weight coefficient pre-computed according to the inverse frequency of class c, aiming to increase the punishment of the model for predicting errors of minority classes (such as tumor pixels), so as to force the model to better learn the segmentation of the target region. The weight of class c is usually calculated according to the inverse frequency of the class c, that is, the less frequent the class is, the greater the weight is, so that when the model incorrectly segments a rare tumor pixel, it will be much more "punished" than an incorrectly segmented common background pixel, thus forcing the model to strive to learn the minority class well. For example Total number of pixels / number of pixels of class c.

[0079] is the boundary loss function, which uses a smooth approximation of the Hausdorff distance based on the boundary point set, such as Shape-aware Loss, which is defined as the distance measure between the predicted boundary and the true boundary:

[0080]

[0081] , represent the boundary point sets of the predicted region and the true region, respectively, N is the total number of predicted boundary points, and M is the total number of true boundary points. represents the squared distance between two points; : For each predicted boundary point, find the nearest true boundary point and calculate the distance; : For each true boundary point, find the nearest predicted boundary point and calculate the distance, optimize the boundary prediction accuracy, and make the model more accurately capture the edge contour of the target. Weight coefficients a = 0.6, b = 0.3, g = 0.1, determined by grid search.

[0082] By constructing a composite loss function with Dice loss, weighted cross-entropy loss and boundary loss function for optimization, the model can perform well in multiple evaluation dimensions, especially suitable for image segmentation tasks that require high boundary accuracy, such as medical image segmentation.

[0083] Training strategy:

[0084]

[0085] ​​​Stage 1 (Pre-training): Pre-train the encoder (e.g. ResNet-50 backbone) on large-scale natural image dataset (e.g. ImageNet) to improve feature extraction capability.

[0086] Stage 2 (Main training): Fine-tune the whole network with annotated medical image data, using a lower learning rate (1e-4), focusing on training the decoder and attention gate unit.

[0087] Stage 3 (Fine-tuning): Freeze the low-level parameters of the encoder, only fine-tune the high-level network and segmentation head, using more aggressive data augmentation (e.g. stronger elastic deformation to simulate tissue softness).

[0088] Further, in some embodiments of the present application, in view of the anatomical characteristics of pyloric obstruction, geometric constraints are introduced for the pyloric obstruction region in the model inference post-processing stage: (1) Based on the segmented gastric cavity and tumor region, the diameter and length of the narrowest part of the obstruction are automatically calculated. (2) If the model output obstruction region is discontinuous anatomically (e.g. located outside the gastric antrum), trigger the graph-based region growing post-processing algorithm to correct it according to pixel intensity and spatial continuity, ensuring that the segmentation result conforms to clinical common sense.

[0089] Optionally, the above network is implemented through mainstream deep learning frameworks (such as PyTorch or TensorFlow). Data preparation requires cutting the registered multi-modal images (CT, MRI) into fixed-size patches (such as 256x256x128) and providing pixel-level annotations. During training, the AdamW optimizer is used with an initial learning rate of 3e-4, and the cosine annealing strategy is used to adjust the learning rate. Model performance is monitored on the validation set using Dice coefficient, Hausdorff distance, etc. Early stopping strategy is used to prevent overfitting.

[0090] The feature extraction and screening module 130 is configured to extract image features from the segmentation result and screen out features related to clinical indicators from the image features through a two-stage feature selection method based on clinical relevance driving, and output the screened image feature vector; wherein the image features include morphological features, deep features and image features.

[0091] The morphological features include tumor volume, maximum diameter, surface area, sphericity, obstruction lumen cross-sectional area and stenosis length ratio.

[0092] The deep features are high-dimensional features extracted by the nnU-Net encoder. Specifically, the deep features (512-dimensional vector) after global average pooling are extracted from the second-to-last layer of the nnU-Net encoder, capturing the abstract semantic information learned by the CNN.

[0093] Radiomics features: first-order statistics, texture features, shape features and wavelet features extracted from the tumor and lymph node ROIs. Specifically, the present embodiment extracts a large number (>1000) of standardized radiomics features from each ROI (tumor, lymph node, etc.) obtained from the segmentation module using open-source libraries such as PyRadiomics, including: (1) first-order statistical features: energy, entropy, skewness, kurtosis, etc. (2) morphological features: volume, surface area, sphericity, etc. (3) texture features: gray level co-occurrence matrix (GLCM), gray level run-length matrix (GLRLM), and gray level size zone matrix (GLSZM) related features. (4) high-order features: applying wavelet filtering and Laplacian of Gaussian (LoG) filtering to extract the above features at multiple scales.

[0094] In the feature screening stage, to solve the overfitting and redundancy problems caused by high-dimensional features, preferably, the present embodiment performs a two-stage feature screening (E) driven by clinical relevance, Figure 4 is a schematic diagram of the feature extraction and screening module 130 of the present embodiment. The following process is used:

[0095] First stage (E1): coarse screening based on statistical significance and clinical importance

[0096] Calculate the statistical correlation (using Spearman rank correlation coefficient or Mann-Whitney U test) between each radiomics feature and the key clinical endpoints (such as obstruction grade, T stage, lymph node metastasis status). Keep the features that are significantly correlated (p-value < 0.05) with at least one clinical endpoint. At the same time, provide a "mandatory feature list" (such as tumor maximum diameter, obstruction minimum cross-sectional area) by domain experts (such as radiologists, surgeons) based on prior knowledge, which are unconditionally entered into the next stage.

[0097] Second stage (E2): fine screening based on regularization and ensemble learning

[0098] Concatenate the features retained in the first stage with deep features and structured clinical data (after standardization) to form a high-dimensional feature vector. Use the combination of stability selection (Stability Selection) and LASSO regression for feature screening. Stability selection: fit the LASSO model through multiple subsampling, and calculate the frequency of each feature being selected. The mathematical expression is:

[0099]

[0100] wherein, : the frequency or stability probability of the t-th feature (e.g. "tumor volume" or "CRP value") is selected, which represents the proportion of the t-th feature selected by the base learner LASSO in all resampling (or subsampling) processes; B is the total number of resampling, is an indicator function; is the feature subset selected by LASSO regression on the b-th resampled subset.

[0101] Then, the features with a frequency exceeding a stability threshold (e.g. 0.8) are selected and retained to form the final feature subset:

[0102]

[0103] In a preferred embodiment of the present application, B = 1000, In each experiment b, a subset is randomly extracted from the original data. LASSO regression is run on this subset to obtain a feature subset , i.e. the features considered important by this experiment. The task of the indicator function is to determine whether the t-th feature is selected in b experiments, and if it is selected, it is recorded as 1; if it is not selected, it is recorded as 0. Finally, the average value of the number of times (the sequence composed of 1 and 0) that the t-th feature is selected in the B experiments is calculated: the total number of times that the t-th feature is selected / the total number of experiments B. Finally, multiple resampling can significantly reduce the risk of instability in feature selection caused by small perturbations in LASSO regression data, thereby obtaining a feature subset with high reliability and strong generalization ability. This method is more robust than single LASSO, and the final selected feature subset will be sent to the multi-modal feature fusion module 140.

[0104] Specifically, the scikit-learn, statsmodels and PyRadiomics libraries in Python can be used to implement this module. First, load the mask and original image output by the lesion recognition and segmentation module 120, extract the radiomics features and deep features. Then, use pandas for data arrangement and statistical analysis. Finally, use LassoCV (LASSO with cross-validation) in scikit-learn and a custom stability selection process to complete feature selection. The entire process can be encapsulated as an automated script, with the input being the segmentation result and clinical data, and the output being the filtered image feature list or vector.

[0105] The multi-modal feature fusion module 140 is configured to fuse the filtered image features with the preprocessed structured clinical data to generate a multi-modal fusion feature vector.

[0106] The multimodal feature fusion module 140 receives the filtered image feature vector from the feature extraction and filtering module 130 and the preprocessed structured clinical data vector from the module 112, and associates and aligns them through the unique identifier of the patient (such as Patient ID). At the same time, all features will be standardized or normalized (such as Z-score) before fusion to ensure that features of different dimensions are comparable.

[0107] Figure 5 is the architecture diagram of the multimodal feature fusion module 140 of the embodiment of the present application. As shown in Figure 5 , the multimodal feature fusion module 140 adopts an attention mechanism-based cross-modal feature fusion method to fuse the filtered image feature vector and the preprocessed structured clinical data vector, generating an information-rich, unified multimodal fusion feature vector. Specifically, the following steps are included:

[0108] (1) Input preprocessing:

[0109] Suppose the filtered image feature vector is , and the structured clinical data vector is . , are the dimensions of the image feature vector and the structured clinical data vector , respectively.

[0110] Suppose the patient Patient001, after feature extraction and filtering, obtains the following standardized features:

[0111] The filtered image feature vector vector , , where the 1st dimension (0.85): the standardized tumor volume, meaning 0.85 standard deviations larger than the average tumor volume. The 2nd dimension (-0.23): the standardized tumor sphericity. The 3rd dimension (1.45): the standardized value of a certain deep feature extracted from the NNUnet encoder. The 4th dimension (0.12): the value of a certain texture feature (such as "entropy"). The 5th dimension (0.67): the standardized minimum cross-sectional area at the obstruction.

[0112] The preprocessed clinical data vector , where, the 1st dimension (0.5): normalized age, (65 - mean 60) / standard deviation 10 = 0.5. The 2nd dimension (-1.2): normalized BMI, (18.5 - mean 22) / standard deviation 3 approximately -1.2. The 3rd dimension (0.0): normalized CRP, (5.2 - mean 5.2) / standard deviation 2 = 0.0. The 4th dimension (1.1): normalized CEA (carcinoembryonic antigen), (8.1 - mean 5.0) / standard deviation 2.8 approximately 1.1.

[0113] First, both are projected to the same feature space dimension by two independent linear transformation layers (or fully connected layers) respectively :

[0114]

[0115]

[0116] where, , are learnable weight matrices; is the bias term; denotes the unified dimension of the projected feature space, which is a hyper-parameter and can be flexibly set according to the task complexity and model capacity requirement.

[0117] (2) Cross-modal attention fusion:

[0118] In order to capture the interaction information between image features and clinical features, a simplified cross-attention mechanism is adopted. In the attention mechanism, each input needs to be converted into three different vectors: Query (query), Key (key) and Value (value). In the embodiment of the present application, a group of features is taken as Query, and another group is taken as Key and Value. For example, taking clinical features as Query and image features as Key / Value:

[0119] Calculate attention weight:

[0120]

[0121] where, , , is a learnable projection matrix; is a scaling factor, that is, the dimension of the attention head; is the dimension of the Value (value) vector. , h is the number of attention heads, so that each head can process a low-dimensional subspace, and finally the outputs of all heads are spliced together, and the total dimension is changed back to ; is the matrix transpose symbol.

[0122] Generate attention-based clinical feature representations: .

[0123] This step allows each clinical indicator (such as CRP level) to "question" the most relevant parts of the imaging features (such as the features of the inflammatory area) and updates the representation of the clinical features accordingly.

[0124] Feature merging and output:

[0125] The original projected features are fused with the attention-modulated features. An embodiment of the invention provides an effective method using a gating mechanism:

[0126]

[0127]

[0128] in, The concatenation operation is represented by σ, where σ is the sigmoid function and ⊙ represents element-wise multiplication. G is the gating vector used to learn and control the information flow.

[0129] Finally, The output of a linear layer is the final multimodal fusion feature vector. It is used for risk assessment in the multi-task risk assessment module 150.

[0130] In this embodiment of the invention, the multimodal fusion feature vector v is a high-dimensional numerical vector that integrates multi-source information from images and clinical forms. It can be imagined as a digital fingerprint that encodes all the key information of the patient together.

[0131] In one specific embodiment, as an exemplary illustration, the parameters are... Set to 6, passing through dimension 5 and dimension 4 Through independent linear layers , Projected onto 3D space, to obtain and ;by For Query, Using the key / value pair, cross-attention calculation is performed to obtain an attention-weighted representation of clinical features. Integration through gating mechanisms and ,get Ultimately, it will After passing through a linear layer, the output is the final multimodal fused feature vector v (the dimension is still 1). ): v = [0.22, -0.85, 1.03, -0.41, 0.67, 0.19].

[0132] The obtained multimodal fusion feature vector v is a brand new 6-dimensional vector, where each dimension is already the result of the deep fusion of image and clinical information. For example, v[0]=0.22: represents a comprehensive signal after the interaction of “age” and “tumor volume”, slightly higher than the average value, suggesting a tumor growth pattern related to age, which is relatively slow. v[1]= -0.85: represents the interaction of tumor texture features (such as entropy, ) and low , significantly lower than the average and negative, suggesting that the patient's low body mass index (BMI) is associated with the relatively uniform texture inside the tumor, which may mean that the patient's nutritional status and the microenvironment of the tumor have a specific relationship. Finally, this vector v will eventually be sent to the multi-task risk assessment module 150 of the distribution regression model, and the task of the model is to learn the complex mapping relationship between these features and the final clinical prediction indicators (such as the grade of pyloric obstruction, T stage probability, regional lymph node metastasis risk, key blood vessel invasion probability, and postoperative complication risk, etc.).

[0133] The multimodal feature fusion module 140 can be constructed as an independent neural network sub-module, and the input is two aligned numerical vectors, and the output is a unified feature vector. The weight matrix , , , and other parameters are trained together with the entire multi-task model.

[0134] The multi-task risk assessment module 150 is used to model the multimodal fusion feature vector by using a semi-structured distribution regression method, to establish a multi-parameter risk prediction model, and to output a multi-task prediction result.

[0135] Figure 6 is the architecture schematic diagram of the multi-task risk assessment module 150 of the embodiment of the present application. As Figure 6 shown, the multi-task risk assessment module 150 takes the multimodal fusion feature vector as input, and its core is to model the conditional distribution of multiple target clinical indicators by using a distribution regression method. In this model, each distribution parameter is expressed in the form of a structured additive predictor as follows:

[0136]

[0137] wherein, The kth parameter (such as mean, variance, skewness, quantile, etc.) representing the conditional distribution of the target variable, used to characterize the distribution characteristics of the multi-task prediction results, and optionally, the multi-task prediction results include: prediction results of pyloric obstruction grade, T stage probability, regional lymph node metastasis risk, key blood vessel invasion probability, and postoperative complication risk, etc. is a linking function used to map the predictor to a legal distribution parameter value; is the predictor mapping expression of the kth distribution parameter; is a feature effect function representing the influence of a specific input variable v on the kth parameter, which can be automatically learned by a traditional model (such as a linear model) or a deep neural network; v is the final multi-modal fusion feature vector of the patient; : a structured additive model obtained by combining multiple feature effect functions, used to enhance the interpretability and generalization ability of the model, which represents the sum of the output results of multiple feature effect functions.

[0138] Further, to adapt to the characteristics of multi-source heterogeneous data, the model decomposes the predictor into three types of feature effect functions:

[0139] structured linear effect , used to model linear clinical factors;

[0140] structured nonlinear effect , used to model nonlinear physiological or laboratory indicators;

[0141] unstructured DNN effect , used to process high-dimensional semantic features extracted through image convolution or text attention mechanism (whose output is the original deep feature representation ), and optionally, specifically used to process high-dimensional image semantic features.

[0142] To further ensure that the features extracted by the DNN effect do not have redundant representations with structured clinical features, the present application introduces an orthogonalization unit to project and decompose the features.

[0143] Specifically, the orthogonalization mechanism is used to separate structured and unstructured feature effects, and is defined as the projection matrix of the structured feature space, which aims to project any vector onto the linear space spanned by the structured clinical features. The size of where d is the number of structured clinical features, the matrix is computed from the pre-processed structured clinical data matrix X (e.g., its rows are patients, columns are clinical features) by mathematical methods (e.g., using Moore-Penrose pseudo-inverse ) and essentially defines a clinical-feature-based space.

[0144] the projection matrix that projects onto the orthogonal complement space of the structured clinical data matrix X (i.e., the space X) is the identity matrix.

[0145] the hidden features output by the DNN the orthogonal complement projection matrix the projection decomposition results in , ensuring that the network learns residual information that cannot be covered by the structured model.

[0146] are the original, unprocessed high-dimensional feature representations extracted from the deep neural network (DNN); are the orthogonalized deep feature vectors, whose information is completely linearly independent (orthogonal) of all structured clinical features, containing only the “new knowledge” or “residual information” contributed by unstructured data (images, texts) that cannot be explained by any existing structured clinical indicators, e.g., it includes a special tumor infiltration pattern or microenvironment feature identified by the DNN from CT images that has not been defined by existing clinical indicators.

[0147] Although contains rich information, it is likely to have a lot of redundancy or overlap with existing structured clinical features (such as age, CRP value, etc.), for example, the DNN may have learned “inflammatory signs” from the images, which is the same thing as “CRP elevated” in clinical data. By using the orthogonal complement projection matrix, any component in the vector that is parallel (redundant) to the structured clinical feature space can be stripped or filtered out, and only the component that is perpendicular (orthogonal, non-redundant) to the structured clinical feature space is retained, thus defining a new space that cannot be explained by clinical features. Using the orthogonalization mechanism can avoid redundancy, improve efficiency, and prevent the model from wasting capacity to repeatedly learn information that has been clearly expressed by structured data, allowing the image DNN and clinical data model to perform their respective functions. orthogonal to the clinical features, any prediction performance improvement brought by is purely new insights mined by the image AI, not confused with clinical data. By stripping the dependency between data sources, the model becomes more robust and may perform more consistently on different populations or data from different centers. ​

[0148] Finally, the model supports multi-modal joint modeling, realizing the fusion of table data and image / text data in the same framework.

[0149] Constructing loss function: model total loss The combination of negative log-likelihood loss and structured nonlinear smoothing term:

[0150]

[0151] : Model total loss function, composed of negative log-likelihood loss and structured nonlinear smoothing term. : Given distribution parameter θ, the log-likelihood probability of target variable Y, measuring the fitting degree of the model to the true label. : Multiple parameter vectors of the conditional distribution of the target variable, used to describe the characteristics of the conditional distribution of the target variable. : Regularization hyperparameter of the jth structured nonlinear effect in the kth distribution parameter, used to control the smoothness of the effect function and prevent overfitting noise. : Weight coefficient vector of the kth distribution parameter Nonlinear effect function. : Penalty matrix, usually a second-order difference matrix or a smoothing matrix, used to impose smoothing constraints on Ensure the smoothness of the nonlinear effect function.

[0152] The above target function of the combination of negative log-likelihood loss and structured nonlinear smoothing term can balance the classification accuracy and prediction stability, improve the credibility of the prediction probability and the continuity of the risk score, and is more suitable for the task of disease assessment in dynamic evolution.

[0153] Individual risk estimation: through the above steps, the model can predict the probability distribution of multiple tasks in real time , providing individual risk estimation for clinicians , to obtain the expected operation and assist in decision-making, improving the timeliness and accuracy of preoperative risk assessment.

[0154] In a specific embodiment of the present application, for example for patient Patient001, the image features: automatic arterial phase CT, venous phase CT, MRI multi-sequence images are extracted. Clinical features: age (65), gender (male), BMI (24.7), CRP (5.2 mg / L), CEA (8.1 ng / mL), etc. Fusion features: the above multi-modal data are processed by the system to generate a fusion feature vector v. Finally, the multi-task risk prediction results are predicted based on the semi-structured distribution regression model, and the system specifically outputs the complete probability distribution of each risk, rather than a single, deterministic prediction value, thereby more comprehensively evaluating the uncertainty and risk range. For example:

[0155] Task 1: Severity classification of pyloric obstruction

[0156] Predicted probability distribution: mild obstruction: 15%, moderate obstruction: 60%, severe obstruction: 25%.

[0157] Individualized risk estimate : moderate obstruction.

[0158] Clinical interpretation: the model suggests that the patient is most likely in moderate obstruction (probability 60%), but there is a 25% chance of severe obstruction. This also suggests to the clinician that even if the gastroscopy may not complete the most in-depth examination due to patient tolerance, the possibility of high obstruction still needs to be highly vigilant, and preoperative nutritional support is crucial.

[0159] Task 2: Probability prediction of pathological T stage

[0160] Predicted probability distribution: T2 stage: 10%, T3 stage: 45%, T4a stage: 40%, T4b stage: 5%.

[0161] Individualized risk estimate : T3 stage.

[0162] Clinical interpretation: the patient has the highest probability of being in T3 stage (tumor penetrating subserosal layer), but it must be noted that the probability of T4a stage (invasion of serosa) is as high as 40%. This means that the tumor is likely to have broken through the lamina propria of the gastric wall, and joint organ resection should be prepared during surgery, rather than simple local resection.

[0163] Task 3: Regional lymph node metastasis risk

[0164] Predicted probability distribution: the model outputs a lymph node metastasis probability of 72%.

[0165] Confidence interval (90%): [68%, 76%].

[0166] Individualized risk estimate (E[Y|θ]): high risk.

[0167] Clinical Interpretation: Extremely high risk of lymph node metastasis with very certain estimation (narrow confidence interval). Supports D2 lymphadenectomy rather than D1 surgery with a smaller range to ensure radicality.

[0168] Task 4: Probability of celiac trunk invasion

[0169] Predicted probability distribution: The model outputs a probability of 18% for vessel invasion.

[0170] Confidence interval (90%): [8%, 32%].

[0171] Individualized risk estimate (E[Y|θ]): Low risk.

[0172] Clinical Interpretation: Although the average probability is low, the confidence interval is wide and the upper limit reaches 32%, indicating that there is considerable uncertainty that cannot be ignored. The subsequent report will provide recommendations: "The tumor is closely related to the celiac trunk artery, although the probability of invasion is low, extra caution should be taken during intraoperative separation, and a blood vessel repair plan should be prepared."

[0173] Task 5: Risk of postoperative pancreatic fistula

[0174] Predicted probability distribution: The model outputs a probability of 30% for postoperative B / C grade pancreatic fistula.

[0175] Confidence interval (90%): [25%, 36%].

[0176] Individualized risk estimate (E[Y|θ]): High risk.

[0177] Clinical Interpretation: The system's predicted risk of pancreatic fistula is much higher than the average level of conventional statistics (usually <10%). This will guide intraoperative decision-making: (1) an abdominal drainage tube must be routinely placed; (2) postoperative drainage fluid amylase levels must be closely monitored; (3) patients and family members must be informed of this specific risk in advance.

[0178] In the embodiments of the present disclosure, the training process of the system is divided into two main stages:

[0179] First stage: independently train the lesion recognition and segmentation module 120, using a large amount of image data with pixel-level annotation, and the optimization target is the composite loss function .

[0180] Second stage: fix the parameters of the trained lesion recognition and segmentation module 120 as a feature extractor. Then, the feature extraction and screening module 130, the multi-modal feature fusion module 140, and the multi-task risk assessment module 150 are jointly trained in an end-to-end manner, and the optimization target is the distribution regression loss function At this stage, the encoder parameters of the optional frozen (not fine-tuned) lesion identification and segmentation module 120 can be frozen to prevent overfitting and accelerate training; if the data volume is very sufficient, it can also be fine-tuned at a very low learning rate.

[0181] The evaluation report generation module 160 is configured to generate a structured evaluation report according to the segmentation result and the multi-task prediction result.

[0182] Further, the structured evaluation report output by the evaluation report generation module 160 includes key segmentation result visualization, quantitative measurement value, prediction classification / probability, surgical difficulty prompt and risk warning, and supports PDF and DICOM SR formats. For example, the surgical difficulty prompt in the report can include: "The tumor is closely adhered to the superior pancreaticoduodenal artery, and an ultrasonic knife is recommended for separation to reduce the risk of bleeding"; the risk warning can include: "Based on the model prediction, the risk of postoperative pancreatic fistula for this patient is 30% (high risk), and it is recommended to place a drainage tube during surgery and closely monitor the amylase level after surgery".

[0183] The implementation of the entire system relies on the Python scientific computing ecosystem (such as NumPy, PyTorch / TensorFlow, scikit-learn, PyRadiomics) and GPU computing resources.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described modules can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0185] In the technical solution of the present application, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0186] It should be noted that in the embodiments of the present application, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0187] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An AI image recognition-based preoperative precise evaluation system for gastric antral cancer with pyloric obstruction, characterized in that, The method comprises the following steps: a data acquisition and preprocessing module for acquiring structured clinical data and multi-modal image data of a patient and performing data preprocessing; a lesion identification and segmentation module for performing lesion identification and segmentation processing on the preprocessed multi-modal image data by using an improved nnU-Net framework, and outputting pixel-level segmentation results including a tumor main body, a pyloric obstruction region, regional lymph nodes and adjacent key structures; wherein the improved nnU-Net framework comprises: introducing an attention gate unit and a multi-scale feature pyramid structure in the encoder-decoder structure; constructing a composite loss function based on the Dice loss, the weighted cross-entropy loss and the boundary loss function for optimization; the encoder adopts 5 layers of down-sampling, and the number of channels of each layer is 32, 64, 128, 256 and 512 respectively; the decoder adopts 4 layers of up-sampling, and the number of channels of each layer is 256, 128, 64 and 32 respectively; the attention gate unit is embedded at the skip connection of the encoder and the decoder, focusing on automatic learning of the lesion area and suppressing irrelevant background information; a multi-scale feature pyramid structure is added at the end of the decoder to fuse feature maps of different scales; Adopting a composite loss function : wherein, is the Dice loss, which measures the overlap between the predicted and the ground truth regions; is the weighted cross-entropy loss, which measures the uncertainty of the class prediction; is the boundary loss function, which adopts a smooth approximation of the Hausdorff distance based on the set of boundary points, defined as the distance measure between the predicted and the ground truth boundaries; and a, b, g are the weight coefficients for the corresponding loss terms; a feature extraction and screening module for extracting image features from the segmentation results and screening out features related to clinical indicators from the image features by using a two-stage feature selection method based on clinical correlation driving, and outputting a screened image feature vector; wherein the image features include morphological features, deep features and image features; a multi-modal feature fusion module for fusing the screened image features with the preprocessed structured clinical data to generate a multi-modal fusion feature vector; a multi-task risk assessment module for modeling the multi-modal fusion feature vector by using a semi-structured distribution regression method, establishing a multi-parameter risk prediction model, and outputting a multi-task prediction result; wherein the multi-task risk assessment module models the conditional distribution of multiple target clinical indicators for the input multi-modal fusion feature vector by using a semi-structured distribution regression method, and each distribution parameter is expressed in the form of a structured additive predictor: wherein, denotes the kth parameter of the target variable conditional distribution, is a link function, is a predictor mapping expression for the kth distribution parameter, is the jth feature effect function, v is the multi-modal fusion feature of the patient; an evaluation report generation module for generating a structured evaluation report according to the segmentation results and the multi-task prediction results.

2. The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition according to claim 1, characterized in that, In the data acquisition and preprocessing module, the structured clinical data includes patient demographic characteristics, laboratory examination indicators, comorbidity information and tumor markers obtained from a clinical information system; and the multi-modal image data includes CT images, MRI images, EUS images and endoscopic images. The preprocessing includes: performing spatial registration, window width and window level adjustment, resampling to isotropic resolution and denoising processing on the multi-modal image data by using an elastic registration algorithm based on mutual information; performing outlier processing and missing value imputation on the structured clinical data. 3.The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition according to claim 1, characterized in that, The feature extraction and screening module extracts image features from the segmentation results, including: the morphological features include tumor volume, maximum diameter, surface area, sphericity, obstruction lumen cross-sectional area and stenosis length ratio; the deep features are high-dimensional feature vectors extracted by the nnU-Net encoder after global average pooling; The imaging features include first-order statistics, texture features, shape features and wavelet features extracted from the tumor and lymph node ROIs. 4.The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition according to claim 3, characterized in that, The two-stage feature selection method based on clinical relevance driving includes: The first stage: coarse screening of features based on statistical significance analysis and clinical prior knowledge, retaining features significantly related to clinical endpoints; The second stage: fine screening of features based on a combination of stability selection and LASSO regression to obtain the final feature subset. 5.The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition according to claim 1, characterized in that, The multi-modal feature fusion module uses a cross-modal feature fusion method based on attention mechanism to fuse the image features and the pre-processed structured clinical data, specifically including: Projecting the image feature vector and the clinical data vector into the same dimension feature space respectively; Taking the clinical feature vector as Query, the image feature vector as Key and Value, and calculating the cross-attention weight; Fusing the original features and the attention-weighted features through a gating mechanism to generate a fused feature vector. 6.The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition according to claim 1, wherein The predictor is decomposed into three types of feature effect functions: Structured linear effects for modeling linear clinical factors; Structured nonlinear effects physiological or laboratory indicators for modeling nonlinear effects; Unstructured DNN effects , for processing high-dimensional semantic features extracted by image convolution or text attention mechanism. 7.The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition according to claim 6, characterized in that, orthogonalization mechanism to separate the structured and unstructured feature effects, define the projection matrix of the structured feature space, the orthogonal complement projection matrix or orthogonal complement space , I is the identity matrix; the hidden features of the DNN output orthogonal complement projection matrix projection decomposition, get , to ensure that the network learns the residual information that the structured model cannot cover; wherein, is the original, unprocessed high-dimensional feature representation extracted from the deep neural network DNN; is the orthogonalized deep feature vector. 8.The preoperative precise evaluation system for gastric antral cancer with pyloric obstruction based on AI image recognition of claim 1, characterized in that, The multi-task risk assessment module outputs multi-task prediction results through a multi-parameter risk prediction model, including: prediction of the degree of pyloric obstruction, probability prediction of T staging, regional lymph node metastasis risk prediction, probability prediction of key blood vessel invasion, and postoperative complication risk prediction; The structured evaluation report output by the evaluation report generation module includes key segmentation result visualization, quantitative measurement value, predicted classification / probability, surgical difficulty prompt and risk warning.

Images