AI image recognition-based preoperative precise evaluation system for antral sinus cancer with pyloric obstruction

Abstract

The invention relates to the field of intelligent medical treatment and artificial intelligence, and discloses an AI image recognition-based antral sinus cancer accompanied pyloric obstruction preoperative precise evaluation system, which comprises a data acquisition and preprocessing module for acquiring and preprocessing structured clinical data and multi-modal image data of a patient; the focus identification and segmentation module is used for carrying out focus identification and segmentation processing on the preprocessed image data by adopting an improved nnU-Net framework; the feature extraction and screening module is used for extracting image features from the segmentation result and screening features related to clinical indexes; the multi-modal feature fusion module is used for carrying out feature fusion on the screened image features and the preprocessed structured clinical data; the multi-task risk assessment module is used for establishing a multi-parameter risk prediction model by adopting a semi-structured distribution regression method and outputting a multi-task prediction result; and the evaluation report generation module is used for generating a structured evaluation report. And intelligent and precise preoperative evaluation and postoperative risk prediction of the antral carcinoma patient are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent medical treatment and artificial intelligence technology, and in particular to a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition. BACKGROUND

[0002] Gastric antrum cancer often complicates with pyloric obstruction, which seriously affects the nutritional status and quality of life of patients. Precise preoperative evaluation (such as obstruction degree, tumor infiltration range, lymph node metastasis status, and adjacent relationship with surrounding blood vessels / viscera) is crucial for formulating a reasonable surgical plan (such as resection range, anastomosis method, and whether to need combined organ resection).

[0003] Current clinical practice is challenged: strong subjectivity in image evaluation: there are subjective differences in doctors' interpretation of CT, MRI, endoscopy and other images, and the ability to identify subtle structures (such as submucosal infiltration, micro-metastatic lesions, and vascular invasion) is limited. Difficulties in multi-modal information integration: clinical decision-making needs to integrate multi-source heterogeneous information such as CT (anatomical structure), MRI (soft tissue resolution), endoscopy (intracavity morphology), pathology (molecular characteristics), and lack of efficient fusion analysis tools. Single evaluation dimension: traditional methods are difficult to simultaneously accurately quantify key indicators such as obstruction severity, tumor burden, and surgical risk. Limited intelligence: there is a lack of intelligent systems that can handle both structured and unstructured data. In addition, the existing technology does not solve the special evaluation needs of gastric antrum cancer with pyloric obstruction, and lacks deep integration with clinical indicators.

[0004] Therefore, there is an urgent need for a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition to realize multi-dimensional and full-distribution precise evaluation and risk prediction. SUMMARY

[0005] In view of the deficiencies in the prior art, the present application aims to provide a preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition, which integrates multi-modal image data such as CT, MRI, and EUS with clinical structured data to realize intelligent and precise preoperative evaluation and postoperative risk prediction for gastric antrum cancer patients.

[0006] To achieve the above-mentioned purpose, the following technical solutions are implemented:

[0007] A preoperative precise evaluation system for gastric antrum cancer with pyloric obstruction based on AI image recognition, comprising:

[0008] A data acquisition and preprocessing module for acquiring structured clinical data and multi-modal image data of patients and performing data preprocessing;

[0009] The lesion recognition and segmentation module adopts an improved nnU-Net framework to perform lesion recognition and segmentation processing on the preprocessed multi-modal image data, and outputs pixel-level segmentation results including a tumor main body, a pyloric obstruction region, regional lymph nodes and adjacent key structures.

[0010] The feature extraction and screening module is configured to extract image features from the segmentation results and screen out features related to clinical indicators from the image features through a two-stage feature selection method based on clinical relevance driving, and output a screened image feature vector; wherein the image features include morphological features, deep features and image features.

[0011] The multi-modal feature fusion module is configured to perform feature fusion on the screened image features and the preprocessed structured clinical data, and generate a multi-modal fusion feature vector.

[0012] The multi-task risk assessment module is configured to model the multi-modal fusion feature vector using a semi-structured distribution regression method, establish a multi-parameter risk prediction model, and output a multi-task prediction result.

[0013] The evaluation report generation module is configured to generate a structured evaluation report according to the segmentation results and the multi-task prediction result.

[0014] Further, in the data acquisition and preprocessing module, the structured clinical data includes patient demographic characteristics, laboratory examination indicators, comorbidity information and tumor markers obtained from a clinical information system; and the multi-modal image data includes CT images, MRI images, EUS images and endoscopic images.

[0015] The preprocessing includes: performing spatial registration on multi-modal image data using an elastic registration algorithm based on mutual information, adjusting window width and window level, resampling to isotropic resolution and denoising processing; performing outlier processing and missing value interpolation on structured clinical data.

[0016] Further, the improved nnU-Net framework adopted by the lesion recognition and segmentation module includes:

[0017] Attention gate units and multi-scale feature pyramid structures are introduced into the encoder-decoder structure.

[0018] A composite loss function is constructed based on Dice loss, weighted cross-entropy loss and boundary loss function for optimization.

[0019] Further, the feature extraction and screening module extracts image features from the segmentation results, including:

[0020] Morphological features include tumor volume, maximum diameter, surface area, sphericity, obstructed lumen cross-sectional area, and stenosis length ratio;

[0021] Deep features are high-dimensional feature vectors extracted by nnU-Net encoder after global average pooling;

[0022] Imaging features include first-order statistics, texture features, shape features, and wavelet features extracted from tumor and lymph node ROIs.

[0023] Further, the two-stage feature selection method based on clinical relevance driving includes:

[0024] First stage: feature rough screening based on statistical significance analysis and clinical prior knowledge, retaining features significantly related to clinical endpoints;

[0025] Second stage: feature fine screening based on the combination of stability selection and LASSO regression to obtain the final feature subset.

[0026] Further, the multi-modal feature fusion module adopts a cross-modal feature fusion method based on attention mechanism to fuse the image features and the preprocessed structured clinical data, specifically including:

[0027] Project the image feature vector and the clinical data vector into the same dimension feature space respectively;

[0028] Take the clinical feature vector as Query, the image feature vector as Key and Value, and calculate the cross-attention weight;

[0029] Fuse the original features and the attention-weighted features through the gating mechanism to generate the fused feature vector.

[0030] Further, the multi-task risk assessment module adopts a semi-structured distribution regression method to model the conditional distribution of multiple target clinical indicators for the input multi-modal fusion feature vector, and each distribution parameter is expressed in the form of a structured additive predictor:

[0031]

[0032] wherein, represents the kth parameter of the conditional distribution of the target variable, is the link function, is the predictor mapping expression of the kth distribution parameter, is the jth feature effect function, and v is the multi-modal fusion feature of the patient.

[0033] Further, the predictor is decomposed into three types of feature effect functions:

[0034] Structured linear effects , for modeling linear clinical factors;

[0035] Structured nonlinear effects , for modeling nonlinear effects of physiological or laboratory indicators;

[0036] Unstructured DNN effects , for processing high-dimensional semantic features extracted by image convolution or text attention mechanisms.

[0037] Further, the structured and unstructured feature effects are separated by an orthogonalization mechanism, defining as the projection matrix of the structured feature space, the orthogonal complement projection matrix of or the orthogonal complement space of , I is the identity matrix; the hidden features of the DNN output are projected by the orthogonal complement projection matrix to obtain , ensuring that the network learns residual information that cannot be covered by the structured model; wherein is the original, unprocessed high-dimensional feature representation extracted from the deep neural network DNN; is the orthogonalized deep feature vector.

[0038] Further, the multi-task risk assessment module outputs multi-task prediction results through a multi-parameter risk prediction model, including: prediction of the degree of pyloric obstruction, probability prediction of T staging, regional lymph node metastasis risk prediction, probability prediction of key blood vessel invasion, and postoperative complication risk prediction;

[0039] The structured evaluation report output by the evaluation report generation module includes key segmentation result visualization, quantitative measurement value, predicted grade / probability, surgical difficulty prompt, and risk warning.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] 1. By introducing the attention gate (AG) mechanism and the boundary reinforcement loss function, the system has significantly better recognition and segmentation accuracy (Dice coefficient, Hausdorff distance) of subtle structures such as gastric antrum cancer lesion boundaries and micro-metastatic lymph nodes than traditional models and manual delineation by doctors, reducing subjective differences in evaluation and improving evaluation accuracy.

[0042] 2. By designing a multi-modal feature fusion module and a semi-structured distribution regression model, the present application can deeply mine complementary information between imageomic features, deep features, and structured clinical data, avoid information redundancy, and significantly improve the AUC and other indicators of the prediction model.

[0043] 3. The application adopts stability selection for feature screening, ensuring the robustness and generalization ability of the feature set of the constructed prediction model, so that it can maintain high performance on different source data sets, reducing the risk of overfitting.

[0044] 4. The system of the application not only provides static indicators such as TNM staging and obstruction degree, but also quantitatively predicts the probability of key blood vessel invasion and the risk of postoperative complications (such as pancreatic fistula), providing unprecedented decision support for surgeons to develop "individualized and risk-controllable" surgical plans (such as resection range, anastomosis method, and whether joint organ resection).

[0045] 5. The application constructs a semi-structured distribution regression model in view of the coexistence of structured (numerical, classification) and unstructured (text, image) in real clinical data, realizes joint modeling of graph structure information and multi-modal features, and improves data utilization efficiency; the semi-structured distribution regression model and the orthogonalization mechanism help doctors understand which clinical indicators and image features the prediction results are based on, and enhance the trust of clinical doctors in AI-assisted decision-making.

[0046] 6. The application liberates doctors from tedious image observation, measurement and recording work, automatically generates structured reports, shortens the evaluation time from tens of minutes to minutes, eliminates the experience difference between doctors of different levels and different hospitals, and promotes the standardization and homogeneity of diagnosis and treatment.

[0047] In summary, the application can improve the treatment effect of patients with gastric antral carcinoma with pyloric obstruction by converting multi-modal data into high-value clinical insights through AI technology.

[0048] It should be understood that the content described in the summary section is not intended to limit the key or important features of the embodiments of the application, nor to limit the scope of the application. Other features of the application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and other features, advantages, and aspects of embodiments of the application will become more apparent by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numerals indicate like elements in the drawings and drawings in which:

[0050] Figure 1 is a module schematic diagram of an AI image recognition-based preoperative precise evaluation system for gastric antral carcinoma with pyloric obstruction according to an embodiment of the application;

[0051] Figure 2This is a schematic diagram of the system framework of a system for accurate preoperative assessment of gastric antral cancer with pyloric obstruction based on AI image recognition according to an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of the architecture of the lesion identification and segmentation module 120 according to an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of the architecture of the feature extraction and screening module 130 according to an embodiment of the present invention;

[0054] Figure 5 Schematic diagram of the architecture of the multimodal feature fusion module 140 according to an embodiment of the present invention

[0055] Figure 6 FIG. 1 is a schematic diagram of the architecture of the multi-task risk assessment module 150 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0058] Figure 1 This is a module diagram of a system for accurate preoperative assessment of gastric antral cancer with pyloric obstruction based on AI image recognition according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the system framework of a system for accurate preoperative evaluation of gastric antral cancer with pyloric obstruction based on AI image recognition according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a system 100 for accurate preoperative assessment of gastric antral cancer with pyloric obstruction based on AI image recognition comprises:

[0059] The data acquisition and preprocessing module 110 is used to acquire the patient's structured clinical data and multimodal imaging data and perform data preprocessing;

[0060] Data acquisition unit 111: used to acquire the patient's structured clinical data and multimodal imaging data.

[0061] The collected data comes from multi-center gastric antral cancer patients with pyloric obstruction (n=1580) multi-modal image data, clinical structured data, time span from January 2019 to December 2024.

[0062] Among them, the multi-modal image data includes: CT image: plain scan and enhancement (arterial phase, venous phase), layer thickness 1mm; MRI image: T1WI, T2WI, DWI sequence, layer thickness 3mm; EUS image: endoscopic ultrasound image, focusing on obtaining the five-layer structure of the gastric wall; Endoscopic image: white light endoscopy, narrow band imaging (NBI) image, etc.

[0063] Among them, the structured clinical data includes patient basic information, laboratory examination indicators and treatment history information obtained from the clinical information system, including: demographic characteristics: age, gender, BMI; Laboratory indicators: hemoglobin, albumin, prealbumin, CRP; Comorbidities: diabetes, cardiovascular disease, etc.; Tumor markers: CEA, CA19-9, CA72-4.

[0064] The data preprocessing unit 112 is used for preprocessing the multi-modal image data including standardization processing and spatial registration, specifically including: using the mutual information-based elastic registration algorithm to realize the spatial alignment of multi-phase and multi-modal images, including adjusting the window width and window level of DICOM images, resampling to isotropic resolution and denoising processing.

[0065] Among them, the preprocessing of structured clinical data includes but is not limited to: data cleaning: outlier processing, missing value interpolation (using KNN algorithm).

[0066] The lesion recognition and segmentation module 120 uses an improved nnU-Net framework to perform lesion recognition and segmentation processing on the preprocessed multi-modal image data, and outputs pixel-level segmentation results including tumor main body, pyloric obstruction area, regional lymph nodes and adjacent key structures.

[0067] Specifically, the nnU-Net segmentation model is constructed: the nnU-Net framework is used to realize automatic architecture optimization, and the nnU-Net model configuration parameters are as shown in Table 1:

[0068] Table 1

[0069]

[0070] To solve the problem of variable lesion morphology and low contrast with surrounding tissues of antral carcinoma, in further embodiments of the present application, the lesion recognition and segmentation module 120 introduces an attention gate and a multi-scale feature pyramid (Feature Pyramid Network, FPN) on the basis of the standard nnU-Net, to accurately segment the antral carcinoma lesions with variable morphology, and output a pixel-level segmentation mask including the tumor main body, the pylorus obstruction area, the regional lymph nodes, and the adjacent key structures such as blood vessels and organs. Figure 3 Fig. 1 is a schematic diagram of the architecture of the lesion recognition and segmentation module 120 of an embodiment of the present application.

[0071] Specifically, the network architecture is configured as follows: the encoder adopts 5 layers of downsampling, and the number of channels of each layer is 32, 64, 128, 256, and 512, respectively; the decoder adopts 4 layers of upsampling, and the number of channels of each layer is 256, 128, 64, and 32, respectively.

[0072] The attention gate unit is embedded at the skip connection of the encoder and the decoder, focusing on the automatic learning of the lesion area and suppressing irrelevant background information. The multi-scale feature pyramid structure is added at the end of the decoder, fusing feature maps of different scales, thereby improving the detection ability of small lesions such as micro-metastatic lymph nodes.

[0073] A composite loss function is adopted :

[0074]

[0075] Wherein: is the Dice loss, which measures the overlap between the predicted area and the true area , represents the intersection size (overlapping part) of and , and represent the area size of and , respectively, when and are completely overlapped, =0; the greater the difference, the greater the loss value, thereby promoting the consistency of the prediction result and the true result at the area level.

[0076] is the weighted cross-entropy loss, which is used to measure the uncertainty of class prediction, and is aimed at the class imbalance problem (such as tumor pixels far less than background pixels), to avoid the model biasing to the majority class;

[0077]

[0078] where S is the total number of pixels in the current batch, and are the true label and model predicted probability of the s-th pixel, respectively, is a weight coefficient pre-computed according to the inverse frequency of class c, aiming to increase the punishment of the model for predicting errors of minority classes (such as tumor pixels), so as to force the model to better learn the segmentation of the target region. The weight of class c is usually calculated according to the inverse frequency of the class c, that is, the less frequent the class is, the greater the weight is, so that when the model incorrectly segments a rare tumor pixel, it will be much more "punished" than an incorrectly segmented common background pixel, thus forcing the model to strive to learn the minority class well. For example Total number of pixels / number of pixels of class c.

[0079] is the boundary loss function, which uses a smooth approximation of the Hausdorff distance based on the boundary point set, such as Shape-aware Loss, which is defined as the distance measure between the predicted boundary and the true boundary:

[0080]

[0081] , represent the boundary point sets of the predicted region and the true region, respectively, N is the total number of predicted boundary points, and M is the total number of true boundary points. represents the squared distance between two points; : For each predicted boundary point, find the nearest true boundary point and calculate the distance; : For each true boundary point, find the nearest predicted boundary point and calculate the distance, optimize the boundary prediction accuracy, and make the model more accurately capture the edge contour of the target. Weight coefficients a = 0.6, b = 0.3, g = 0.1, determined by grid search.

[0082] By constructing a composite loss function with Dice loss, weighted cross-entropy loss and boundary loss function for optimization, the model can perform well in multiple evaluation dimensions, especially suitable for image segmentation tasks that require high boundary accuracy, such as medical image segmentation.

[0083] Training strategy:

[0084]

[0085] ​​​Stage 1 (Pre-training): Pre-train the encoder (e.g. ResNet-50 backbone) on large-scale natural image dataset (e.g. ImageNet) to improve feature extraction capability.

[0086] Stage 2 (Main training): Fine-tune the whole network with annotated medical image data, using a lower learning rate (1e-4), focusing on training the decoder and attention gate unit.

[0087] Stage 3 (Fine-tuning): Freeze the low-level parameters of the encoder, only fine-tune the high-level network and segmentation head, using more aggressive data augmentation (e.g. stronger elastic deformation to simulate tissue softness).

[0088] Further, in some embodiments of the present application, in view of the anatomical characteristics of pyloric obstruction, geometric constraints are introduced for the pyloric obstruction region in the model inference post-processing stage: (1) Based on the segmented gastric cavity and tumor region, the diameter and length of the narrowest part of the obstruction are automatically calculated. (2) If the model output obstruction region is discontinuous anatomically (e.g. located outside the gastric antrum), trigger the graph-based region growing post-processing algorithm to correct it according to pixel intensity and spatial continuity, ensuring that the segmentation result conforms to clinical common sense.

[0089] Optionally, the above network is implemented through mainstream deep learning frameworks (such as PyTorch or TensorFlow). Data preparation requires cutting the registered multi-modal images (CT, MRI) into fixed-size patches (such as 256x256x128) and providing pixel-level annotations. During training, the AdamW optimizer is used with an initial learning rate of 3e-4, and the cosine annealing strategy is used to adjust the learning rate. Model performance is monitored on the validation set using Dice coefficient, Hausdorff distance, etc. Early stopping strategy is used to prevent overfitting.

[0090] The feature extraction and screening module 130 is configured to extract image features from the segmentation result and screen out features related to clinical indicators from the image features through a two-stage feature selection method based on clinical relevance driving, and output the screened image feature vector; wherein the image features include morphological features, deep features and image features.

[0091] The morphological features include tumor volume, maximum diameter, surface area, sphericity, obstruction lumen cross-sectional area and stenosis length ratio.

[0092] The deep features are high-dimensional features extracted by the nnU-Net encoder. Specifically, the deep features (512-dimensional vector) after global average pooling are extracted from the second-to-last layer of the nnU-Net encoder, capturing the abstract semantic information learned by the CNN.

[0093] Radiomics features: first-order statistics, texture features, shape features and wavelet features extracted from the tumor and lymph node ROIs. Specifically, the present embodiment extracts a large number (>1000) of standardized radiomics features from each ROI (tumor, lymph node, etc.) obtained from the segmentation module using open-source libraries such as PyRadiomics, including: (1) first-order statistical features: energy, entropy, skewness, kurtosis, etc. (2) morphological features: volume, surface area, sphericity, etc. (3) texture features: gray level co-occurrence matrix (GLCM), gray level run-length matrix (GLRLM), and gray level size zone matrix (GLSZM) related features. (4) high-order features: applying wavelet filtering and Laplacian of Gaussian (LoG) filtering to extract the above features at multiple scales.

[0094] In the feature screening stage, to solve the overfitting and redundancy problems caused by high-dimensional features, preferably, the present embodiment performs a two-stage feature screening (E) driven by clinical relevance, Figure 4 is a schematic diagram of the feature extraction and screening module 130 of the present embodiment. The following process is used:

[0095] First stage (E1): coarse screening based on statistical significance and clinical importance

[0096] Calculate the statistical correlation (using Spearman rank correlation coefficient or Mann-Whitney U test) between each radiomics feature and the key clinical endpoints (such as obstruction grade, T stage, lymph node metastasis status). Keep the features that are significantly correlated (p-value < 0.05) with at least one clinical endpoint. At the same time, provide a "mandatory feature list" (such as tumor maximum diameter, obstruction minimum cross-sectional area) by domain experts (such as radiologists, surgeons) based on prior knowledge, which are unconditionally entered into the next stage.

[0097] Second stage (E2): fine screening based on regularization and ensemble learning

[0098] Concatenate the features retained in the first stage with deep features and structured clinical data (after standardization) to form a high-dimensional feature vector. Use the combination of stability selection (Stability Selection) and LASSO regression for feature screening. Stability selection: fit the LASSO model through multiple subsampling, and calculate the frequency of each feature being selected. The mathematical expression is:

[0099]

[0100] wherein, : The selection frequency or stability probability for the t-th feature (such as "tumor volume" or "CRP value"), which represents the proportion of the t-th feature selected by the base learner LASSO in all resampling (or subsampling) processes; B is the total number of resampling times, is the indicator function; is the feature subset selected by LASSO regression on the b-th resampling subset.

[0101] Then, select and keep the (For example, 0.8) features constitute the final feature subset:

[0102]

[0103] In a preferred embodiment of the present invention, let B=1000, In each experiment b, a subset is randomly selected from the original data. LASSO regression is run on this subset to obtain a feature subset , which is the feature that this experiment considers important. Indicator function The task is to determine whether feature t is selected in b experiments. If it is selected, it is recorded as 1; if it is not selected, it is recorded as 0. The final calculation is to find the average of the number of times feature t is selected in these B experiments (a sequence of 1 and 0): The total number of times feature t was selected is divided by the total number of experiments, B. Finally, multiple resampling through stability selection significantly reduces the risk of unstable feature selection results due to minor data perturbations in LASSO regression, thereby obtaining a feature subset with high reliability and strong generalization capabilities. This method is more robust than a single LASSO. The final selected feature subset is then fed into the multimodal feature fusion module 140.

[0104] Specifically, this module can be implemented using the scikit-learn, statsmodels, and PyRadiomics libraries in Python. First, the mask and original image output from the lesion identification and segmentation module 120 are loaded to extract radiomics and deep features. Subsequently, data organization and statistical analysis are performed using pandas. Finally, feature screening is performed using LassoCV (LASSO with cross-validation) in scikit-learn and a custom stability selection process. The entire process can be encapsulated as an automated script, with segmentation results and clinical data as input and a list or vector of filtered imaging features as output.

[0105] A multimodal feature fusion module 140 is configured to fuse the screened image features with the pre-processed structured clinical data to generate a multimodal fusion feature vector;

[0106] The multimodal feature fusion module 140 receives the filtered image feature vector from the feature extraction and filtering module 130 and the preprocessed structured clinical data vector from the module 112, and associates and aligns them through the unique identifier of the patient (such as Patient ID). At the same time, all features are standardized or normalized (such as Z-score) before fusion to ensure that features of different dimensions are comparable.

[0107] Figure 5 is the architecture diagram of the multimodal feature fusion module 140 of the embodiment of the present application. As shown in Figure 5 , the multimodal feature fusion module 140 adopts an attention mechanism-based cross-modal feature fusion method to fuse the filtered image feature vector and the preprocessed structured clinical data vector, generating an information-rich, unified multimodal fusion feature vector. Specifically, the following steps are included:

[0108] (1) Input preprocessing:

[0109] Suppose the filtered image feature vector is , and the structured clinical data vector is . , are the dimensions of the image feature vector and the structured clinical data vector , respectively.

[0110] Suppose the patient Patient001, after feature extraction and filtering, obtains the following standardized features:

[0111] The filtered image feature vector vector , , where the 1st dimension (0.85): the standardized tumor volume, meaning 0.85 standard deviations larger than the average tumor volume. The 2nd dimension (-0.23): the standardized tumor sphericity. The 3rd dimension (1.45): the standardized value of a certain deep feature extracted from the NNUnet encoder. The 4th dimension (0.12): the value of a certain texture feature (such as "entropy"). The 5th dimension (0.67): the standardized minimum cross-sectional area at the obstruction.

[0112] The preprocessed clinical data vector , where, the 1st dimension (0.5): normalized age, (65 - mean 60) / standard deviation 10 = 0.5. The 2nd dimension (-1.2): normalized BMI, (18.5 - mean 22) / standard deviation 3 approximately -1.2. The 3rd dimension (0.0): normalized CRP, (5.2 - mean 5.2) / standard deviation 2 = 0.0. The 4th dimension (1.1): normalized CEA (carcinoembryonic antigen), (8.1 - mean 5.0) / standard deviation 2.8 approximately 1.1.

[0113] First, both are projected to the same feature space dimension by two independent linear transformation layers (or fully connected layers) respectively :

[0114]

[0115]

[0116] where, , are learnable weight matrices; is the bias term; denotes the unified dimension of the projected feature space, which is a hyper-parameter and can be flexibly set according to the task complexity and model capacity requirement.

[0117] (2) Cross-modal attention fusion:

[0118] In order to capture the interaction information between image features and clinical features, a simplified cross-attention mechanism is adopted. In the attention mechanism, each input needs to be converted into three different vectors: Query (query), Key (key) and Value (value). In the embodiment of the present application, a group of features is taken as Query, and another group is taken as Key and Value. For example, taking clinical features as Query and image features as Key / Value:

[0119] Calculate attention weight:

[0120]

[0121] where, , , is a learnable projection matrix; is a scaling factor, that is, the dimension of the attention head; is the dimension of the Value (value) vector. , h is the number of attention heads, so that each head can process a low-dimensional subspace, and finally the outputs of all heads are spliced together, and the total dimension is changed back to ; is the matrix transpose symbol.

[0122] Generating Attention-Based Clinical Feature Representations: .

[0123] This step allows each clinical indicator (such as CRP level) to "interrogate" the most relevant part of the image features (such as the characteristics of the inflammatory area) and update the representation of the clinical features accordingly.

[0124] Feature merging and output:

[0125] The original projected features are fused with the features modulated by attention. An effective way provided by the embodiment of the present invention is to use a gating mechanism:

[0126]

[0127]

[0128] in, represents the concatenation operation, σ is the sigmoid function, ⊙ is the element-wise multiplication, and G is the gating vector used to learn and control the information flow.

[0129] Finally, The output through a linear layer is the final multimodal fusion feature vector , used by the multi-task risk assessment module 150 to perform risk assessment.

[0130] In an embodiment of the present invention, the multimodal fusion feature vector v is a high-dimensional numerical vector that fuses multi-source information from images and clinical tables. It can be imagined as a digital fingerprint that encodes all the key information of the patient.

[0131] In a specific embodiment, as an exemplary illustration, the parameter Set to 6, through the dimension of 5 and dimension 4 Pass through independent linear layers , , projected onto dimensional space, we get and ;by For Query, As Key / Value, cross attention calculation is performed to obtain the attention-weighted clinical feature representation Fusion through gating mechanism and ,get . Will eventually After a linear layer, the output is the final multimodal fusion feature vector v (the dimension is still ): v = [0.22, -0.85, 1.03, -0.41, 0.67, 0.19].

[0132] The resulting multimodal fusion feature vector v is a brand new 6-dimensional vector, where each dimension is already the result of the deep fusion of image and clinical information. For example, v[0]=0.22: represents a comprehensive signal of the interaction between “age” and “tumor volume”, slightly higher than the average, suggesting a tumor growth pattern related to age, relatively slow progress. v[1]= -0.85: represents the interaction between tumor texture features (such as entropy, ) and low , significantly lower than the average and negative, suggesting that patients with lower body mass index (BMI) are associated with relatively uniform texture inside the tumor, which may mean that the patient's nutritional status and the microenvironment of the tumor are related. Finally, this vector v will eventually be sent to the multi-task risk assessment module 150 of the distribution regression model, and the task of the model is to learn the complex mapping relationship between these features and the final clinical prediction indicators (such as the grade of pyloric obstruction, T stage probability, regional lymph node metastasis risk, key blood vessel invasion probability, and postoperative complication risk, etc.).

[0133] This multimodal feature fusion module 140 can be constructed as an independent neural network submodule, with the input being two aligned numerical vectors and the output being a unified feature vector. The weight matrix , , , and other parameters are trained together with the entire multi-task model.

[0134] The multi-task risk assessment module 150 is used to model the multimodal fusion feature vector using a semi-structured distribution regression method, establish a multi-parameter risk prediction model, and output a multi-task prediction result.

[0135] Figure 6 is the architecture diagram of the multi-task risk assessment module 150 of the embodiment of the present application. As Figure 6 shown, the multi-task risk assessment module 150 takes the multimodal fusion feature vector as input, and its core is to model the conditional distribution of multiple target clinical indicators using a distribution regression method. In this model, each distribution parameter is expressed in the form of the following structured additive predictor:

[0136]

[0137] where, The kth parameter (such as mean, variance, skewness, quantile, etc.) representing the conditional distribution of the target variable, used to characterize the distribution characteristics of the multi-task prediction results, and optionally, the multi-task prediction results include: prediction results of pyloric obstruction grade, T stage probability, regional lymph node metastasis risk, key blood vessel invasion probability, and postoperative complication risk, etc. is a linking function used to map the predictor to a legal distribution parameter value; is the predictor mapping expression of the kth distribution parameter; is a feature effect function representing the influence of a specific input variable v on the kth parameter, which can be automatically learned by a traditional model (such as a linear model) or a deep neural network; v is the final multi-modal fusion feature vector of the patient; : a structured additive model obtained by combining multiple feature effect functions, used to enhance the interpretability and generalization ability of the model, which represents the sum of the output results of multiple feature effect functions.

[0138] Further, to adapt to the characteristics of multi-source heterogeneous data, the model decomposes the predictor into three types of feature effect functions:

[0139] structured linear effect , used to model linear clinical factors;

[0140] structured nonlinear effect , used to model nonlinear physiological or laboratory indicators;

[0141] unstructured DNN effect , used to process high-dimensional semantic features extracted through image convolution or text attention mechanism (whose output is the original deep feature representation ), and optionally, specifically used to process high-dimensional image semantic features.

[0142] To further ensure that the features extracted by the DNN effect do not have redundant representations with structured clinical features, the present application introduces an orthogonalization unit to project and decompose the features.

[0143] Specifically, the orthogonalization mechanism is used to separate structured and unstructured feature effects, and is defined as the projection matrix of the structured feature space, which aims to project any vector onto the linear space spanned by the structured clinical features. The size of where d is the number of structured clinical features, the matrix is computed from the pre-processed structured clinical data matrix X (e.g., its rows are patients, columns are clinical features) by mathematical methods (e.g., using Moore-Penrose pseudo-inverse ) and essentially defines a clinical-feature-based space.

[0144] the projection matrix that projects onto the orthogonal complement space of the structured clinical data matrix X (i.e., the space X) is the identity matrix.

[0145] the hidden features output by the DNN the orthogonal complement projection matrix the projection decomposition results in , ensuring that the network learns the residual information that the structured model cannot cover.

[0146] are the original, unprocessed high-dimensional feature representations extracted from the deep neural network (DNN); are the orthogonalized deep feature vectors, whose information is completely linearly independent (orthogonal) of all structured clinical features, containing only the “new knowledge” or “residual information” contributed by unstructured data (images, texts) that cannot be explained by any existing structured clinical indicators, e.g., it includes a special tumor infiltration pattern or microenvironment feature identified by the DNN from CT images that has not been defined by existing clinical indicators.

[0147] Although contains rich information, this information is likely to have a lot of redundancy or overlap with existing structured clinical features (such as age, CRP value, etc.), for example, the DNN may have learned “inflammatory signs” from the images, which is the same thing as “CRP elevated” in clinical data. By using the orthogonal complement projection matrix, any component in the vector that is parallel (redundant) to the structured clinical feature space can be stripped or filtered out, and only the component that is perpendicular (orthogonal, non-redundant) to the structured clinical feature space is retained, thus defining a new space that cannot be explained by clinical features. Using the orthogonalization mechanism can avoid redundancy, improve efficiency, and prevent the model from wasting capacity to repeatedly learn information that has been clearly expressed by structured data, allowing the image DNN and clinical data model to perform their respective functions. is orthogonal to the clinical features, any prediction performance improvement brought by is purely new insights mined by the image AI, not confused with clinical data. By stripping the dependency between data sources, the model becomes more robust and may perform more consistently on different populations or data from different centers. ​

[0148] Finally, the model supports multi-modal joint modeling, realizing the fusion of table data and image / text data in the same framework.

[0149] Constructing loss function: model total loss The combination of negative log-likelihood loss and structured nonlinear smoothing term:

[0150]

[0151] : Model total loss function, composed of negative log-likelihood loss and structured nonlinear smoothing term. : Given distribution parameter θ, the log-likelihood probability of target variable Y, measuring the fitting degree of the model to the true label. : Multiple parameter vectors of the conditional distribution of the target variable, used to describe the characteristics of the conditional distribution of the target variable. : Regularization hyperparameter of the jth structured nonlinear effect in the kth distribution parameter, used to control the smoothness of the effect function and prevent overfitting noise. : Weight coefficient vector of the kth distribution parameter Nonlinear effect function. : Penalty matrix, usually a second-order difference matrix or a smoothing matrix, used to impose smoothing constraints on Ensure the smoothness of the nonlinear effect function.

[0152] The above target function of the combination of negative log-likelihood loss and structured nonlinear smoothing term can balance the classification accuracy and prediction stability, improve the credibility of the prediction probability and the continuity of the risk score, and is more suitable for the task of disease assessment in dynamic evolution.

[0153] Individual risk estimation: through the above steps, the model can predict the probability distribution of multiple tasks in real time , providing individual risk estimation for clinicians , to obtain the expected operation and assist in decision-making, improving the timeliness and accuracy of preoperative risk assessment.

[0154] In a specific embodiment of the present application, for example for patient Patient001, the image features: automatic arterial phase CT, venous phase CT, MRI multi-sequence images are extracted. Clinical features: age (65), gender (male), BMI (24.7), CRP (5.2 mg / L), CEA (8.1 ng / mL), etc. Fusion features: the above multi-modal data are processed by the system to generate a fusion feature vector v. Finally, the multi-task risk prediction results are predicted based on the semi-structured distribution regression model, and the system specifically outputs the complete probability distribution of each risk, rather than a single, deterministic prediction value, thereby more comprehensively evaluating the uncertainty and risk range. For example:

[0155] Task 1: Severity classification of pyloric obstruction

[0156] Predicted probability distribution: mild obstruction: 15%, moderate obstruction: 60%, severe obstruction: 25%.

[0157] Individualized risk estimate : moderate obstruction.

[0158] Clinical interpretation: the model suggests that the patient is most likely in moderate obstruction (probability 60%), but there is a 25% chance of severe obstruction. This also suggests to the clinician that even if the gastroscopy may not complete the most in-depth examination due to patient tolerance, the possibility of high obstruction still needs to be highly vigilant, and preoperative nutritional support is crucial.

[0159] Task 2: Probability prediction of pathological T stage

[0160] Predicted probability distribution: T2 stage: 10%, T3 stage: 45%, T4a stage: 40%, T4b stage: 5%.

[0161] Individualized risk estimate : T3 stage.

[0162] Clinical interpretation: the patient has the highest probability of being in T3 stage (tumor penetrating subserosal layer), but it must be noted that the probability of T4a stage (invasion of serosa) is as high as 40%. This means that the tumor is likely to have broken through the lamina propria of the gastric wall, and joint organ resection should be prepared during surgery, rather than simple local resection.

[0163] Task 3: Regional lymph node metastasis risk

[0164] Predicted probability distribution: the model outputs a lymph node metastasis probability of 72%.

[0165] Confidence interval (90%): [68%, 76%].

[0166] Individualized risk estimate (E[Y|θ]): high risk.

[0167] Clinical Interpretation: Extremely high risk of lymph node metastasis with very certain estimation (narrow confidence interval). Supports D2 lymphadenectomy rather than D1 surgery with a smaller range to ensure radicality.

[0168] Task 4: Probability of celiac trunk invasion

[0169] Predicted probability distribution: The model outputs a probability of 18% for vessel invasion.

[0170] Confidence interval (90%): [8%, 32%].

[0171] Individualized risk estimate (E[Y|θ]): Low risk.

[0172] Clinical Interpretation: Although the average probability is low, the confidence interval is wide and the upper limit reaches 32%, indicating that there is considerable uncertainty that cannot be ignored. The subsequent report will provide recommendations: "The tumor is closely related to the celiac trunk artery, although the probability of invasion is low, extra caution should be taken during intraoperative separation, and a blood vessel repair plan should be prepared."

[0173] Task 5: Risk of postoperative pancreatic fistula

[0174] Predicted probability distribution: The model outputs a probability of 30% for postoperative B / C grade pancreatic fistula.

[0175] Confidence interval (90%): [25%, 36%].

[0176] Individualized risk estimate (E[Y|θ]): High risk.

[0177] Clinical Interpretation: The system's predicted risk of pancreatic fistula is much higher than the average level of conventional statistics (usually <10%). This will guide intraoperative decision-making: (1) an abdominal drainage tube must be routinely placed; (2) postoperative drainage fluid amylase levels must be closely monitored; (3) patients and family members must be informed of this specific risk in advance.

[0178] In the embodiments of the present disclosure, the training process of the system is divided into two main stages:

[0179] First stage: independently train the lesion recognition and segmentation module 120, using a large amount of image data with pixel-level annotation, and the optimization target is the composite loss function .

[0180] Second stage: fix the parameters of the trained lesion recognition and segmentation module 120 as a feature extractor. Then, the feature extraction and screening module 130, the multi-modal feature fusion module 140, and the multi-task risk assessment module 150 are jointly trained in an end-to-end manner, and the optimization target is the distribution regression loss function At this stage, the encoder parameters of the optional frozen (not fine-tuned) lesion identification and segmentation module 120 can be frozen to prevent overfitting and accelerate training; if the data volume is very sufficient, it can also be fine-tuned at a very low learning rate.

[0181] The evaluation report generation module 160 is configured to generate a structured evaluation report according to the segmentation result and the multi-task prediction result.

[0182] Further, the structured evaluation report output by the evaluation report generation module 160 includes key segmentation result visualization, quantitative measurement value, prediction classification / probability, surgical difficulty prompt and risk warning, and supports PDF and DICOM SR formats. For example, the surgical difficulty prompt in the report can include: "The tumor is closely adhered to the superior pancreaticoduodenal artery, and an ultrasonic knife is recommended for separation to reduce the risk of bleeding"; the risk warning can include: "Based on the model prediction, the risk of postoperative pancreatic fistula for this patient is 30% (high risk), and it is recommended to place a drainage tube during surgery and closely monitor the amylase level after surgery".

[0183] The implementation of the entire system relies on the Python scientific computing ecosystem (such as NumPy, PyTorch / TensorFlow, scikit-learn, PyRadiomics) and GPU computing resources.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described modules can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0185] In the technical solution of the present application, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0186] It should be noted that in the embodiments of the present application, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "includes a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0187] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A precise preoperative assessment system for gastric antral cancer with pyloric obstruction based on AI image recognition, characterized by: include: Data acquisition and preprocessing module, used to acquire patients' structured clinical data and multimodal imaging data and perform data preprocessing; The lesion recognition and segmentation module uses an improved nnU-Net framework to perform lesion recognition and segmentation on preprocessed multimodal imaging data, outputting pixel-level segmentation results including the tumor body, pyloric obstruction area, regional lymph nodes, and adjacent key structures. a feature extraction and screening module, configured to extract image features from the segmentation results, screen features related to clinical indicators from the image features using a two-stage feature selection method driven by clinical relevance, and output a screened image feature vector; wherein the image features include morphological features, depth features, and radiomics features; A multimodal feature fusion module, configured to fuse the screened image features with the pre-processed structured clinical data to generate a multimodal fusion feature vector; A multi-task risk assessment module is used to model the multimodal fusion feature vector using a semi-structured distribution regression method, establish a multi-parameter risk prediction model, and output a multi-task prediction result; An evaluation report generation module is used to generate a structured evaluation report based on the segmentation result and the multi-task prediction result.

2. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 1 is characterized by: In the data acquisition and preprocessing module, the structured clinical data includes patient demographic characteristics, laboratory test indicators, comorbidity information, and tumor markers obtained from the clinical information system; the multimodal imaging data includes CT images, MRI images, EUS images, and endoscopic images; The preprocessing includes: performing spatial registration on multimodal imaging data using an elastic registration algorithm based on mutual information, adjusting window width and window position, resampling to isotropic resolution, and denoising; and performing outlier processing and missing value interpolation on structured clinical data.

3. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 1 is characterized by: The improved nnU-Net framework used in the lesion recognition and segmentation module includes: Introducing attention gate units and multi-scale feature pyramid structures into the encoder-decoder structure; A composite loss function is constructed based on Dice loss, weighted cross entropy loss and boundary loss function for optimization.

4. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 1 is characterized by: The feature extraction and screening module extracts image features from the segmentation results, including: Morphological characteristics included tumor volume, maximum diameter, surface area, sphericity, obstructed lumen cross-sectional area, and stenosis length ratio; The deep feature is the high-dimensional feature vector after global average pooling extracted by the nnU-Net encoder; Radiomics features include first-order statistics, texture features, shape features, and wavelet features extracted from tumor and lymph node ROIs.

5. The AI-based image recognition-based accurate preoperative assessment system for gastric antral cancer with pyloric obstruction according to claim 4 is characterized in that: The two-stage feature selection method driven by clinical relevance includes: Phase 1: Feature coarse screening based on statistical significance analysis and clinical prior knowledge, retaining features that are significantly associated with clinical endpoints; The second stage: Feature fine screening is performed based on a method combining stability selection and LASSO regression to obtain the final feature subset.

6. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 1 is characterized by: The multimodal feature fusion module uses a cross-modal feature fusion method based on an attention mechanism to fuse the image features with the pre-processed structured clinical data, specifically including: Project the image feature vector and clinical data vector into the feature space of the same dimension respectively; Using the clinical feature vector as the query and the image feature vector as the key and value, the cross-attention weight is calculated; The original features and attention-weighted features are fused through a gating mechanism to generate a fused feature vector.

7. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 1 is characterized by: The multi-task risk assessment module uses a semi-structured distribution regression method to model the conditional distribution of multiple target clinical indicators for the input multimodal fusion feature vector. Each distribution parameter is expressed in the form of a structured additive predictor: in, represents the kth parameter of the conditional distribution of the target variable, is the link function, is the predictor mapping expression for the kth distribution parameter, is the jth feature effect function, and v is the multimodal fusion feature of the patient.

8. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 7 is characterized by: The predictor is decomposed into three types of feature effect functions: Structured linear effect , used to model linear clinical factors; Structural nonlinear effects , physiological or laboratory indicators for modeling nonlinear effects; Unstructured DNN Effect , used to process high-dimensional semantic features extracted by image convolution or text attention mechanism.

9. The AI ​​image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 8 is characterized by: Adopting the orthogonalization mechanism to separate the structured and unstructured feature effects, we define is the projection matrix of the structured feature space, The orthogonal complement projection matrix or orthogonal complement space , I is the identity matrix; the latent features output by DNN Orthogonal complementary projection matrix Projection decomposition, we get , ensuring that the network learns the residual information that the structured model cannot cover; among them, It is the original, unprocessed high-dimensional feature representation extracted from the deep neural network DNN; is the orthogonalized depth feature vector.

10. The AI-based image recognition-based preoperative accurate assessment system for gastric antral cancer with pyloric obstruction according to claim 1 is characterized in that: The multi-task risk assessment module outputs multi-task prediction results through a multi-parameter risk prediction model, including: pyloric obstruction grade prediction, T stage probability prediction, regional lymph node metastasis risk prediction, key vascular invasion probability prediction, and postoperative complication risk prediction; The structured evaluation report output by the evaluation report generation module includes visualization of key segmentation results, quantitative measurement values, predicted grades / probabilities, surgical difficulty prompts, and risk warnings.

Images