A stas detection method and system based on multi-modal data features

Abstract

The application is suitable for the field of artificial intelligence and bioinformatics, and relates to a STAS detection method and system based on multi-modal data features, which comprises: preprocessing CT images and pathological WSI, and constructing a spatial topology graph of TME; extracting multi-scale feature representation of WSI through an inflation attention mechanism; performing graph convolution processing on the spatial topology graph of TME, and extracting feature representation of TME; extracting weighted feature representation of CT images through an attention mechanism; performing bidirectional interactive attention calculation on the multi-scale feature representation of WSI, the feature representation of TME and the weighted feature representation of CT images through a multi-head attention mechanism, and generating fusion features; performing STAS state classification based on the fusion features, and training the model by using a joint optimization strategy of supervised cross-modal contrast alignment loss and classification loss. The application significantly improves the accuracy, robustness and interpretability of STAS detection.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and bioinformatics, and particularly relates to a STAS detection method and system based on multimodal data features. Background Technology

[0002] STAS (Small Acinar Spread) refers to the spread of tumor cells (such as micropapillary clusters, solid cell nests, and single cells) into the alveolar spaces beyond the boundaries of the main tumor. Numerous studies have confirmed that STAS is closely associated with higher tumor recurrence rates, lymph node metastasis risk, and shorter overall survival, making it a significant high-risk pathological feature affecting the prognosis of lung cancer patients. Accurate detection of STAS is of crucial clinical significance for guiding surgical approaches (such as the choice between lobectomy and sublobar resection), adjuvant therapy decisions, and prognostic assessment.

[0003] Despite the significant clinical value of STAS diagnosis, current diagnostic methods, including intraoperative frozen sections and postoperative permanently embedded paraffin sections, face considerable challenges. Frozen sections are currently the primary means of intraoperative STAS assessment. However, the accuracy is greatly limited due to tissue shrinkage, cell deformation, and loss of microscopic details caused by the freezing process. Studies show that compared to postoperative paraffin sections, frozen sections have a sensitivity of only 44%-55% and an accuracy of approximately 74%-77% for STAS detection, resulting in a high rate of missed diagnoses. This can lead to intraoperative misinterpretation, affecting the determination of the optimal surgical plan (such as the extent of resection) and increasing the risk of postoperative recurrence. Paraffin-embedded sections are the gold standard for diagnosing STAS, offering high histological resolution. However, the diagnostic process heavily relies on manual reading by pathologists, making it a labor-intensive and inefficient task. The diverse microscopic morphology of STAS, with some subtypes (such as single cells) only identifiable under high magnification, coupled with variations in tissue section quality, processing artifacts, and subjective differences between observers, all contribute to the risk of missed or misdiagnosed STAS. This problem is particularly pronounced in regions where pathologists are scarce.

[0004] To overcome the above limitations, researchers began to explore the use of deep learning models to automatically extract information from medical images to assist in the diagnosis of STAS.

[0005] On the one hand, radiomics and deep learning models based on preoperative computed tomography (CT) scans have shown potential in predicting STAS. Studies have shown that STAS-positive patients more often exhibit features such as a high proportion of solid components, lobulation, spiculation, and vascular convergence on CT images. Some models have achieved good predictive performance in internal and external validation sets. However, CT-based methods have inherent limitations: First, radiomics features are easily affected by CT scan parameters (such as reconstructed kernel and slice thickness) and segmentation methods, resulting in poor feature stability and insufficient cross-center reproducibility. Second, these high-dimensional features are mostly combinations of mathematical models, lacking a direct and interpretable correlation with tumor biological behavior, making it difficult for clinicians to understand and trust their predictive basis.

[0006] On the other hand, deep learning methods based on postoperative whole-slice images (WSI) of histopathology have also been used for the diagnosis of STAS. For example, the STASNet model was developed, with an AUC of 0.72-0.78 for predicting STAS at the slice level. Another study proposed the VERN model, which achieved a high AUC value in internal validation. These methods directly analyze the gold standard images, theoretically offering greater advantages. However, existing pathological image analysis methods mostly employ the multi-instance learning (MIL) paradigm, focusing primarily on features at the image patch level. They struggle to effectively model the spatial topology of the tumor microenvironment (TME), which is crucial for understanding how tumor cells detach from the primary lesion and spread within air cavities. Furthermore, relying solely on pathological images for postoperative diagnosis, while more accurate than CT, still falls short of meeting the needs of preoperative decision-making.

[0007] With the development of multimodal data fusion technology, fusing CT imaging and histopathological image information is expected to comprehensively reveal the biological characteristics of tumors from both macroscopic (tumor morphology, boundaries, and relationship with surrounding structures) and microscopic (cell morphology, tissue structure, and tumor microenvironment) levels, thereby improving the ability to identify complex invasive behaviors such as STAS. However, existing multimodal fusion strategies (such as early, intermediate, and late-stage fusion) often fail to fully consider the multi-scale characteristics within different modalities (such as the low-magnification global features and high-magnification cellular features of pathological images) when handling STAS diagnostic tasks, and also lack deep cross-modal interaction mechanisms to effectively integrate complementary information from CT and pathology.

[0008] Therefore, how to effectively extract and fuse multi-scale features of pathological images, topological features of tumor microenvironment, and macroscopic features of CT images within a multimodal framework to achieve accurate detection of STAS and improve the interpretability and generalization ability of the model is a problem that urgently needs to be solved by researchers in this field. Summary of the Invention

[0009] To address the shortcomings of existing technologies, the purpose of this invention is to provide a STAS detection method based on multimodal data features, thereby solving the problem of low accuracy in existing STAS detection methods. In addition, this invention also provides a STAS detection system based on multimodal data features.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a STAS detection method based on multimodal data features, comprising the following steps:

[0012] S10. Preprocess CT (computed tomography) images and pathological WSI (whole slice scanning technology) images, extract CT image features and multi-scale pathological image block features respectively, and construct a spatial topology map of TME (tumor microenvironment);

[0013] S20. Process the multi-scale pathological image patch features through the dilated attention mechanism to extract the multi-scale feature representation of WSI;

[0014] S30. Perform graph convolution processing on the spatial topology graph of the TME to extract the feature representation of the TME;

[0015] S40. The CT image features are weighted using an attention mechanism to extract the weighted feature representation of the CT image;

[0016] S50. A bidirectional interactive attention calculation is performed on the multi-scale feature representation of the WSI, the feature representation of the TME, and the weighted feature representation of the CT image through a multi-head attention mechanism to generate fused features.

[0017] S60. Based on the fusion features, STAS state classification is performed, and a multimodal multiscale attention model (MAME model) is trained using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss. The trained model is then used for STAS detection.

[0018] Furthermore, the specific steps of S10 are as follows:

[0019] S101. Use the Otsu algorithm to segment the tissue region of WSI, and use a sliding window strategy to cut it into image blocks at at least two different magnifications;

[0020] S102. Perform color normalization processing on the segmented image blocks;

[0021] S103. Use a pre-trained pathology-based model to extract the feature vector of each image patch to form a multi-scale image patch feature set.

[0022] S104. Use a cell segmentation model to classify and segment cells in WSI, and construct the spatial topology graph of TME using the KNN nearest neighbor algorithm based on the spatial positional relationship of cells. In this graph, nodes represent cells and their related features, and edges represent the relationships between cells.

[0023] Furthermore, the specific steps of S20 are as follows:

[0024] S201. Add a learnable positional bias to the input multi-scale pathological image block feature sequence to obtain a feature representation containing positional information;

[0025] S202. Input the feature sequence containing location information into the stacked encoders. Each encoder is processed by an inflated self-attention mechanism. The inflated self-attention mechanism divides the long sequence into segments, performs dense attention calculation within the segments, and performs sparse attention calculation between the segments by a set inflated step size.

[0026] S203. Perform global average pooling on the length dimension of the sequence output by the encoder to obtain a fixed-dimensional WSI multi-scale feature representation.

[0027] Furthermore, S20 also includes step S204: weighted fusion of the multi-scale feature representation and the feature representation of the TME through an adaptive feature fusion unit, dynamically adjusting the contribution of pathological features from different scales or sources using a learnable weight vector, and concatenating and nonlinearly mapping the weighted features to output the fused pathological feature representation.

[0028] Furthermore, the specific steps of S30 are as follows:

[0029] S301. Input the spatial topology graph of the TME into the graph attention network, and use the GraphSAGE convolution operation to update the node representation by aggregating the features of the neighboring nodes of each node.

[0030] S302. Perform adaptive max pooling on the updated node feature set to convert it into a fixed-length TME feature vector.

[0031] Furthermore, the specific steps of S40 are as follows:

[0032] S401. Calculate attention weights for the input CT image features;

[0033] S402. The CT image features are weighted and summed according to the attention weights to generate a weighted feature representation of the CT image.

[0034] Furthermore, the specific steps of S50 are as follows:

[0035] S501. The multi-scale feature representation of WSI and / or the feature representation of TME are used as the first modal feature, and the weighted feature representation of CT image is used as the second modal feature.

[0036] S502. Using a multi-head attention mechanism, with the first modal feature as the query and the second modal feature as the key and value, calculate the attention representation of the first modal feature on the second modal feature;

[0037] S503. Using the second modal feature as the query and the first modal feature as the key and value, calculate the attention representation of the second modal feature on the first modal feature;

[0038] S504. Perform residual connection and layer normalization on the attention representations in both directions to obtain the cross-modal fusion features with enhanced interaction.

[0039] Furthermore, the total loss function of the model in S60 is as follows:

[0040]

[0041] in, This is a supervised cross-modal contrastive alignment loss, used to bring cross-modal feature distances closer to similar samples and widen the distance between dissimilar samples; To supervise the binary cross-entropy classification loss; These are time-varying weighting coefficients used to stabilize the model in the early stages of training.

[0042] Secondly, the present invention also provides a STAS detection system based on multimodal data features, comprising:

[0043] The multimodal data preprocessing module is used to preprocess CT images and WSI, extract CT image features and multi-scale pathological image block features respectively, and construct the spatial topology map of TME;

[0044] A multi-scale pathology expert module, connected to the multi-modal data preprocessing module, is used to process the multi-scale pathological image patch features through an dilated attention mechanism and extract the multi-scale feature representation of WSI.

[0045] The graph attention network module is connected to the multimodal data preprocessing module and is used to perform graph convolution processing on the spatial topology graph of the TME to extract the feature representation of the TME.

[0046] The image attention expert module, connected to the multimodal data preprocessing module, is used to weight the CT image features through an attention mechanism and extract the weighted feature representation of the CT image.

[0047] A cross-modal bidirectional attention fusion module is connected to the multi-scale pathology expert module, the graph attention network module, and the image attention expert module, respectively. It is used to perform bidirectional interactive attention calculation on the multi-scale feature representation of WSI, the feature representation of TME, and the weighted feature representation of CT image through a multi-head attention mechanism to generate fused features.

[0048] The classification and optimization module is connected to the cross-modal bidirectional attention fusion module. It is used to perform STAS state classification based on the fusion features and to train the entire system using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss.

[0049] The STAS detection method and system based on multimodal data features provided by this invention have at least the following advantages compared with the prior art:

[0050] Existing STAS detection methods have relatively low accuracy. This invention, by fusing macroscopic information from CT images with microscopic information from pathological images, and combining multi-scale feature extraction and deep cross-modal interaction mechanisms, significantly improves the accuracy, robustness, and interpretability of STAS diagnosis, providing strong support for precision diagnosis and treatment of lung cancer. This invention is adaptable to the differences in data distribution across different medical centers, exhibiting good universality across centers, devices, and populations, laying a solid foundation for its widespread clinical application. This invention utilizes multiple innovative modules working collaboratively. The multi-scale pathology expert module employs an expanded self-attention mechanism, effectively addressing the computational complexity issue in processing long WSI sequence data while preserving the global and local features of pathological images at different magnifications, thus aligning more closely with the actual diagnostic thinking of pathologists. The graph attention network module introduces the spatial topology of the TME into STAS diagnosis for the first time, modeling the spatial relationships and interactions between cells through graph neural networks, uncovering microenvironment features that are difficult to capture using traditional methods, and providing a new technical perspective for understanding tumor invasion behavior. The cross-modal bidirectional attention fusion module achieves bidirectional interaction between CT and pathological features through multi-head attention, enabling the information from the two modalities to mutually enhance and complement each other, generating a more information-rich fusion representation, avoiding the problem of simple superposition or loss of modal information in traditional fusion methods. The image attention expert module can output the attention weight of each CT feature. The cross-modal bidirectional attention mechanism provides attention relationships between modalities, making the model's decision-making process interpretable and enhancing clinicians' trust and willingness to adopt it. The joint optimization strategy proposed in this invention, which combines supervised cross-modal contrastive alignment loss and binary cross-entropy classification loss, uses the InfoNCE loss function to bring the feature representations of similar cross-modal samples closer together and push away dissimilar samples, thereby aligning CT features and pathological features in a common semantic space. This enhances the consistency and discriminative power of cross-modal features. The piecewise warming strategy (λ(t) gradually increases from 0 to the target value) allows the model to prioritize learning the classification boundary in the early stages of training and gradually strengthen cross-modal alignment in the later stages, avoiding convergence instability or representation collapse caused by excessive constraints in the early stages. Attached Figure Description

[0051] To more clearly illustrate the solution of the present invention, a brief introduction will be given to the drawings used in the description of the embodiments below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 A flowchart of a STAS detection method based on multimodal data features provided in an embodiment of the present invention;

[0053] Figure 2The ROC curve of the multimodal multiscale attention model in the five-fold cross-validation for diagnosing multimodal data in a STAS detection method based on multimodal data features provided in an embodiment of the present invention;

[0054] Figure 3 The PRC curve of the multimodal multiscale attention model in the five-fold cross-validation for diagnosing multimodal data in a STAS detection method based on multimodal data features provided in an embodiment of the present invention;

[0055] Figure 4 This diagram illustrates the accuracy, precision, recall, F1 score, specificity, and AUC value of a multimodal multiscale attention model trained with five-fold cross-validation in a STAS detection method based on multimodal data features provided in this embodiment of the invention.

[0056] Figure 5 This is a schematic diagram of the confusion matrix used by the multimodal multiscale attention model to diagnose multimodal data in a STAS detection method based on multimodal data features provided in an embodiment of the present invention;

[0057] Figure 6 ROC and PRC curves of a multimodal multiscale attention model validated on the XY-CSU multimodal dataset in a STAS detection method based on multimodal data features provided in this embodiment of the invention;

[0058] Figure 7 ROC and PRC curves of a multimodal multiscale attention model validated on the SXY-CSU multimodal dataset in a STAS detection method based on multimodal data features provided in this embodiment of the invention;

[0059] Figure 8 ROC and PRC curves of a multimodal multiscale attention model validated on the FAH-NHU multimodal dataset in a STAS detection method based on multimodal data features provided in this embodiment of the invention;

[0060] Figure 9 ROC and PRC curves of the multimodal multiscale attention model validated on the PCPH multimodal dataset in a STAS detection method based on multimodal data features provided in this embodiment of the invention;

[0061] Figure 10 This is a schematic diagram of the confusion matrix used in a STAS detection method based on multimodal data features provided in an embodiment of the present invention to validate a multimodal multiscale attention model on multicenter multimodal datasets such as XY-CSU, SXY-CSU, FAH-NHU, and PCPH.

[0062] Figure 11 The correction curves of the multimodal multiscale attention model in the STAS detection method based on multimodal data features provided in this embodiment of the invention are validated on the XY-CSU, SXY-CSU, FAH-NHU, and PCPH multicenter multimodal datasets.

[0063] Figure 12 This invention provides a schematic diagram illustrating the Specificity, Sensitivity, Accuracy, NPV, and PPV values ​​of a multimodal multiscale attention model validated on the XY-CSU, SXY-CSU, FAH-NHU, and PCPH multicenter multimodal datasets in a STAS detection method based on multimodal data features.

[0064] Figure 13 This is a structural block diagram of a STAS detection system based on multimodal data features provided in an embodiment of the present invention. Detailed Implementation

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. For example, terms such as “length,” “width,” “upper,” “lower,” “left,” “right,” “front,” “rear,” “vertical,” “horizontal,” “top,” “bottom,” “inner,” and “outer” indicate orientations or positions based on the accompanying drawings and are for ease of description only, and should not be construed as limiting the technical solution. The terms “comprising” and “having,” and any variations thereof, in the specification, claims, and accompanying drawings of this invention, are intended to cover non-exclusive inclusion. The terms “first,” “second,” etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects and not to describe a particular order. In the specification, claims, and accompanying drawings of this invention, when an element is referred to as “fixed to,” “mounted to,” “set on,” or “connected to” another element, it may be directly or indirectly located on that other element. For example, when an element is referred to as “connected to” another element, it may be directly or indirectly connected to that other element. Furthermore, the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0066] This invention provides a STAS detection method based on multimodal data features, applied to the detection of STAS in lung cancer patients. The STAS detection method based on multimodal data features includes the following steps:

[0067] S10. Preprocess CT images and pathological WSIs to extract CT image features and multi-scale pathological image patch features, and construct the spatial topology map of the TME. S20. Process multi-scale pathological image patch features using an dilated attention mechanism to extract multi-scale feature representations of the WSI. S30. Perform graph convolution processing on the spatial topology map of the TME to extract feature representations of the TME. S40. Weight CT image features using an attention mechanism to extract weighted feature representations of the CT images. S50. Perform bidirectional interactive attention calculation on the multi-scale feature representations of the WSI, the feature representations of the TME, and the weighted feature representations of the CT images using a multi-head attention mechanism to generate fused features. S60. Perform STAS state classification based on the fused features, and train a multi-modal multi-scale attention model using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss. The trained model is then used for STAS detection.

[0068] This invention significantly improves the accuracy, robustness, and interpretability of STAS diagnosis by fusing macroscopic information from CT images with microscopic information from pathological images, combined with multi-scale feature extraction and deep cross-modal interaction mechanisms, providing strong support for the precision diagnosis and treatment of lung cancer.

[0069] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0070] This invention provides a STAS detection method based on multimodal data features, applied to the detection of STAS in lung cancer patients, combining... Figures 1 to 12 In this embodiment, the STAS detection method based on multimodal data features includes the following steps:

[0071] S10. Preprocess the CT images and pathological WSI, extract the CT image features and multi-scale pathological image block features respectively, and construct the spatial topology map of the TME.

[0072] Specifically, in this embodiment, since pathologists analyze and diagnose based on 20X WSI using multi-scale features, including global pathological features under low magnification, tissue features under medium magnification, and cellular features under high magnification, a multi-scale feature extraction method is used in the experiment to extract pyramidal structure features from the WSI. The specific steps of step S10 are as follows:

[0073] S101. The Otsu algorithm is used to automatically calculate the threshold to segment the tissue, background and blurred regions of WSI. The algorithm determines the optimal threshold by minimizing the intra-class variance and divides the 20X-based WSI into 256*256 and 512*512 patches respectively by using a sliding window strategy, and obtains the spatial coordinates of the patches.

[0074] S102. To mitigate the impact of differences in scanning equipment and slide quality on the model's generalization performance, all blocks in the dataset were preprocessed and normalized using the Macenko normalization algorithm before feature extraction from 256*256 and 512*512 patches using Prov-Gigapath and CONCHv1.5.

[0075] S103. Using a pre-trained pathology-based model, feature vectors are extracted from each image patch to form a multi-scale image patch feature set. Each patch, after being processed by the Prov-Gigapath and CONCHv1.5 models, is uniformly represented as a one-dimensional vector with lengths s of 1536 and 768. Each WSI can be represented as a set of two types of multiple one-dimensional features. ), where n represents the number of patches.

[0076] S104. To construct the tumor microenvironment features in WSI, a cell segmentation model (HD-Yolo) was used to classify and segment cells in WSI. Based on the spatial relationships of cells, the spatial topology map of TME was constructed using the KNN (K=9) nearest neighbor algorithm. The graph uses nodes and edges, where nodes represent cells and their associated features, and edges represent the relationships between cells. Therefore, the TME features of WSI can be represented as follows: ,Right now The WSI preprocessing process for the internal training and validation sets is consistent with that for the multi-center test set, including patch segmentation, feature extraction, and construction of the spatial topology graphs for both WSI and TME.

[0077] In this embodiment, all patients underwent a CT scan before surgery. The open-source software ITK-SNAP (Version 3.6, www.itksnap.org) was used to manually segment the region of interest, and the lesion outline was manually drawn layer by layer on the lung cancer lesion image and then verified. For patients with multiple lesions, segmentation was performed only on the tumor lesions obtained from biopsy or surgery. All doctors operated under a double-blind system, extracting a total of 1317 features, including 14 shape features, 18 first-order statistical features, 24 Gy-level co-occurrence matrix (GLCM) features, 14 Gy-level dependency matrix (GLDM) features, 16 Gy-level running length matrix (GLRLM) features, 16 Gy-level size region matrix (GLSZM) features, and 5 adjacent gray-level difference matrix (NGTDM) features, totaling 107 features in the original domain. Furthermore, the above first-order and texture features (GLCM, GLRLM, GLSZM, GLDM) were repeatedly calculated on 5 derived domains (exponential, gradient, logarithm, square, squareroot). The NGTDM features were used, with 93 features per domain, resulting in a total of 465 features derived from the domains. Simultaneously, similar features were calculated on 8 wavelet subbands (LLL, LLH, LHL, HLL, LHH, HLH, HHL, HHH), for a total of 744 wavelet-filtered features. Large blood vessels and bronchi were excluded from the nodule volume as much as possible. The area within 3 mm of the tumor region was defined as the peritumoral region, and similar feature extraction was performed on the peritumoral region. The extracted features were normalized using the min-max normalization method.

[0078] S20. Multi-scale pathological image block features are processed through the dilated attention mechanism to extract multi-scale feature representations of WSI.

[0079] Specifically, in this embodiment, step S20 consists of the following steps:

[0080] S201. Add a learnable positional bias to the input multi-scale pathological image block feature sequence to obtain a feature representation containing positional information.

[0081] S202. Input the feature sequence containing location information into the stacked encoders. Each encoder uses an inflated self-attention mechanism for processing. The inflated self-attention mechanism divides the long sequence into segments, performs dense attention calculation within each segment, and performs sparse attention calculation between segments using a set inflated step size.

[0082] S203. Perform global average pooling on the length dimension of the sequence output by the encoder to obtain a fixed-dimensional WSI multi-scale feature representation.

[0083] S204. An adaptive feature fusion unit is used to perform weighted fusion of multi-scale feature representation and TME feature representation. The contribution of pathological features from different scales or sources is dynamically adjusted using a learnable weight vector. The weighted features are then concatenated and nonlinearly mapped to output the fused pathological feature representation.

[0084] In this embodiment, each WSI image can be represented as a long sequence of data with location information at multiple scales. Standard self-attention mechanisms face the problem of huge computational and memory consumption. To address this issue, a Multi-Scale Pathology Expert (MSPE) module is proposed, aiming to extract high-dimensional representations from the features of patches at different scales in histopathological images and weight these features through an attention mechanism. The MSPE module consists of four encoders, one normalization layer (LN), and one fully connected layer. The encoder processing method mainly uses dilated attention (DSA) and self-attention mechanisms to effectively process long sequence data. Its feature processing mainly includes feature location encoding, dilated attention mechanism, feature forward propagation, and feature normalization.

[0085] First, to enable the model to learn contextual information at each location, learnable biases allow the model to automatically adjust the weights of location representations based on data during training, thus enabling the model to adaptively learn the most useful location information. Compared to traditional location encoding methods, learnable biases are simpler and more effective. Specifically, input features... and position encoding The model is encoded in the following way:

[0086] ;

[0087] in, It is a feature representation that includes positional encoding. This is a learnable positional bias. Next, the Encoder uses Dilated Self-Attention (DSA) to divide the input sequence into several segments. Dense attention computation is performed within each segment, while sparse attention computation is used between segments. This simulates a sparse distribution of long-term dependencies. The attention computation for each segment only focuses on segments closer to the current segment, reducing computational cost. Specifically, it divides the sequence of each WSI feature into multiple segments, each segment having a length of... Within each segment, the model uses a dense attention mechanism for computation; while between segments, the model employs an dilated attention mechanism, which performs sparse sampling between adjacent segments by setting a dilation step size. This design can simulate the pattern of "the farther the distance, the sparser the attention allocation" in long sequences, thereby significantly reducing computational complexity while preserving long-distance contextual information.

[0088] The self-attention mechanism is the core of dilated self-attention. It weights each element by calculating the similarity between elements in the input sequence to obtain the contextual information of each element. Specifically, the attention mechanism can be understood as a weighted summation of each feature, where the weights are dynamically calculated based on the correlation between features. Specifically, given an input matrix... ,in, It refers to the batch size. It is the length of the feature sequence (i.e., the number of patches). This refers to the dimension of each feature, and the attention calculation process is as follows:

[0089] ;

[0090] in, For attention matrix; , , These represent the query vector, key vector, and value vector, respectively. It is the dimension of the key; For the learned weight matrix, a sparse strategy is adopted for cross-segment attention, focusing only on partial information from adjacent segments. Key / value pairs are sparsely sampled from several adjacent segments through dilation. Next, a feedforward network is used to output a vector representation of the WSI feature sequence, consisting of two fully connected (FC) layers and... The activation functions are used to process each feature. To maintain the stability of the model, the output features are... The model goes through a normalization layer (LN) to ensure that the features input to each layer have a uniform scale, thereby accelerating the training process and improving model performance. Finally, a linear layer maps the model to the final dimension. This yields feature representations at different scales for each WSI. The process is as follows:

[0091] ;

[0092] The process of each Encoder processing features from different scale patches of WSI is the same. The features processed by the Encoder need further normalization. The feature extraction process is as follows:

[0093] ;

[0094] After four encoders and normalization processes, the output sequence is finally mapped through a fully connected (FC) layer to obtain the final representation. To further enhance the representation of global information, the model also performs global average pooling on the output sequence along the length dimension (L dimension) to obtain a fixed-dimensional representation vector. The pooling operation is as follows:

[0095] ;

[0096] in, For each location's feature vector, This is the sequence length. The pooled features are used for subsequent feature fusion. Ultimately, WSI's 20 and 10 At the specified magnification, after feature extraction by the MSPE module, the output features of all patches can be represented as follows: .

[0097] Given that features at multi-scale pathology (such as 20x and 10x magnification) often carry complementary information to TME, an adaptive feature fusion module is proposed to fuse multi-scale pathological information. The adaptive feature fusion uses learnable modal weights as the front-end gating, and then performs nonlinear mapping on the concatenated features through a small MLP to obtain a unified low-dimensional fused representation. This structure combines adaptability (weight learning) and expressiveness (nonlinear projection), and the number of parameters and computational cost are controllable. Specifically, assuming the multi-scale features are represented as and , a learnable scalar weight vector is designed to dynamically learn the multi-scale features, and the multi-scale features are concatenated and output. The process can be represented as:

[0098] ;

[0099] Intuitively, in this process, when the noise of a certain modality is high or the correlation is weak, the module will automatically adjust the feature output. The final output features need to go through two layers of perceptron and layer normalization operations, which can be represented as:

[0100] ;

[0101] in, for Activation function These are learnable weights. Final output. It serves as input to the downstream classification head or cross-modal alignment loss.

[0102] S30. Perform graph convolution processing on the spatial topology graph of TME to extract the feature representation of TME.

[0103] Specifically, in this embodiment, step S30 consists of the following steps:

[0104] S301. Input the spatial topology graph of TME into the graph attention network, and use GraphSAGE convolution operation to update the node representation by aggregating the features of the neighboring nodes of each node.

[0105] S302. Perform adaptive max pooling on the updated node feature set to convert it into a fixed-length TME feature vector.

[0106] In this embodiment, given the spatial topology map of the TME provided by the histopathological image, a graph attention network module is designed specifically to process the output graph structure data. This module employs a graph neural network (GNN) structure to learn the nodes and edges of the TME's spatial topology map to extract STAS-related features from the TME. Specifically, the graph attention network uses SAGEConv (GraphSAGE convolution), a convolution operation commonly used in graph data, which computes node representations by aggregating features of neighboring nodes. Given the node feature matrix in the graph... edge structure of graph The output of this convolution operation is:

[0107] ;

[0108] in, It is a characteristic of the node. This refers to the edge information of the graph. SAGEConv updates the node representation by aggregating the neighborhood information of each node. Then, the graph attention network uses the ReLU activation function and layer normalization to ensure that the features maintain a good distribution across multiple layers. The process is as follows:

[0109] ;

[0110] To obtain a fixed-length feature vector, the model... Adaptive MaxPool1d is performed. This pooling operation automatically adjusts the feature dimension of the output based on the length of the input sequence, so that the final output has a fixed length dimension. The process is as follows:

[0111] ;

[0112] To ensure consistency of output features, the model uses LN to standardize the pooled results:

[0113] ;

[0114] Ultimately, the output It is a node feature representation after multi-layer processing. It can effectively capture important pathological features related to STAS in TME and provide powerful feature input for subsequent feature fusion.

[0115] S40. Weight the CT image features using an attention mechanism and extract the weighted feature representation of the CT image.

[0116] Specifically, in this embodiment, step S40 consists of the following steps:

[0117] S401. Calculate attention weights for the input CT image features.

[0118] S402. The CT image features are weighted and summed according to the attention weights to generate a weighted feature representation of the CT image.

[0119] In this embodiment, an attention expert module is proposed in the MAME model. This module employs a simplified attention mechanism to extract weighted features from the CT features and applies them to downstream classification tasks. The module dynamically assigns weights to each feature through the attention mechanism, thereby improving the model's ability to focus on important features and further enhancing its performance. Specifically, assuming the input CT matrix features are... , Indicates batch size, This is the input feature dimension. An attention mechanism is used to extract feature representations and generate corresponding attention weights. Then, a fully connected layer is used to process the formaldehyde-treated features. Mapping to target dimension Layer normalization is used to process the output, ensuring the stability of the training process. The process is as follows:

[0120] ;

[0121] The attention expert module not only effectively extracts key information from CT images but also calculates the attention weight for each feature, providing interpretability for the model's decision-making process. In this way, the CT expert module provides a powerful feature representation for subsequent feature fusion, enhancing model performance. This attention-based feature weighting method allows the model to focus on important feature regions and suppress irrelevant features when processing CT images, thereby improving the model's accuracy and robustness in complex tasks.

[0122] S50. A bidirectional interactive attention calculation is performed on the multi-scale feature representation of WSI, the feature representation of TME, and the weighted feature representation of CT images through a multi-head attention mechanism to generate fused features.

[0123] Specifically, in this embodiment, step S50 consists of the following steps:

[0124] S501. The multi-scale feature representation of WSI and / or the feature representation of TME are used as the first modal feature, and the weighted feature representation of CT images is used as the second modal feature.

[0125] S502. Using a multi-head attention mechanism, with the first modal feature as the query and the second modal feature as the key and value, calculate the attention representation of the first modal feature on the second modal feature.

[0126] S503. Using the second modal feature as the query and the first modal feature as the key and value, calculate the attention representation of the second modal feature on the first modal feature.

[0127] S504. Perform residual connection and layer normalization on the attention representations in both directions to obtain the cross-modal fusion features with enhanced interaction.

[0128] In this embodiment, a cross-modal bidirectional attention fusion module is proposed during the multimodal feature fusion process. This module aims to enhance the model's representational capabilities by enabling different features to influence and reinforce each other through deep interactive processing. Specifically, the cross-modal bidirectional attention fusion module performs bidirectional interactive attention calculations on multi-scale features and CT image features using a multi-head attention mechanism. First, the input multi-scale pathological features... and CT imaging features The features need to be projected onto the input features through a Linear layer to generate the query. , ),key( , ),value( , Multi-head attention mechanism for calculating the attention of pathological image features to CT image features ( ) and the attention of CT image features to pathological image features ( The processes are as follows:

[0129] ;

[0130] Under a multi-headed mechanism, the aforementioned attention is actually individual The subspaces are executed in parallel, and splicing and linear integration are performed at the head dimension, thus achieving "joint attention across different representation subspaces." The weighted results of pathological features and CT image features are subjected to residual summation and layer normalization to obtain the final interactive output.

[0131] ;

[0132] Standardizing the feature channels within a single sample can maintain consistent normalization behavior and stabilize hidden state dynamics across different batch sizes.

[0133] In summary, the cross-modal bidirectional attention fusion module takes bidirectional cross-modal multi-head attention as its core, and through... and Two reciprocal interactions between them explicitly model the cross-modal dependency of "who should care about whom"; then, residuals and Solid training, output ( and This design incorporates attention weights across modalities. It maintains consistency with the Transformer paradigm and collaborative attention literature in principle, while achieving extremely low computational overhead and good interpretability through a single-token interface, facilitating direct reuse in multimodal classification and alignment losses.

[0134] S60. STAS state classification is performed based on fusion features, and a multimodal multiscale attention model is trained using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss. The trained model is then used for STAS detection.

[0135] Specifically, in this embodiment, the total loss function of the model in step S60 is as follows:

[0136] ;

[0137] in, This is a supervised cross-modal contrastive alignment loss, used to bring cross-modal feature distances closer to similar samples and widen the distance between dissimilar samples; To supervise the binary cross-entropy classification loss; These are time-varying weighting coefficients used to stabilize the model in the early stages of training.

[0138] In this embodiment, a classification and optimization module is proposed to perform the classification task. This module adaptively assigns modality weights to the feature vectors of the pathology modality and the CT modality, and then performs a learnable weighted fusion to complete the mapping. Assume that the features of the CT modality and the pathology modality are respectively... , Learnable weight scalars The adaptive weighted fusion process is as follows:

[0139] ;

[0140] Next, the adaptive classification head uses a two-layer perceptron to output classification based on the features. Assume the weights and biases of the first perceptron are... The weights and biases of the second-layer perceptron are: and adopt Activation and Dropout:

[0141] ;

[0142] The final output can be represented as:

[0143] ;

[0144] To simultaneously improve discriminative performance and cross-modal consistency, a joint optimization of supervised cross-modal contrastive alignment loss and supervised binary cross-entropy classification loss is employed. Assume a mini-batch is... ,in , These are embeddings of pathological features and CT features, respectively. For category labels. First, the embedded modal features are normalized, i.e. Define the cross-modal similarity matrix:

[0145] ;

[0146] And construct a supervisory mask A symmetrical InfoNCE objective is adopted, namely attention feature fusion from pathological features to CT features (…). Attention feature fusion from CT features to pathological features ( ) and CT feature to pathological feature ). This principle promotes closer proximity of similar cross-modal samples within a common semantic space, utilizing the heterogeneity principle. Extensive experiments have shown that this can be represented as:

[0147] ;

[0148] Secondly, the binary classification result output by the classifier. , tag as Then its supervised binary cross-entropy classification loss can be expressed as:

[0149] ;

[0150] Therefore, the total loss function of the optimization model is:

[0151] ;

[0152] in, To stabilize the model during early training by staged temperature increases. (Previous) The number of epochs increases linearly from 0 to... Then, keep the constant. This allows you to first learn to identify the boundary and then gradually strengthen cross-modal alignment, avoiding excessively strong constraints in the early stages that could lead to convergence instability or representation collapse.

[0153] To improve the accuracy and efficiency of postoperative STAS diagnosis, a multimodal multiscale attention model (MAME) was trained using CT radiomics data and PS data from 562 lung cancer patients provided by a hospital to diagnose whether patients had STAS. Figures 2 to 5 The performance of the MAME model trained using the five-fold cross-validation method was analyzed. For multimodal patient data diagnosis, the model obtained from the third fold of the five-fold cross-validation method showed an AUC of approximately 0.9694 and a PRC of 0.9598 in the in-domain test. The average AUC and PRC of the five-fold cross-validation model in the in-domain test were 0.9554 and 0.9419, respectively. Furthermore, the model obtained from the third fold cross-validation method exhibited the best predictive performance, with accuracy, AUC, precision, recall, and F1 score of approximately 0.9286, 0.9636, 0.9291, 0.9286, and 0.9285, respectively. The average accuracy, AUC, precision, recall, and F1 score of the five-fold cross-validation results were approximately 0.919, 0.9461, 0.921, 0.92, and 0.9199, respectively. Therefore, MAME achieved good performance in patient-level STAS prediction. Secondly, the confusion matrices of the first, second, and fourth folds in the five-fold cross-validation results showed high false negatives and low false positives; the confusion matrices of the second and fifth folds showed low false negatives and high false positives, suggesting that the model may not adequately identify certain types of samples (such as certain variants in the positive class). However, overall, the five confusion matrices indicate that MAME can accurately diagnose both STAS and non-STAS patients. In summary, MAME trained using the five-fold cross-validation method based on multimodal data achieves clinical-level performance in diagnosing whether lung cancer patients have STAS.

[0154] To verify the performance of the MAME model trained on multimodal data as superior to the single-modal model, 12 radiomics models were first trained using five-fold cross-validation based on CT data from SXY-CSU. These models were RF

[35] , LR

[36] , SVC

[37] , KNN

[38] , DT

[39] , GNB

[40] , GBC

[41] , LSTM

[42] , CNN

[43] , MLP

[44] , FCN

[45] , and FTTransformer

[46] . Next, 11 multi-instance learning models were trained on PS data from SXY-CSU (ABMIL

[49] , ACMIL

[50] , DGR-MIL

[51] , DSMIL

[52] , HIPT

[53] , TransMIL

[54] , DTFD_MIL

[55] , CLAM_MB

[56] , CLAM_SB

[56] , IBMIL

[57] , and PatchGCN

[58] ). In each training phase, four out of the five folds are used to train the model, and the remaining fold is used to validate the model's performance. A detailed analysis of the diagnosis of STAS using unimodal and multimodal models will be conducted separately. First, the results of the five-fold cross-validation for the unimodal model in Table 1 will be analyzed. Specifically, 12 models trained on CT data were evaluated, with metrics including accuracy, AUC, precision, recall, and F1 score. The MLP model performed best, with an accuracy of 0.81 (95% CI: 0.78–0.84) and an AUC of 0.84 (95% CI: 0.80–0.89). Its precision, recall, and F1 score were all relatively balanced (all above 0.80). The RF and GBC models followed closely behind, but their precision and recall were slightly lower. In comparison, SVC performed poorly, with an accuracy of only 0.49 (95% CI: 0.47–0.52) and an AUC of 0.44 (95% CI: 0.34–0.53). Despite its high precision, its recall was very low. LR and KNN also performed only moderately, with accuracies of 0.69 and 0.61, respectively. The LSTM model, however, demonstrated good performance, with an accuracy of 0.76 (95% CI: 0.72–0.80) and an AUC of 0.82 (95% CI: 0.79–0.85). The GNB model performed poorly overall, with both accuracy and AUC being low. While FCN and FTTransformer models showed promise, further optimization is still needed. Overall, MLP and RF performed best in CT-based diagnosis, achieving a good balance in precision, recall, and AUC, while other models (such as SVC and FTTransformer) require further improvement for better clinical application. Secondly, 11 MIL models were evaluated based on FS data.TransMIL, based on the Transformer, performed best with an accuracy of 0.91 (95% CI: 0.88–0.94) and an AUC of 0.94 (95% CI: 0.90–0.98), exhibiting high precision (0.92), recall (0.91), and F1 score (0.91). ABMIL followed closely with an accuracy of 0.90 (95% CI: 0.83–0.96) and an AUC of 0.93 (95% CI: 0.88–0.97), also achieving high precision, recall, and F1 score (0.90). Other models, including ACMIL and DGR-MIL, showed stable performance with accuracies of 0.89 and 0.91, respectively; DSMIL and HIPT also performed well with accuracies of 0.89 and 0.90, respectively. While the PatchGCN model performed slightly worse (accuracy: 0.87, AUC: 0.89), it still showed excellent performance in terms of precision, recall, and F1 score. Overall, TransMIL, ABMIL, and ACMIL performed best on WSI-based tasks. However, the overall diagnostic performance of unimodal models remained lower than that of MAME models, highlighting the need for multimodal data to diagnose whether patients have STAS.

[0155] Importantly, to verify the superior performance of the MAME model compared to the multimodal model, the results of five-fold cross-validation were analyzed for six models with different fusion strategies (BPF_fusion

[65] , FSV_fusion

[73] , FWA_fusion

[66] , mid_fusion

[67] , cross_fusion

[68] , and FSC_fusion

[64] ). Extensive experimental results show that the multimodal model exhibits excellent classification performance across multiple evaluation metrics. All models achieve high accuracy, exceeding 0.88, with narrow 95% confidence intervals, indicating robust performance across different data trade-offs. Among them, cross_fusion and FSC_fusion have the highest accuracy, reaching 0.90 (95% CI, 0.88-0.91), demonstrating superior stability and efficiency. Regarding the AUC metric, which reflects the model's ability to distinguish classes, all models achieved AUC values ​​exceeding 0.90, with FWA_fusion and mid_fusion exhibiting the highest AUC values ​​at 0.92 (95% CI 0.91–0.92 and 0.88–0.96, respectively). These results highlight the models' strong discriminative power. All models maintained high precision, with mid_fusion and cross_fusion achieving the highest precision at 0.90 (95% CI 0.86–0.94 and 0.88–0.92, respectively), demonstrating their superior ability to correctly identify positive samples. Furthermore, all multimodal models consistently maintained high recall rates, all exceeding 0.87. FWA_fusion achieved a recall rate as high as 0.89 (95% CI 0.87–0.91), indicating excellent sensitivity in capturing positive samples. The F1 score further confirms the overall effectiveness of the models. Cross-fusion achieved the highest F1 score of 0.90 (95% CI 0.88–0.91), exceeding 0.88, reflecting its optimal balance between precision and recall. Overall, these models demonstrated excellent classification ability and consistency, with Cross-fusion and FSC-fusion performing particularly well. However, MAME outperformed other multimodal baseline models in accuracy, AUC, precision, recall, and F1 score, highlighting its robustness and applicability in STAS diagnosis and laying a solid foundation for further applications and multicenter generalization.

[0156] Table 1

[0157]

[0158] To verify the multi-center generalization ability of the MAME model, validation was performed on external datasets including XY-CSU, TXY-CSU, FAH-NHU, and PCPH. The collection and processing procedures for the multimodal data were the same as those used for the data collected in SXY-CSU. Figures 6 to 12First, the MAME model achieved an AUC of 0.9468 and a PRC of 0.911 on the XY-CSU dataset, demonstrating excellent overall classification ability, particularly strong performance in handling positive class tasks, making it suitable for class imbalance problems. On the TXY-CSU dataset, the MAME model achieved an AUC of 0.8509 and a PRC of 0.8271, indicating good overall classification performance and effectively distinguishing between positive and negative classes. On the FAH-NHU dataset, the MAME model achieved an AUC of 0.8722 and a PRC of 0.8586, demonstrating excellent diagnostic performance. On the PCPH dataset, the MAME model achieved an AUC of 0.8321 and a PRC of 0.8176, showing good classification ability, but there is still room for improvement compared to other datasets. Furthermore, tests on multi-center, multimodal datasets show that the MAME model performs differently across various datasets. Overall, the model excels in identifying negative classes, particularly in the XY-CSU and TXY-CSU datasets, where it achieves high accuracy in negative class prediction. However, identifying positive classes presents challenges on some datasets, especially the PCPH dataset, where positive class recall is low. Although the model achieves high accuracy on the FAH-NHU and PCPH datasets, further optimization is needed for positive class precision and recall. Moreover, the calibration curves of the MAME model on the four datasets demonstrate varying calibration effects. On the XY-CSU and TXY-CSU datasets, the model performs well with low Brier scores, indicating a small discrepancy between predicted probabilities and actual labels. The calibration curve for the FAH-NHU dataset shows an ideal trend, despite slight bias; conversely, the PCPH dataset exhibits greater calibration bias and a higher Brier score, suggesting room for improvement in the model's prediction accuracy on this dataset. Finally, the model demonstrates differences in dataset-specific characteristics in the test results. In the XY-CSU dataset, the model exhibits high specificity (0.9661) and negative predictive value (NPV, 0.9844), indicating its particularly accurate identification of negative samples. However, its low positive predictive value (PPV, 0.76) suggests that its accuracy in positive class prediction is not as good as that in negative class prediction. The TXY-CSU dataset shows extremely high sensitivity (0.9808), indicating excellent performance in identifying positive samples, but its low specificity (0.5455) results in poor performance in identifying negative samples. The FAH-NHU dataset demonstrates good balanced performance, with high sensitivity (0.9608) and accuracy (0.913), while also showing excellent performance in positive predictive value (0.875) and negative predictive value (0.9245).In the PCPH dataset, although the model's sensitivity (0.9792) is high, indicating accurate positive class identification, its specificity (0.6111) is low, showing poor prediction accuracy for negative class samples. In summary, the model performs well on the XY-CSU and FAH-NHU datasets, especially demonstrating high accuracy in identifying negative classes; however, on the TXY-CSU and PCPH datasets, while the model performs well in identifying positive samples, there is still room for improvement in identifying negative classes.

[0159] This invention also provides a STAS detection system based on multimodal data features, such as... Figure 13 As shown, in this embodiment, the system includes:

[0160] The multimodal data preprocessing module is used to preprocess CT images and WSI, extracting CT image features and multi-scale pathological image patch features respectively, and constructing the spatial topology map of the TME; the multi-scale pathology expert module, connected to the multimodal data preprocessing module, is used to process multi-scale pathological image patch features through a dilated attention mechanism to extract multi-scale feature representations of WSI; the graph attention network module, connected to the multimodal data preprocessing module, is used to perform graph convolution processing on the spatial topology map of the TME to extract feature representations of the TME; the image attention expert module, connected to the multimodal data preprocessing module, is used to preprocess CT images and WSI through an attention mechanism. The system extracts weighted feature representations from CT images by weighting the T-image features. A cross-modal bidirectional attention fusion module, connected to the multi-scale pathology expert module, graph attention network module, and image attention expert module, performs bidirectional interactive attention calculations on the multi-scale feature representations of WSI, the feature representations of TME, and the weighted feature representations of CT images using a multi-head attention mechanism to generate fused features. A classification and optimization module, connected to the cross-modal bidirectional attention fusion module, performs STAS state classification based on the fused features and trains the entire system using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss.

[0161] The STAS detection method and system based on multimodal data features described in the above embodiments have lower accuracy compared to existing technologies. This invention, by fusing macroscopic information from CT images with microscopic information from pathological images, and combining multi-scale feature extraction and deep cross-modal interaction mechanisms, significantly improves the accuracy, robustness, and interpretability of STAS diagnosis, providing strong support for precision diagnosis and treatment of lung cancer. This invention can adapt to the differences in data distribution across different medical centers, exhibiting good universality across centers, devices, and populations, laying a solid foundation for clinical application. This invention utilizes multiple innovative modules working collaboratively. The multi-scale pathology expert module employs an expanded self-attention mechanism, effectively addressing the computational complexity issue in processing long WSI sequence data while preserving the global and local features of pathological images at different magnifications, thus aligning more closely with the actual diagnostic thinking of pathologists. The graph attention network module introduces the spatial topology of the TME into STAS diagnosis for the first time, modeling the spatial relationships and interactions between cells through graph neural networks, uncovering microenvironment features that are difficult to capture using traditional methods, and providing a new technical perspective for understanding tumor invasion behavior. The cross-modal bidirectional attention fusion module achieves bidirectional interaction between CT and pathological features through multi-head attention, enabling the information from the two modalities to mutually enhance and complement each other, generating a more information-rich fusion representation, avoiding the problem of simple superposition or loss of modal information in traditional fusion methods. The image attention expert module can output the attention weight of each CT feature. The cross-modal bidirectional attention mechanism provides attention relationships between modalities, making the model's decision-making process interpretable and enhancing clinicians' trust and willingness to adopt it. The joint optimization strategy proposed in this invention, which combines supervised cross-modal contrastive alignment loss and binary cross-entropy classification loss, uses the InfoNCE loss function to bring the feature representations of similar cross-modal samples closer together and push away dissimilar samples, thereby aligning CT features and pathological features in a common semantic space. This enhances the consistency and discriminative power of cross-modal features. The piecewise warming strategy (λ(t) gradually increases from 0 to the target value) allows the model to prioritize learning the classification boundary in the early stages of training and gradually strengthen cross-modal alignment in the later stages, avoiding convergence instability or representation collapse caused by excessive constraints in the early stages.

[0162] Obviously, the embodiments described above are merely preferred embodiments of the present invention, and not all embodiments. The accompanying drawings illustrate preferred embodiments of the present invention, but do not limit the scope of the patent. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this invention.

Claims

1. A STAS detection method based on multimodal data features, characterized in that, Includes the following steps: S10. Preprocess the CT images and pathological WSI, extract the CT image features and multi-scale pathological image block features respectively, and construct the spatial topology map of TME. S20. Process the multi-scale pathological image patch features through the dilated attention mechanism to extract the multi-scale feature representation of WSI; S30. Perform graph convolution processing on the spatial topology graph of the TME to extract the feature representation of the TME; S40. The CT image features are weighted using an attention mechanism to extract the weighted feature representation of the CT image; S50. A bidirectional interactive attention calculation is performed on the multi-scale feature representation of the WSI, the feature representation of the TME, and the weighted feature representation of the CT image through a multi-head attention mechanism to generate fused features. S60. Based on the fusion features, STAS state classification is performed, and a multimodal multiscale attention model is trained using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss. The trained model is then used for STAS detection.

2. The STAS detection method based on multimodal data features according to claim 1, characterized in that, The specific steps of S10 are as follows: S101. Use the Otsu algorithm to segment the tissue region of WSI, and use a sliding window strategy to cut it into image blocks at at least two different magnifications; S102. Perform color normalization processing on the segmented image blocks; S103. Use a pre-trained pathology-based model to extract the feature vector of each image patch to form a multi-scale image patch feature set. S104. Use a cell segmentation model to classify and segment cells in WSI, and construct the spatial topology graph of TME using the KNN nearest neighbor algorithm based on the spatial positional relationship of cells. In this graph, nodes represent cells and their related features, and edges represent the relationships between cells.

3. The STAS detection method based on multimodal data features according to claim 2, characterized in that, The specific steps of S20 are as follows: S201. Add a learnable positional bias to the input multi-scale pathological image block feature sequence to obtain a feature representation containing positional information; S202. Input the feature sequence containing location information into the stacked encoders. Each encoder is processed by an inflated self-attention mechanism. The inflated self-attention mechanism divides the long sequence into segments, performs dense attention calculation within the segments, and performs sparse attention calculation between the segments by a set inflated step size. S203. Perform global average pooling on the length dimension of the sequence output by the encoder to obtain a fixed-dimensional WSI multi-scale feature representation.

4. The STAS detection method based on multimodal data features according to claim 3, characterized in that, S20 further includes step S204: weighted fusion of the multi-scale feature representation and the feature representation of the TME through an adaptive feature fusion unit, dynamically adjusting the contribution of pathological features from different scales or sources using a learnable weight vector, and concatenating and nonlinearly mapping the weighted features to output the fused pathological feature representation.

5. The STAS detection method based on multimodal data features according to claim 1, characterized in that, The specific steps of S30 are as follows: S301. Input the spatial topology graph of the TME into the graph attention network, and use the GraphSAGE convolution operation to update the node representation by aggregating the features of the neighboring nodes of each node. S302. Perform adaptive max pooling on the updated node feature set to convert it into a fixed-length TME feature vector.

6. The STAS detection method based on multimodal data features according to claim 1, characterized in that, The specific steps of S40 are as follows: S401. Calculate attention weights for the input CT image features; S402. The CT image features are weighted and summed according to the attention weights to generate a weighted feature representation of the CT image.

7. The STAS detection method based on multimodal data features according to claim 1, characterized in that, The specific steps of S50 are as follows: S501. The multi-scale feature representation of WSI and / or the feature representation of TME are used as the first modal feature, and the weighted feature representation of CT image is used as the second modal feature. S502. Using a multi-head attention mechanism, with the first modal feature as the query and the second modal feature as the key and value, calculate the attention representation of the first modal feature on the second modal feature; S503. Using the second modal feature as the query and the first modal feature as the key and value, calculate the attention representation of the second modal feature on the first modal feature; S504. Perform residual connection and layer normalization on the attention representations in both directions to obtain the cross-modal fusion features with enhanced interaction.

8. The STAS detection method based on multimodal data features according to claim 1, characterized in that, The total loss function of the model in S60 is as follows: in, This is a supervised cross-modal contrastive alignment loss, used to bring cross-modal feature distances closer to similar samples and widen the distance between dissimilar samples; To supervise the binary cross-entropy classification loss; These are time-varying weighting coefficients used to stabilize the model in the early stages of training.

9. A detection system employing the detection method as described in any one of claims 1 to 8, characterized in that, include: The multimodal data preprocessing module is used to preprocess CT images and WSI, extract CT image features and multi-scale pathological image block features respectively, and construct the spatial topology map of TME; A multi-scale pathology expert module, connected to the multi-modal data preprocessing module, is used to process the multi-scale pathological image patch features through an dilated attention mechanism and extract the multi-scale feature representation of WSI. The graph attention network module is connected to the multimodal data preprocessing module and is used to perform graph convolution processing on the spatial topology graph of the TME to extract the feature representation of the TME. The image attention expert module, connected to the multimodal data preprocessing module, is used to weight the CT image features through an attention mechanism and extract the weighted feature representation of the CT image. A cross-modal bidirectional attention fusion module is connected to the multi-scale pathology expert module, the graph attention network module, and the image attention expert module, respectively. It is used to perform bidirectional interactive attention calculation on the multi-scale feature representation of WSI, the feature representation of TME, and the weighted feature representation of CT image through a multi-head attention mechanism to generate fused features. The classification and optimization module is connected to the cross-modal bidirectional attention fusion module. It is used to perform STAS state classification based on the fusion features and to train the entire system using a joint optimization strategy of supervised cross-modal contrastive alignment loss and classification loss.

Images