Method for predicting non-small cell lung cancer based on pd-l1 immunohistochemical staining

Abstract

The application provides a non-small cell lung cancer prediction method based on PD-L1 immunohistochemical staining, relates to the field of artificial intelligence, and comprises the following steps: S1, multi-modal original data acquisition and preprocessing; S2, intra-modal feature enhancement and structured modeling; S3, non-Euclidean space feature mapping; S4, multi-modal feature fusion and interactive modeling; and S5, immunotherapy response prediction. The method takes PD-L1 immunohistochemistry as a necessary core input, supports flexible integration of other multi-modal data, adaptively processes the differences and missing problems of modalities, accurately captures multi-dimensional information association, realizes adaptive multi-modal data fusion prediction, and improves the accuracy and clinical applicability of the prediction of the immunotherapy benefit of a non-small cell lung cancer patient.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method for predicting the benefit of multimodal immunotherapy for non-small cell lung cancer based on PD-L1 immunohistochemical staining. Background Technology

[0002] Non-small cell lung cancer (NSCLC) Non-Small Cell Lung Cancer Non-small cell lung cancer (NSCLC), a major type of lung cancer, has a persistently high incidence and mortality rate, seriously threatening human health. In recent years, immunotherapy, represented by PD-1 / PD-L1 immune checkpoint inhibitors, has become an important means of treating NSCLC. This approach has significantly prolonged the survival of some patients and improved their quality of life. However, clinical practice shows that there are significant individual differences in patients' responses to immunotherapy, and only a portion of patients can benefit from it. Blindly using this approach may not only lead to treatment ineffectiveness but also increase medical costs and the risk of adverse reactions. Therefore, how to accurately predict the likelihood of patients benefiting from immunotherapy has become a key issue that urgently needs to be addressed in clinical practice.

[0003] Currently, clinical practice mainly relies on PD-L1 immunohistochemistry (PD-L1 immunohistochemistry). Immunohistochemistry PD-L1 expression levels, used as a predictive indicator of immunotherapy efficacy, detect the expression of PD-L1 protein on the surface of tumor cells or immune cells, thus providing a reference for treatment selection. However, the predictive accuracy of a single modality is limited: on the one hand, it can only reflect some characteristics of the local immune microenvironment of the tumor and cannot comprehensively cover multi-dimensional information such as tumor heterogeneity, genomic variations, imaging features, and clinical course; on the other hand, some patients with low or negative PD-L1 expression may still benefit from immunotherapy, indicating that relying solely on this indicator is insufficient for accurate individualized prediction. To overcome the limitations of single biomarkers, existing technologies attempt to integrate multimodal data such as PD-L1 immunohistochemistry, cell-free RNA sequencing, lung CT imaging, and multi-channel immunohistochemical Western blotting (WSI) for prediction. However, existing multimodal fusion methods still have significant shortcomings: First, they often require all modal data to be complete, making it difficult to adapt to scenarios where some patients lack modal data due to testing limitations in clinical practice, severely restricting the feasibility of clinical applications. Second, multimodal data exhibits significant heterogeneity, with large differences in data distribution, feature dimensions, and semantic information among different modalities. Traditional fusion methods struggle to effectively uncover potential correlations between modalities, resulting in limited predictive performance. Summary of the Invention

[0004] In view of the problems in the prior art, the purpose of the present application is to provide a non-small cell lung cancer multi-modal immunotherapy benefit prediction method based on PD-L1 immunohistochemical staining, to solve the limitation in the prior art that all modal data must be complete to make prediction, which realizes adaptive multi-modal data fusion prediction by taking PD-L1 immunohistochemistry as the necessary core input, supporting flexible integration of other multi-modal data, adaptively processing modal differences and missing problems, and accurately capturing multi-dimensional information association, thereby improving the accuracy and clinical applicability of non-small cell lung cancer patient immunotherapy benefit prediction.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A non-small cell lung cancer prediction method based on PD-L1 immunohistochemical staining, comprising:

[0007] Step S1, multi-modal original data acquisition and preprocessing: acquiring core data of small cell lung cancer meeting the clinical standard, wherein the PD-L1 immunohistochemical pathological image is taken as the necessary core data input, and the core data is preprocessed;

[0008] Step S2, intra-modal feature enhancement and structured modeling: the features of the PD-L1 immunohistochemical pathological image are enhanced and structured, and the other input core data is enhanced and processed;

[0009] Step S3, non-Euclidean space feature mapping: first, introduce a high-dimensional mapping mechanism based on kernel function to realize the non-linear alignment of multi-modal enhanced features in a unified feature space; then, design a mapping function to convert the kernel space relationship into an explicit high-dimensional embedding; finally, calculate the distribution center of each modality in the high-dimensional space as the representative embedding of the modality;

[0010] Step S4, multi-modal feature fusion and interaction modeling: based on the correlation weight, a multi-modal interaction matrix is constructed to fuse the features of each modality to obtain a comprehensive feature vector; then, a self-attention structure is used to capture the high-order interaction relationship between modalities;

[0011] Step S5, immunotherapy response prediction: after the fusion feature vector is transformed by the attention structure, it is input into the prediction model, and a confidence evaluation module is introduced to jointly evaluate the number and quality of available modalities.

[0012] Based on the further optimization of the above scheme, the core data further includes but is not limited to gene expression data, multi-channel immunohistochemical WSI and CT images.

[0013] Based on the further optimization of the above scheme, in step S1, the preprocessing of the core data of the PD-L1 immunohistochemical pathological image comprises:

[0014] Step S11, in order to realize multi-level and quantitative characterization of pathological information of the basic module in the reserved area, the multi-channel feature extraction and fusion method is used for preprocessing: in the color channel, the color deconvolution method is used to decompose the original pixel matrix of the immunohistochemical section into a brown (DAB) channel and a blue (hematoxylin) channel: I(x,y)

[0015] ;

[0016] In the formula: I(x,y) represents the pixel intensity at the coordinate (x,y) position; I 0 represents the reference light intensity; M OD represents the optical density matrix; W represents the staining base matrix; C represents the decomposition obtained staining intensity vector;

[0017] The intensity matrix of the brown channel C is separated from the staining intensity vector C DAB , and the positive proportion of the brown channel r pos and the average staining intensity s DAB ;

[0018] ;

[0019] In the formula: N pos represents the number of positive pixels in the brown channel (by presetting a positive intensity threshold, if the staining intensity of a pixel is higher than the positive intensity threshold, the corresponding pixel is a positive pixel); N total represents the total number of pixels in the brown channel;

[0020] Step S12, at the cell structure level, the morphological features of single cells are extracted based on the nuclear segmentation model HoVer-Net, including cell area A i , circularity R i and gray level co-occurrence matrix (GLCM) statistics, , P i represents the cell perimeter;

[0021] All cells are mapped to a set of spatial coordinates , and a graph is constructed by the near neighbor relationship , and the adjacency matrix is:

[0022] ​ ;

[0023] wherein: denotes a neighborhood threshold, denotes a scale parameter;

[0024] and obtains the degree centrality based on the constructed graph D cen , clustering coefficient C i and graph entropy H(G) , ; for depicting the tissue heterogeneity;

[0025] Step S13, at the deep feature level, a pre-trained multi-modal representation learning model UNI2 is used to perform forward inference on each module to obtain a deep feature vector ;

[0026] and the hand-crafted feature vector h k and the deep feature vector f k are normalized respectively:

[0027] ;

[0028] wherein: denote the mean and standard deviation of the feature h , respectively, denote the mean and standard deviation of the feature f , respectively;

[0029] The initial feature set is formed by using a weighted splicing method:

[0030] ;

[0031] wherein: denotes vector splicing; denotes the weight coefficient of the fusion of hand-crafted and deep features; This method solves the specific problems such as strong staining difference and complex cell structure in immunohistochemical pathological images, so as to ensure that the extracted features contain not only interpretable pathological parameters, but also high-dimensional expression ability of deep models.

[0032] Based on the further optimization of the above scheme, in the step S1, the pre-processing of the core data further includes: pre-processing the immunohistochemical WSI, standardizing the gene expression data, and performing gray scale calibration and spatial normalization on the CT image;

[0033] Wherein, the pre-processing of the immunohistochemical WSI is realized by using the above step S12 and step S13;

[0034] Step S14, standardizing the gene expression data to eliminate the dimensional influence among different genes:

[0035] ;

[0036] In the formula: g k represents the original expression value of the i-th gene, k represents the average value of the i-th gene expression value, k represents the standard deviation of the i-th gene expression value; k

[0037] Step S15, gray calibration and spatial normalization of the CT image to unify the gray range and spatial scale of the image:

[0038] ;

[0039] In the formula: c m represents the original gray value of the CT image, a , b respectively represent the calibration coefficients; I(m,n,p) represents the original pixel value of the CT image, represents the pixel value after spatial normalization, represents an interpolation kernel function (linear interpolation or cubic spline interpolation can be used, and the kernel function of linear interpolation is a piecewise linear function), r x , r y , r z respectively represent the scaling factors of the corresponding coordinate axes (obtained by the ratio of the pixel spacing of the original CT image to the pixel spacing of the target spatial scale);

[0040] Based on the further optimization of the above scheme, after the "preprocessing of core data" in the step S1, the currently available data modal combination is also automatically analyzed by a modal intelligent recognition module. Specifically, the modal differentiation degree is enhanced by a multi-scale frequency domain feature extraction method, and the comparability among different modalities is ensured by combining with a normalized signature representation, so as to realize effective recognition in the case of visual similarity or uneven data distribution.

[0041] For the i-th modal data input i X i , a multi-scale feature extraction method combining spatial convolution and frequency domain change is adopted:

[0042] ; ​​​​

[0043] wherein: Conv S () denotes a convolution operation with a scale of S ; FFT() denotes a fast Fourier transform; denotes a union operation of a set;

[0044] Subsequently, the feature set is projected to a unified representation space by a mapping function:

[0045] ;

[0046] wherein: denote a weight matrix and a bias vector for performing a linear transformation on the multi-scale feature set F i ;

[0047] Finally, a predefined library of modal signatures is constructed: and the input signature Sig i is compared with the reference signatures in the library for similarity:

[0048] ;

[0049] A set of modal categories is obtained, and then the optimal modal combination is dynamically determined by a fusion requirement function: (the fusion requirement function can be any one of a weight-voting-based fusion requirement function and an optimization-objective-based fusion requirement function according to actual conditions):

[0050] ;

[0051] The above steps not only maintain high accuracy in the presence of noise or missing modalities, but also adaptively adjust the combination strategy according to the task target, thereby breaking through the limitation of fixed modal combination in traditional methods and significantly improving the practicability of the system in complex clinical environments.

[0052] Based on the further optimization of the above scheme, in the step S2, the feature enhancement and structure optimization of the PD-L1 immunohistochemical pathological image features include constructing a hybrid topological structure:

[0053] The basic unit features are taken as nodes to form a basic association graph, and the connection strength between nodes is determined according to the spatial position relationship:

[0054] ;

[0055] wherein: r ij denotes the connection strength between the i i th node and the j jThe connection strength between nodes; d ij The spatial distance of the two nodes corresponding to the basis unit is represented by d (i, j) ; k d The distance influence coefficient is represented by β;

[0056] At the same time, multiple nodes with semantic association are aggregated to form high-order units, and cross-regional association is constructed through hyper-edges, and the weight of the hyper-edge is:

[0057] ;

[0058] In the formula: m c The number of nodes contained in the hyper-edge is represented by n (i, j) ; The semantic similarity between node q 1 and node q 2 is represented by s (i, j) ;

[0059] In the formula: q The semantic similarity between node q 1 and node 2 is determined by the multi-level feature vector obtained through semantic enhancement contrast learning, and the original feature of each node is first mapped to a unified semantic space through a semantic fusion mapping function (which can adopt a mapping function based on MLP or CNN):

[0060] ;

[0061] In the formula: And respectively represent the normalized vectors after the fused semantic features;

[0062] Through the multi-scale semantic contrast loss function L SECL , the positive sample pairs are pulled closer, and the negative sample pairs are pushed farther apart, so as to improve the semantic consistency of the features in the training process, thereby enhancing the discriminability of the node semantic representation:

[0063] ;

[0064] In the formula: sim() The cosine similarity function is represented by cos (·) ; The temperature coefficient is represented by T; N The total number of contrast samples is represented by N;

[0065] On the basis of the above, in order to suppress the similarity deviation caused by noise or missing modalities, a weighted correction term is further introduced in the similarity definition:

[0066] ;

[0067] In the formula: denotes a dynamically adjusted weight coefficient (adaptively learned based on local aggregation information of nodes); denotes a smoothing parameter (used to control the sensitivity of distance measurement in high-dimensional space);

[0068] Through the above method, the node q 1 can q 2 semantic similarity S(q 1 ,q 2 ) Meanwhile, directional information and geometric distance information are fused, and a discriminative semantic vector is obtained by combining contrast learning enhancement. Compared with the prior art which only relies on a single cosine metric, the method can maintain higher robustness in a heterogeneous modal feature space, suppress the interference of noise features on similarity estimation, and enhance the cross-modal semantic alignment capability, thereby improving the reliability and generalization performance of the entire semantic graph modeling.

[0069] Finally, a graph learning algorithm is used to update the features of the mixed topology, and to strengthen the fusion expression of local and global associations.

[0070] Based on further optimization of the above scheme, the step S2 of "enhancing the core data of other inputs" includes: performing feature enhancement and structure optimization on the multi-channel immunohistochemical WSI, performing feature dimension reduction on the standardized results of gene expression data, and performing multi-scale extraction on the CT image features.

[0071] Based on further optimization of the above scheme, the "performing feature enhancement and structure optimization on the multi-channel immunohistochemical WSI" includes: first extracting the vector features of each channel to form an initial feature set , and obtaining a channel correlation matrix:

[0072] ;

[0073] and enhancing:

[0074] ;

[0075] In the formula: denotes an adjustment coefficient; through the enhancement method, complementary features are introduced while the channel independent characteristics are retained, thereby improving the discriminability of cross-channel feature expression;

[0076] Then, spatial continuity constraints are introduced to optimize the structure of the enhanced features. Assuming that the representation of the features in the spatial position x is , the constraint term is:

[0077] ;

[0078] In the formula: Indicates the neighborhood;

[0079] By minimizing the above constraints, the features are made to maintain a smooth distribution in continuous areas of the organization and remain sensitive in boundary areas, thereby improving the model's robustness to organizational structures and the accuracy of boundary area recognition.

[0080] Based on further optimization of the above scheme, step S3, "introducing a high-dimensional mapping mechanism based on kernel functions to achieve nonlinear alignment of multimodal enhancement features in a unified feature space," specifically means:

[0081] Through kernel function Enhanced features from different modalities (Where m∈{1,2,…,M} represents the mode number) Nonlinear mapping to a high-dimensional feature space :

[0082] ;

[0083] To account for the differences in statistical distribution and nonlinear characteristics of different modes, a multi-kernel combination strategy is introduced in addition to a single kernel function, employing a set of kernel functions: Including Gaussian RBF kernels and polynomial kernels:

[0084] Gaussian RBF kernel: ;

[0085] Polynomial kernel: ;

[0086] In the formula: The bandwidth parameter of RBF (controls the "width" of the Gaussian function and determines the rate at which similarity decays with distance); Q C Represents the constant term (usually) Q C ≥0), c p Indicates the degree of the polynomial (the degree controlling the nonlinear transformation);

[0087] Finally, a unified kernel function is achieved through weighted combination:

[0088] ;

[0089] In the formula: This represents the weight coefficient of the k-th kernel function. and (This can be determined adaptively through optimization based on the maximum mutual information criterion or the minimum reconstruction error criterion).

[0090] By the above manner, the nonlinear mapping and alignment of different modal characteristics in a unified high-dimensional space can be realized while maintaining the statistical characteristics, thereby enhancing the comparability and fusion discrimination performance between modes; compared with the existing single-core method, the multi-core weighting strategy can more effectively cope with the cross-modal distribution inconsistency problem, and improve the robustness and generalization ability of multi-modal fusion.

[0091] Based on further optimization of the above scheme, in the step S3, the "designing a mapping function to convert the kernel space relationship into an explicit high-dimensional embedding" is specifically:

[0092] Since the kernel matrix K H Only reflects the implicit similarity, it is not convenient for direct use by downstream models, therefore a mapping function g() is further designed to convert the kernel space relationship into an explicit high-dimensional embedding, and the initial display is realized through feature decomposition:

[0093] ;

[0094] In the formula: respectively represent the eigenvectors and eigenvalues of the kernel matrix;

[0095] Subsequently, in order to improve the trainability and downstream adaptability, a learnable nonlinear mapping is introduced:

[0096] ;

[0097] In the formula: W h , b h respectively represent the weight matrix and bias vector of the nonlinear mapping; represents a nonlinear activation function;

[0098] Finally, a unified high-dimensional embedding h(Z) is obtained; the purpose is to convert the implicit similarity of the kernel space into a structured, compact and optimizable explicit representation, thereby significantly improving the discrimination performance and generalization ability of multi-modal fusion.

[0099] Based on further optimization of the above scheme, in the step S3, the "calculating the distribution center of each mode in the high-dimensional space as the representative embedding of the mode" is specifically:

[0100] ;

[0101] In the formula: E m represents the representative embedding of the m mode, t represents the number of features of the mode; represents a space mapping function;f ms representing the first m modal feature. s

[0102] Based on further optimization of the above scheme, in the step S4, "constructing a multi-modal interaction matrix based on the correlation weight, and obtaining a comprehensive feature vector by fusing the features of each modality" includes:

[0103] First, the distribution difference between the representative embeddings of different modalities is obtained by using the distance measurement formula:

[0104] ;

[0105] Then, the distribution difference is converted into the correlation weight between modalities:

[0106] ;

[0107] After that, a multi-modal interaction matrix is constructed based on the correlation weight, and the key correlation information is strengthened through the attention mechanism, and the attention weight is:

[0108] ;

[0109] Finally, the comprehensive feature vector is obtained by fusing the features of each modality:

[0110] ;

[0111] In the formula: represents the fusion weight coefficient of the mth modality.

[0112] Based on further optimization of the above scheme, in the step S4, "using a self-attention structure to capture high-order interaction relationships between modalities" includes:

[0113] First, the representative embedding of each modality is taken as the input of the self-attention, and the representative embedding of each modality is one-to-one corresponding;

[0114] Then, the correlation between modalities is modeled through a multi-head self-attention mechanism (the multi-head self-attention mechanism can simultaneously focus on the correlation information in different subspaces):

[0115] ;

[0116] In the formula:

[0117] ;

[0118] In the formula: Q, K, V are query, key, and value matrices respectively; represents a learnable parameter matrix; d k ​denotes the dimension of each head;

[0119] The multi-head self-attention mechanism outputs the fused feature representation, and a unified fusion feature is generated through average pooling for subsequent prediction.

[0120] ;

[0121] In the formula, L represents the length of the self-attention mechanism output feature; f l denotes the feature of the i-th position. l

[0122] Based on the further optimization of the above scheme, the step S5 is specifically:

[0123] Although the step S4 has obtained the comprehensive feature vector of the fused multi-modal by constructing a multi-modal interaction matrix based on the correlation weight and fusing the features of each modality F all , but if it is directly used for prediction, the potential high-order dependency relationship between the modes is easily ignored, resulting in insufficient prediction accuracy and robustness; therefore, after obtaining the comprehensive feature vector F all , a self-attention structure is used to capture the high-order interaction relationship between the modes, and a self-attention mechanism is introduced to obtain an enhanced feature representation with global context awareness: F all , and the enhanced feature representation is input into a prediction model: Att()

[0124] ;

[0125] In the formula: W Y , b Y respectively represent the weight matrix and the bias vector of the prediction model; denotes a nonlinear activation function; Y represents the prediction probability of the immunotherapy response;

[0126] The prediction probability output by the model is Y, and the significance level of the modality subset is statistically analyzed, and the confidence score is:

[0127] ;

[0128] In the formula: p m denotes the p-value of the i-th modality under the hypothesis test m p ​​​Value (key indicator for hypothesis testing, i.e. the probability of observing the current sample result or a more extreme result under the premise that the null hypothesis is true);

[0129] Pre-set confidence score threshold T th If T T th , the prediction result is determined as high confidence; otherwise, the output result is low confidence, and a warning is given.

[0130] The technical effects of the present application are as follows:

[0131] The non-small cell lung cancer multi-modal immunotherapy benefit prediction method based on PD-L1 immunohistochemistry essential input provided by the present application can adapt to the detection capabilities of different medical institutions by taking PD-L1 immunohistochemistry data as essential input, significantly enhances the clinical applicability, so that various medical institutions can apply the method for immunotherapy benefit prediction according to their actual situation, without being unable to carry out related prediction work due to detection condition restrictions, and has wide application range and strong practicality. At the same time, with the increase of available modalities (such as gene sequencing data, lung CT images and channel immunohistochemistry WSI, etc.), the prediction accuracy of the method can be gradually improved, realizing the increment of prediction precision, and providing more accurate prediction results for the clinic under different data conditions. In addition, medical institutions can selectively increase detection items according to their resource allocation and actual needs, so that the resource allocation is more flexible, avoiding the problems of resource waste or insufficient allocation caused by fixed detection item requirements.

[0132] The method only takes PD-L1 immunohistochemistry as essential input, allows medical institutions to choose whether to increase other detection items according to actual situation, thereby avoiding unnecessary complete detection, optimizing cost-effectiveness, realizing personalized detection strategy, reducing detection cost while ensuring prediction effect. Moreover, the method introduces a confidence evaluation module, which can provide prediction reliability according to the number and quality of available modalities, providing optimal prediction and reliable confidence evaluation support based on existing data for clinical decision-making, which helps clinicians make more scientific and reasonable treatment decisions. In summary, the method of the present application has significant advantages in clinical applicability, prediction accuracy, resource allocation, cost-effectiveness and clinical decision support, etc., can effectively meet the needs of different medical institutions, improve the accuracy and reliability of non-small cell lung cancer immunotherapy benefit prediction, and has important clinical application value. BRIEF DESCRIPTION OF DRAWINGS

[0133] Figure 1 It is a schematic diagram for modal intra-feature enhancement and structured modeling in the embodiments of the present application. ​

[0134] Figure 2 This is a cross-modal feature space mapping and fusion interaction diagram in an embodiment of the present invention.

[0135] Figure 3 This is a flowchart of high-order interaction modeling and prediction in an embodiment of the present invention. Detailed Implementation

[0136] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0137] Example 1:

[0138] A method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining includes:

[0139] Step S1: Multimodal raw data acquisition and preprocessing: Collect core data of small cell lung cancer that meets clinical standards. Among them, PD-L1 immunohistochemical pathological images are required core data inputs. At the same time, selective inclusion of other modal data such as gene expression data, multi-channel immunohistochemical WSI, and CT images is supported, but not limited to gene expression data, multi-channel immunohistochemical WSI, and CT images.

[0140] The core data is preprocessed, specifically as follows:

[0141] Preprocessing of core data from PD-L1 immunohistochemical pathological images:

[0142] Step S11: To achieve multi-level and quantitative characterization of the pathological information of the basic modules within the preserved area, preprocessing is performed using a multi-pathway feature extraction and fusion method: In the color channel, a color deconvolution method is used to transform the original pixel matrix of the immunohistochemical slice. I(x,y) Decomposed into brown (DAB) channels and blue (hematoxylin) channels:

[0143] ;

[0144] In the formula: I(x,y) Representing coordinates (x,y) Pixel intensity at the location; I 0 represents the reference light intensity; M OD Represents the optical density matrix; W Represents the coloring basis matrix; C This represents the staining intensity vector obtained from the decomposition;

[0145] From the color intensity vector C The intensity matrix of the brown channel separated from the image. C DABand the positive proportion of the brown channel is obtained quantitatively r pos and the average staining intensity s DAB ;

[0146] ;

[0147] wherein: N pos represents the number of positive pixels in the brown channel (a pixel is a positive pixel if the staining intensity of the pixel is higher than a preset positive intensity threshold); N total represents the total number of pixels in the brown channel;

[0148] Step S12, at the level of cell structure, morphological features of individual cells are extracted based on the nuclear segmentation model HoVer-Net, including cell area A i , circularity R i and gray level co-occurrence matrix (GLCM) statistics, , P i represents the cell perimeter;

[0149] All cells are mapped to a set of spatial coordinates , a graph is constructed through a near-neighbor relationship , and the adjacency matrix thereof is:

[0150] ;

[0151] wherein: represents a neighborhood threshold, represents a scale parameter;

[0152] and degree centrality D cen , clustering coefficient C i and graph entropy H(G) , are obtained based on the constructed graph; used to characterize tissue heterogeneity;

[0153] Step S13, at the level of deep features, a pre-trained multi-modal representation learning model UNI2 is used to perform forward inference on each module to obtain a deep feature vector ;

[0154] and the hand-crafted feature vector h k and the deep feature vector f kNormalization is performed:

[0155]

[0156] In the formula: respectively represent the mean and standard deviation of the characteristics h respectively represent the mean and standard deviation of the characteristics f

[0157] An initial feature set is formed by weighted splicing:

[0158]

[0159] In the formula: represents vector splicing; represents the weight coefficient of the fusion of manual and deep features. This method solves the specific problems of strong staining difference and complex cell structure in immunohistochemical pathological images, thereby ensuring that the extracted features contain interpretable pathological parameters and have high-dimensional expression ability of deep models.

[0160] If the immunohistochemical WSI data is included, the immunohistochemical WSI is preprocessed by the above steps S12 and S13.

[0161] If the gene expression data is included, step S14 is performed; in step S14, the gene expression data is standardized to eliminate the dimensional influence between different genes:

[0162]

[0163] In the formula: g k represents the original expression value of the i-th gene, k represents the average value of the i-th gene expression value, represents the standard deviation of the i-th gene expression value. k k

[0164] If the CT image is included, step S15 is performed; in step S15, the CT image is subjected to gray calibration and spatial normalization to unify the gray range and spatial scale of the image:

[0165]

[0166] In the formula: c m represents the original gray value of the CT image, a , b respectively represent the calibration coefficients; I(m,n,p) represents the original pixel value of the CT image, ​​​​​​​​​denotes the spatially normalized pixel value, denotes the interpolation kernel function (linear interpolation or cubic spline interpolation can be used, and the kernel function of linear interpolation is a piecewise linear function), r x , r y , r z denote the scaling factors of the corresponding coordinate axes (obtained by the ratio of the pixel spacing of the original CT image to the pixel spacing of the target space scale).

[0167] After the "preprocessing of core data", the current available data modal combination is also automatically analyzed by the modal intelligent recognition module, specifically: through the multi-scale frequency domain consistent feature extraction method to enhance the modal distinction, and then combined with the normalized signature representation to ensure the comparability between different modalities, so as to realize effective identification in the case of visual similarity or uneven data distribution;

[0168] For the input of the first i modal data X i , a multi-scale feature extraction method combining spatial convolution and frequency domain change is adopted:

[0169] ;

[0170] In the formula: Conv S () denotes the convolution operation with a scale of S ; FFT() denotes the fast Fourier transform; denotes the set union operation;

[0171] Subsequently, the feature set is projected to a unified representation space through a mapping function:

[0172] ;

[0173] In the formula: denote the weight matrix and bias vector for linear transformation of the multi-scale feature set F i ;

[0174] Finally, a pre-defined modal signature library is constructed: , and the input signature Sig i is compared with the reference signature in the library for similarity calculation:

[0175] ;

[0176] The modal category set Then, by merging the demand functions (The fusion requirement function can be either a weighted voting-based fusion requirement function or an optimization objective-based fusion requirement function, depending on the actual situation.) Dynamically determine the optimal mode combination:

[0177] ;

[0178] The above steps not only maintain high accuracy in the presence of noise or missing modalities, but also adaptively adjust the combination strategy according to the task objective, thereby breaking through the limitation of fixed modal combinations in traditional methods and significantly improving the practicality of the system in complex clinical environments.

[0179] Step S2, Intramodal Feature Enhancement and Structured Modeling: Feature enhancement and structure optimization are performed on PD-L1 immunohistochemical pathological image features, including the construction of hybrid topology structures.

[0180] A basic association graph is formed using the basic unit features as nodes, and the connection strength between nodes is determined based on their spatial relationships:

[0181] ;

[0182] In the formula: r ij Indicates the first i The node and the first j The connection strength between nodes; d ij This represents the spatial distance between the basic units corresponding to the two nodes; k d Indicates the distance influence coefficient;

[0183] Simultaneously, multiple nodes with semantic relationships are aggregated to form higher-order units, and cross-regional connections are constructed through hyperedges, with the hyperedge weights being:

[0184] ;

[0185] In the formula: m c Indicates the number of nodes contained in the superedge; Represents a node q 1 and node q Semantic similarity of 2;

[0186] Among them, nodes q 1 and node q The semantic similarity of 2 is determined by the multi-level feature vector obtained through semantically enhanced contrastive learning. First, the original features of each node are processed. semantic fusion mapping function (A mapping function based on MLP or CNN can be used) Projected onto a unified semantic space:

[0187] ;

[0188] In the formula: These represent the normalized vectors after fusing semantic features;

[0189] Using a multi-scale semantic contrast loss function L SECL By bringing positive sample pairs closer together and pushing negative sample pairs further apart, the semantic consistency of features is improved during training, thereby enhancing the discriminative power of node semantic representations.

[0190] ;

[0191] In the formula: sim() Represents the cosine similarity function; Indicates the temperature coefficient; N Indicates the total number of samples compared;

[0192] Building upon the above, to suppress similarity shifts caused by noise or missing modes, a weighted correction term is further introduced into the similarity definition:

[0193] ;

[0194] In the formula: This indicates that the weight coefficients are dynamically adjusted (obtained through adaptive learning based on local node aggregation information). This represents the smoothing parameter (used to control the sensitivity of distance metrics in high-dimensional spaces).

[0195] Using the above method, the node q 1 and node q Semantic similarity of 2 S(q 1 ,q 2 ) Simultaneously, it integrates directional and geometric distance information and combines contrastive learning to enhance the discriminative semantic vector. Compared with existing methods that rely solely on a single cosine metric, this method maintains higher robustness in heterogeneous modal feature spaces, suppresses the interference of noise features on similarity estimation, and enhances cross-modal semantic alignment capabilities, thereby improving the reliability and generalization performance of the entire semantic graph modeling.

[0196] Finally, a graph learning algorithm is used to update the features of the hybrid topology, enhancing the fusion representation of local and global relationships:

[0197] ;

[0198] In the formula: Indicates the updated number i The characteristics of each node; N(i) Represents a node ia set of associated nodes of the first node; h j representing initial features of the first node; j b i representing a corresponding bias term.

[0199] The core data of other inputs are enhanced, including feature enhancement and structure optimization for multi-channel immunohistochemical WSI, feature dimension reduction on the standardized results of gene expression data, and multi-scale extraction of CT image features.

[0200] If a certain modality does not exist, the corresponding step is skipped; the system updates the feature space adaptively according to the available modalities.

[0201] Feature enhancement and structure optimization for multi-channel immunohistochemical WSI include: first, extract the vector features of each channel to form an initial feature set , and obtain the channel correlation matrix:

[0202] ;

[0203] and enhance:

[0204] ;

[0205] In the formula: represents an adjustment coefficient; through this enhancement method, complementary features are introduced while the independent characteristics of the channels are retained, improving the discriminability of cross-channel feature expression;

[0206] After that, spatial continuity constraints are introduced to optimize the structure of the enhanced features. Assuming that the representation of the feature at the spatial position x is , the constraint term is:

[0207] ;

[0208] In the formula: represents the neighborhood;

[0209] By minimizing the above constraint, the feature maintains a smooth distribution in the continuous region of the tissue and maintains sensitivity in the boundary region, thereby improving the robustness of the model to the structure of the tissue and the recognition accuracy of the boundary region.

[0210] For gene expression data, if exists, the standardized results of this modality are reduced in dimension to retain the core gene features, and if does not exist, this step of the modality is skipped. The dimension reduction process satisfies:

[0211] ;

[0212] ​In the formula: denotes the normalized gene expression matrix; T zh denotes the dimensionality reduction conversion matrix; G r denotes the core gene feature matrix after dimensionality reduction.

[0213] For the CT image, if exists, the multi-scale extraction is performed on the modality feature, and multi-level features are obtained through different scale feature extraction windows to form an enhanced feature vector of the CT image; if does not exist, this step of the modality is skipped.

[0214] Step S3, non-Euclidean space feature mapping: first, a high-dimensional mapping mechanism based on a kernel function is introduced to realize the nonlinear alignment of multi-modality enhanced features in a unified feature space; specifically:

[0215] Through the kernel function , the enhanced features from different modalities (wherein, m∈{1,2,…,M}, represents the modality number) are nonlinearly mapped to a high-dimensional feature space :

[0216] ;

[0217] In order to take into account the statistical distribution difference and nonlinear feature mode of different modalities, in addition to a single kernel function, a multi-kernel combination strategy is introduced, and a kernel function set is adopted: , including a Gaussian RBF kernel and a polynomial kernel:

[0218] Gaussian RBF kernel: ;

[0219] Polynomial kernel: ;

[0220] In the formula: denotes the bandwidth parameter of RBF (controls the "width" of the Gaussian function, determines the decay speed of similarity with distance); Q C denotes a constant term (usually Q C ≥0), c p denotes the polynomial degree (controls the range of nonlinear transformation);

[0221] Finally, a unified kernel function is obtained through weighted combination:

[0222] ;

[0223] In the formula: denotes the weight coefficient of the kth kernel function (0 and , which can be adaptively determined by optimization based on the maximum mutual information criterion or the minimum reconstruction error criterion);

[0224] In this way, the nonlinear mapping and alignment of different modal characteristics in a unified high-dimensional space can be realized while maintaining the statistical characteristics of different modal characteristics, thereby enhancing the comparability and fusion discrimination performance between modalities. Compared with existing single-core methods, the multi-core weighting strategy can more effectively cope with the cross-modal distribution inconsistency problem and improve the robustness and generalization ability of multi-modal fusion.

[0225] Then, a mapping function is designed to convert the kernel space relationship into an explicit high-dimensional embedding, specifically:

[0226] Since the kernel matrix K H only reflects the implicit similarity and is not directly convenient for downstream models, a mapping function g() is further designed to convert the kernel space relationship into an explicit high-dimensional embedding, and the initial display is realized through eigenvalue decomposition:

[0227] ;

[0228] In the formula: respectively represent the eigenvectors and eigenvalues of the kernel matrix;

[0229] Subsequently, to improve the trainability and downstream adaptability, a learnable nonlinear mapping is introduced:

[0230] ;

[0231] In the formula: W h , b h respectively represent the weight matrix and bias vector of the nonlinear mapping; represents the nonlinear activation function;

[0232] Finally, a unified high-dimensional embedding h(Z) is obtained; the purpose is to convert the implicit similarity of the kernel space into a structured, compact, and optimizable explicit representation, thereby significantly improving the discrimination performance and generalization ability of multi-modal fusion.

[0233] Finally, the distribution center of each modality in the high-dimensional space is calculated as the representative embedding of the modality, specifically:

[0234] ;

[0235] In the formula: E m represents the representative embedding of the m modality, ta number of features of the modality; a spatial mapping function; f ms a first m a first s feature of the modality.

[0236] The above process is a cross-modal feature space mapping and multi-modal feature fusion and interaction modeling process, which is based on the currently available modal features. First, the necessary PD-L1 immunohistochemical pathology image enhancement features are mapped to a unified high-dimensional feature space, and for other existing modal features, they are also mapped to this space through a kernel function; if a modality is missing, only the existing modal features are processed. The distribution center of each existing modality in the high-dimensional space is calculated as a representative embedding, and no related calculation is performed for the missing modality.

[0237] Step S4, multi-modal feature fusion and interaction modeling: based on the correlation weight, a multi-modal interaction matrix is constructed, and the comprehensive feature vector is obtained by fusing the features of each modality; including:

[0238] First, the distance measurement formula is used to obtain the distribution difference between the representative embeddings of different modalities:

[0239] ;

[0240] Then, the distribution difference is converted into the correlation weight between modalities:

[0241] ;

[0242] After that, based on the correlation weight, a multi-modal interaction matrix is constructed, and the key correlation information is strengthened through the attention mechanism, and the attention weight is:

[0243] ;

[0244] Finally, the features of each modality are fused to obtain a comprehensive feature vector:

[0245] ;

[0246] In the formula: m represents the fusion weight coefficient of the mth modality.

[0247] In the fusion stage, if there is only a PD-L1 immunohistochemical single modality, its features are directly used as the fusion basis; if there are multiple modalities, the key correlation information between modalities is strengthened through the attention mechanism to realize feature fusion. This processing method makes the fusion process able to dynamically adjust according to the actual existing modalities, without relying on complete modality data.

[0248] After that, the self-attention structure is used to capture the high-order interaction relationship between modalities; including:

[0249] First, the representative embedding of each modality is taken as the input of self-attention (at least including the PD-L1 immunohistochemistry modality), and the representative embedding of each modality corresponds one by one;

[0250] Then, the correlation between the modalities is modeled by a multi-head self-attention mechanism (the multi-head self-attention mechanism can simultaneously focus on the associated information in different subspaces):

[0251] ;

[0252] Among them:

[0253] ;

[0254] In the formula: Q, K, V are query, key, value matrices respectively; represents a learnable parameter matrix; d k represents the dimension of each head;

[0255] The multi-head self-attention mechanism outputs the fused feature representation, and a unified fused feature is generated by average pooling for subsequent prediction. The feature after average pooling is:

[0256] ;

[0257] In the formula: L represents the length of the self-attention mechanism output feature; f l represents the feature of the l th position.

[0258] If there is only a PD-L1 immunohistochemistry single modality, the self-attention mechanism degenerates into single-modality internal feature mining and does not model across modalities. The self-attention mechanism outputs the fused feature representation, and a unified fused feature is generated and input to the prediction model.

[0259] Step S5, immunotherapy response prediction: After the fusion feature vector is transformed by the attention structure, it is input to the prediction model, and a confidence evaluation module is introduced to jointly evaluate the number and quality of available modalities. Specifically:

[0260] Although the comprehensive feature vector of the fused multi-modality has been obtained in step S4 "based on the correlation weight to construct a multi-modality interaction matrix, and fuse the features of each modality to obtain a comprehensive feature vector" F all , but if it is directly used for prediction, the potential high-order dependency between modalities is easily ignored, resulting in insufficient prediction accuracy and robustness; therefore, after obtaining the comprehensive feature vector F allAfter that, the self-attention mechanism is introduced to capture the high-order interaction relationship between the modalities, and the comprehensive feature vector F all Through attention transformation Att() Dynamic modeling of conditional dependence between modalities, obtaining enhanced feature representation with global context awareness, and inputting into the prediction model:

[0261] ;

[0262] In the formula: W Y , b Y respectively represent the weight matrix and bias vector of the prediction model; , represents a nonlinear activation function; Y represents the prediction probability of immunotherapy response;

[0263] The prediction probability output by the model is Y, and the significance level of the modal subset is statistically analyzed, and the confidence score is:

[0264] ;

[0265] In the formula: p m represents the m value obtained by the p th modality under the hypothesis test (the key index for hypothesis testing, that is, the probability of observing the current sample result or more extreme result under the premise that the null hypothesis is true);

[0266] The confidence score threshold is pre-set T th If T > T th , the prediction result is determined as high confidence; otherwise, the output result is low confidence and a warning is given.

[0267] For missing modalities, the adaptive non-Euclidean space fusion module is used to construct a dynamic feature relationship graph with the PD-L1 immunohistochemical feature as the center node, to realize intelligent compensation of missing modalities and ensure the robustness of prediction.

[0268] Embodiment 2:

[0269] As another preferred embodiment of the present application, on the basis of embodiment 1, the fusion demand function in step S16 is fused, and if the fusion demand function is based on the weight voting fusion demand function, the contribution degree of each modality is measured by the weight , and the function expression is:

[0270] ;

[0271] wherein: denotes an indicator function, which is 1 if the condition is satisfied, and 0 otherwise; the modal combination with the highest votes is selected by weighted voting for each candidate class c;

[0272] The fusion requirement function If the fusion requirement function is based on the optimization objective, the target is to "maximize the inter-modality complementarity", and the function expression is:

[0273] ;

[0274] wherein: denotes the task accuracy of the modal subset (used to measure the performance of the modal subset in completing the target task), denotes the information redundancy in the modal subset (the linear correlation between two modal features can be measured by the Pearson correlation coefficient); denotes the trade-off coefficient;

[0275] The optimal modal combination is selected by optimizing the target.

Claims

1. A method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining, characterized in that: include: Step S1: Multimodal raw data acquisition and preprocessing: Acquire core data of small cell lung cancer that meets clinical standards. Among them, PD-L1 immunohistochemical pathological images are required core data inputs, and the core data are preprocessed. The core data also includes gene expression data, multi-channel immunohistochemical WSI, and CT images. Step S2, Intramodal Feature Enhancement and Structured Modeling: Feature enhancement and structure optimization are performed on PD-L1 immunohistochemical pathological image features, including the construction of hybrid topology structures. A basic association graph is formed using the basic unit features as nodes, and the connection strength between nodes is determined based on their spatial relationships: ； In the formula: r ij Indicates the first i The node and the first j The connection strength between nodes; d ij This represents the spatial distance between the basic units corresponding to the two nodes; k d Indicates the distance influence coefficient; Simultaneously, multiple nodes with semantic relationships are aggregated to form higher-order units, and cross-regional connections are constructed through hyperedges, with the hyperedge weights being: ； In the formula: m c Indicates the number of nodes contained in the superedge; Represents a node q 1 and node q Semantic similarity of 2; Among them, nodes q 1 and node q The semantic similarity of 2 is determined by the multi-level feature vector obtained through semantically enhanced contrastive learning. First, the original features of each node are processed. semantic fusion mapping function Projected onto a unified semantic space: ； In the formula: These represent the normalized vectors after fusing semantic features; Using a multi-scale semantic contrast loss function L SECL By bringing positive sample pairs closer together and pushing negative sample pairs further apart, the semantic consistency of features is improved during training, thereby enhancing the discriminative power of node semantic representations. ； In the formula: sim() Represents the cosine similarity function; Indicates the temperature coefficient; N Indicates the total number of samples compared; Building upon the above, to suppress similarity shifts caused by noise or missing modes, a weighted correction term is further introduced into the similarity definition: ； In the formula: This indicates that the weighting coefficients are dynamically adjusted. Indicates the smoothing parameter; Finally, a graph learning algorithm is used to update the features of the hybrid topology, enhancing the fusion representation of local and global relationships; It also enhances the core data of other inputs; Step S3, Non-Euclidean Space Feature Mapping: First, a high-dimensional mapping mechanism based on kernel functions is introduced to achieve non-linear alignment of multimodal enhancement features in a unified feature space; then, a mapping function is designed to transform the kernel space relationship into an explicit high-dimensional embedding; finally, the distribution center of each modality in the high-dimensional space is calculated as the representative embedding of the modality. Step S4, Multimodal Feature Fusion and Interaction Modeling: Construct a multimodal interaction matrix based on association weights, fuse features from various modalities to obtain a comprehensive feature vector; then, use a self-attention structure to capture higher-order interaction relationships between modalities; Step S5, Immunotherapy Response Prediction: After the fused feature vector undergoes attention structure transformation, it is input into the prediction model, and a confidence assessment module is introduced to jointly evaluate the quantity and quality of available modalities.

2. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 1, characterized in that: In step S1, the preprocessing of the core data of the PD-L1 immunohistochemical pathological image includes: Step S11: To achieve multi-level and quantitative characterization of the pathological information of the basic modules within the preserved area, preprocessing is performed using a multi-pathway feature extraction and fusion method: In the color channel, a color deconvolution method is used to transform the original pixel matrix of the immunohistochemical slice. I(x,y) Decomposed into brown and blue channels: ； In the formula: I(x,y) Representing coordinates (x,y) Pixel intensity at the location; I 0 represents the reference light intensity; M OD Represents the optical density matrix; W Represents the coloring basis matrix; C This represents the staining intensity vector obtained from the decomposition; From the color intensity vector C The intensity matrix of the brown channel separated from the image. C DAB And quantitatively obtain the positive proportion of the brown channel. r pos and average staining intensity s DAB ; ； In the formula: N pos Indicates the number of positive pixels in the brown channel; N total This indicates the total number of pixels in the brown channel; Step S12: At the cellular structure level, extract morphological features of individual cells based on the HoVer-Net nuclear segmentation model, including cell area. A i Roundness R i And the gray-level co-occurrence matrix statistics, , P i Indicates cell perimeter; All cells are mapped to a set of spatial coordinates. Graph constructed through nearest neighbor relationships Its adjacency matrix is: ； In the formula: Indicates the neighborhood threshold. Indicates the scale parameter; And obtain the degree centrality based on the constructed graph. D cen Clustering coefficient C i and graph entropy H(G) , Used to characterize tissue heterogeneity; Step S13: At the deep feature level, a pre-trained multimodal representation learning model UNI2 is used for each module. Perform forward inference to obtain deep feature vectors ; And respectively for the handmade feature vectors h k With deep feature vectors f k Normalize: ； In the formula: Representing features respectively h The mean and standard deviation, Representing features respectively f The mean and standard deviation; The initial feature set is formed by weighted concatenation: ； In the formula: This indicates vector concatenation; This represents the weighting coefficients for the fusion of manual and deep features.

3. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 2, characterized in that: After step S1 "preprocessing the core data", the modality intelligent recognition module automatically analyzes the currently available data modality combinations. Specifically, it enhances modality discrimination by extracting features in the multi-scale frequency domain and combines normalized signature representation to ensure comparability between different modalities, thereby achieving effective recognition in cases of visual similarity or uneven data distribution. For the input of the first i Modal data X i A multi-scale feature extraction method combining spatial convolution and frequency domain transformation is employed. ； In the formula: Conv S () The scale is represented as S Convolution operations; FFT() Indicates Fast Fourier Transform; The union operation represents the set operation; Subsequently, the feature set is projected onto a unified representation space using a mapping function: ； In the formula: Representing the multi-scale feature sets respectively F i The weight matrix and bias vector for linear transformation; Finally, a predefined modal signature library is constructed: And will enter a signature Sig i Calculate similarity with the baseline signature in the database: ； Obtain the set of modal categories Then, by merging the demand functions Dynamically determine the optimal mode combination: 。 4. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 3, characterized in that: In step S3, "introducing a high-dimensional mapping mechanism based on kernel functions to achieve nonlinear alignment of multimodal enhanced features in a unified feature space" specifically means: Through kernel function Enhanced features from different modalities Nonlinear mapping to high-dimensional feature space : ； To account for the differences in statistical distribution and nonlinear characteristics of different modes, a multi-kernel combination strategy is introduced in addition to a single kernel function, employing a set of kernel functions: Including Gaussian RBF kernels and polynomial kernels: Gaussian RBF kernel: ; Polynomial kernel: ; In the formula: Indicates the bandwidth parameter of RBF; Q C Represents a constant term. c p Indicates the degree of the polynomial; Finally, a unified kernel function is achieved through weighted combination: ； In the formula: This represents the weight coefficient of the k-th kernel function.

5. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 4, characterized in that: In step S3, "designing a mapping function to transform the kernel space relation into an explicit high-dimensional embedding" specifically means: Due to the kernel matrix K H Reflecting only implicit similarity is not convenient for direct use in downstream models, therefore a mapping function needs to be further designed. g() The kernel space relation is transformed into an explicit high-dimensional embedding, and the initial display is achieved through eigenvalue decomposition: ； In the formula: These represent the eigenvectors and eigenvalues ​​of the kernel matrix, respectively. Subsequently, to improve trainability and downstream adaptability, a learnable nonlinear mapping is introduced: ； In the formula: W h , b h These represent the weight matrix and bias vector of the nonlinear mapping, respectively. Represents a nonlinear activation function; Ultimately, a unified high-dimensional embedding is achieved. h(Z) ; That The aim is to transform the implicit similarity of the kernel space into a structured, compact, and optimizable explicit representation, thereby significantly improving the discriminative performance and generalization ability of multimodal fusion.

6. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 5, characterized in that: In step S3, "calculating the distribution center of each mode in the high-dimensional space as a representative embedding of the mode" specifically means: ； In the formula: E m Indicates the first m Representative embedding of modalities t This indicates the number of features in this mode; Represents a space mapping function; f ms Indicates the first m The first modality s One characteristic.

7. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 6, characterized in that: In step S4, "constructing a multimodal interaction matrix based on association weights and fusing features from each modality to obtain a comprehensive feature vector" includes: First, the distributional dissimilarity among representative embeddings of different modalities is obtained using a distance metric formula: ； Then, the distribution dissimilarity is converted into correlation weights between modes: ； Next, a multimodal interaction matrix is ​​constructed based on the association weights, and key association information is reinforced through an attention mechanism. The attention weights are: ； Finally, the features from each modality are fused to obtain a comprehensive feature vector: ； In the formula: This represents the fusion weight coefficient for the m-th mode.

8. The method for predicting non-small cell lung cancer based on PD-L1 immunohistochemical staining according to claim 7, characterized in that: In step S4, "capturing higher-order interaction relationships between modalities using a self-attention structure" includes: First, the representative embeddings of each modality are used as the input to self-attention, with each modality's representative embedding corresponding to another one-to-one. Then, the correlation between modalities is modeled using a multi-head self-attention mechanism: ； in: ； In the formula: Q, K, and V are the query, key, and value matrices, respectively; Represents the learnable parameter matrix; d k Indicates the dimension of each head; The multi-head self-attention mechanism outputs a fused feature representation, which is then processed by average pooling to generate a unified fused feature for subsequent prediction. The feature after average pooling is as follows: ； In the formula: L represents the length of the output feature of the self-attention mechanism; f l Indicates the first l Features of each location.

Images