Bladder cancer survival analysis method and system based on digitized pathological sections

Abstract

The application discloses a bladder cancer survival analysis method based on digital pathological sections and relates to the technical field of survival prediction. Genomic data and clinical data are processed into static variable vectors with equal sizes. The static variable vectors are learned, and static variable selection features are output. The static variable selection features are learned by using a first gated residual network, and static variable features are output. Pathological section feature sets are processed, and pathological section first features are obtained. The static variable selection features are learned by using a second gated residual network, and static variable intermediate features are output. The third gated residual network is used to learn the pathological section first features and the static variable intermediate features, and pathological section second features are obtained. The static variable features and the pathological section second features are fused, and fused features are obtained. The fused features are learned, and a value reflecting survival risk is output. The application aims to provide an accurate survival analysis model.

Description

Technical Field

[0001] This invention relates to the field of survival prediction technology, and in particular to a method and system for bladder cancer survival analysis based on digital pathological slides. Background Technology

[0002] Bladder cancer, a common malignant tumor of the urinary system, causes widespread health problems worldwide. Despite significant advances in medicine over the past few decades, the diagnosis and treatment of bladder cancer still face many challenges. Among these, accurate assessment of survival analysis and prognosis for bladder cancer patients is particularly crucial.

[0003] Existing methods for analyzing bladder cancer survival mainly rely on limited clinical data, such as the patient's age, gender, and tumor stage. This single-feature analysis is difficult to comprehensively reflect the patient's condition and survival prognosis, and ignores other important factors that may affect survival.

[0004] With the continuous development of molecular biology and imaging technologies, more and more transcriptomic and radiomic information about bladder cancer patients is becoming available, providing new opportunities for in-depth exploration of the pathogenesis and prognostic factors of bladder cancer. Some survival analysis methods rely on clinical data and transcriptomic information, but neglect the potential impact of radiomic information on survival prognosis. However, morphological features contained in medical imaging and histological images are equally important in survival analysis; yet existing methods often fail to integrate this information effectively.

[0005] Existing technologies often fail to adequately account for individual differences among patients. Each patient's survival is influenced by a variety of factors, including clinical characteristics and molecular-level differences, and current methods often fail to fully consider these differences, resulting in poor generalization of predictive outcomes. Summary of the Invention

[0006] To overcome the shortcomings of the existing technology, this invention provides a bladder cancer survival analysis method based on digital pathological slides. The aim is to provide an accurate survival prediction model to help physicians better assess the survival risk of bladder cancer patients and to provide a basis for personalized treatment strategies. This method integrates features from multiple aspects, including clinical data, transcriptomics information (genomic data), and radiomics information (pathological slide feature sets), to construct a comprehensive survival analysis model for more accurate survival prediction.

[0007] To achieve the above objectives, the present invention adopts the following technical solution, including:

[0008] A method for bladder cancer survival analysis based on digital pathological sections, comprising the following steps:

[0009] S1, through the data embedding module, processes genomic data and clinical data into one-dimensional vectors of equal size, which serve as static variable vectors;

[0010] S2, using the Static Variable Extractor (SVS), learn the static variable vector obtained in step S1 and output static variable selection features;

[0011] S3, using the first gated residual network GRN1, learn the static variable selection features output in step S2 and output the static variable features;

[0012] S4. Using the Multi-Instance Learning (MIL) module, the feature set of the pathological slides is processed to obtain the first feature of the pathological slides.

[0013] S5. Using the second gated residual network GRN2, the static variable selection features output in step S2 are learned, and the intermediate features of the static variables are output.

[0014] S6. Using the third gated residual network GRN3, the second feature of the pathological slice is obtained by learning based on the first feature of the pathological slice and the intermediate features of the static variables.

[0015] S7. Using the feature fusion module, the static variable features output in step S3 and the second feature of the pathological slide output in step S6 are fused to obtain the fused features.

[0016] S8 uses a Long Short-Term Memory (LSTM) network to learn the fused features and outputs a value that reflects the survival risk.

[0017] Preferably, step S4 is implemented in the following manner:

[0018] S41, the patient's pathological slide is divided into N patches. The ResNet-50 model is used to extract features from each patch and convert them into 1×1024 vectors. Each pathological slide can be converted into an N×1024 vector matrix, i.e., the pathological slide feature set. The pathological slide feature set is used as the input of the multi-instance learning module MIL, with the pathological slide as the bag and the patches obtained from the pathological slide segmentation as instances.

[0019] S42, in the Multi-Instance Learning Module (MIL), the first layer is a fully connected layer W. FC1 Dimension W FC1 ∈R 512×1024 This transforms each patch feature from a 1024-dimensional vector into a 512-dimensional vector:

[0020]

[0021] Among them, s m ∈R1×1024 m=1,…,N,s m h is the feature vector of the m-th patch of the pathological section; m ∈R 1×512 This is the output of the first fully connected layer; the superscript T indicates matrix transpose.

[0022] S43, followed by two attention network layers W FC2-1 and W FC2-2 The dimensions are W. FC2-1 ∈R 256×512 and W FC2-2 ∈R 256×512 Each of the layers passing through the first fully connected layer W is respectively... FC1 The processed output vector h m From 512 dimensions to 256 dimensions:

[0023]

[0024]

[0025] Where, q m,1 ∈R 1×256 For the first attention network layer W FC2-1 The output, q m,2 ∈R 1×256 For the second attention network layer W FC2-2 The output;

[0026] S44 utilizes a fully connected layer W FC-a Transform the patch features into a scalar, namely the patch attention feature g. m Fully connected layer W FC-a ∈R 1×256 Then, the patch attention feature g is used. m Calculate the attention score for each instance, i.e., each tile:

[0027]

[0028]

[0029] Where tanh and sigmoid are activation functions; ⊙ is the Hadamard product, which is the one-to-one multiplication of two matrices; exp is the exponentiation operation with base e; g m The attention feature for the m-th patch; To calculate the sum of attention features for all patches; N is the number of patches segmented from the pathological slide; score m ∈R 1×1 Let m be the attention score for the m-th patch;

[0030] S45, using the attention score m With the first layer, i.e., the fully connected layer W FC1 The processed output vector h m To perform attention pooling, obtain new features h. slide :

[0031]

[0032] Among them, h slide ∈R 1×512 This is the output value of the Multi-Instance Learning Module (MIL), which is the first feature of the pathological slide.

[0033] Preferably, in step S1, the clinical data includes the patient's gender, age, TNM stage of the patient's pathological section, and treatment method; the genomic data includes oncogenes, tumor suppressor factors, and subtyping indicators.

[0034] Preferably, in step S1, the genomic data and clinical data include categorical data and numerical data. The data embedding module uses a word embedding model for categorical data. First, the categorical data is preprocessed, and each type of categorical data uses a word embedding to generate a corresponding word embedding vector. Then, the mapping network is used to convert the word embedding vectors corresponding to the numerical data and categorical data into n one-dimensional vectors of equal size, i.e., static variable vectors.

[0035] Preferably, in step S2, the input to the static variable extractor SVS is S = {s1, ..., s...} n},s i ∈R m This indicates that there are n static variable vectors input into the static variable extractor, each with dimension m. The static variable extractor (SVS) includes multiple gated residual networks (GRNs), and each GRN is used to compute the values ​​of each input s. i The eigenvalues ​​are calculated, and the contribution of the input to the output value is calculated using a gated residual network (GRN); the specific method is as follows:

[0036] S21, the input n static variable vectors are processed by gated residual operations, and the corresponding eigenvalues ​​are output:

[0037] gOut i =GRN i (s i );

[0038] Among them, GRN i For the GRN feature converter corresponding to the i-th static variable vector, s i Let gOut be the i-th static variable vector. iThe output result of the gated residual operation on the i-th static variable vector;

[0039] S22, summing the outputs of the gated residual operation on n static variable vectors:

[0040] gOut=concat([gOutn1,…,gOut n ]);

[0041] Wherein, gOut is the superposition feature of n static variable vectors after gated residual operation;

[0042] S23, input the n static variable vectors into the gated residual network (GRN) for gated residual operation and softmax operation, and finally generate the coefficient values ​​a of the n static variable vectors:

[0043] a=softmax([aGRN1(s1),…,aGRN1(s n )]);

[0044] S23, multiply the coefficient value a and the superimposed feature gOut after the gated residual operation to generate a new feature Out:

[0045] Out = a·gOut;

[0046] Among them, the new feature Out is the static variable selection feature.

[0047] Preferably, the gated residual network (GRN) is obtained by combining gated linear units and residual network structures; the input vector g of the gated residual network GRN is a one-dimensional vector, g∈R. m m is the number of features of the vector; the extra input vector e of the gated residual network GRN is a one-dimensional vector, e∈R m The additional input vector e of the gated residual network GRN is optional.

[0048] The calculation process of the gated residual network GRN is as follows:

[0049] S31, the input vector g and the additional input vector e, after passing through the first Dense layer and the ELU activation function, output...

[0050]

[0051] in, This refers to the first Dense layer that the input vector g passes through. The first Dense layer through which the additional input vector e passes. d represents the parameters of the model;

[0052] S32, Features output from step S31 After the second Dense layer W D2 Then, output

[0053]

[0054] Among them, W D2 ∈R d ;

[0055] S33, the second Dense layer W D2 Output characteristics Passing through two GLU networks respectively and After the nonlinear transformation, the output

[0056]

[0057] in, σ is the Sigmoid activation function;

[0058] S34, features optimized by the GLU network. Add the input vector g, and then normalize it using the feature layer. The output is the output of the GRN network.

[0059]

[0060] Here, LayerNorm represents feature layer normalization.

[0061] Preferably, neither the first gated residual network GRN1 nor the second gated residual network GRN2 has an additional input vector e, and the third gated residual network GRN3 uses the intermediate features of the static variables output in step S5, i.e., the output of the second gated residual network GRN2, as the additional input vector e.

[0062] Preferably, in step S7, the feature fusion module is used to fuse the static variable selection features output in step S3 and the second feature of the pathological slide output in step S5 to obtain the fused features. The specific process is as follows:

[0063] Out F =concat([F1,F2])

[0064] Where F1 is the output of the first gated residual network GRN1, i.e., the static variable feature; F2 is the output of the third gated residual network GRN3, i.e., the second feature of the pathological section; Out F These are the features after fusion.

[0065] This invention also provides a bladder cancer survival analysis system based on digital pathological slides, applicable to the aforementioned bladder cancer survival analysis method based on digital pathological slides. The system model includes:

[0066] The data embedding module is used to process genomic and clinical data into vectors of equal size as static variable vectors, and then transmit the static variable vectors to the static variable extractor SVS.

[0067] The Static Variable Extractor (SVS) is used to learn the static variable vector, output static variable selection features, and transmit the static variable selection features to the first gated residual network (GRN1) and the second gated residual network (GRN2), respectively.

[0068] The first gated residual network GRN1 is used to learn the selected features of static variables, output the static variable features, and transmit the static variable features to the feature fusion module.

[0069] The Multi-Instance Learning Module (MIL) is used to process the feature set of pathological slides to obtain the first feature of the pathological slides, and then transmit the first feature of the pathological slides to the third gated residual network (GRN3).

[0070] The second gated residual network GRN2 is used to learn the selected features of the static variables, output the intermediate features of the static variables, and transmit the intermediate features of the static variables to the third gated residual network GRN3.

[0071] The third gated residual network GRN3 is used to learn the first feature of the pathological section and the intermediate features of the static variable to obtain the second feature of the pathological section.

[0072] The feature fusion module is used to fuse static variable features and secondary features of pathological sections to obtain fused features.

[0073] The Long Short-Term Memory (LSTM) module is used to learn the fused features and output a value that reflects the survival risk.

[0074] The advantages of this invention are:

[0075] (1) This invention proposes an innovative method for bladder cancer survival analysis, aiming to fully utilize multi-faceted characteristic information to more accurately predict patient survival. This method not only integrates clinical data but also combines genomic and radiomics information, thus providing comprehensive data support at both the molecular and morphological levels.

[0076] (2) This invention effectively integrates features from multiple aspects such as clinical data, transcriptomics information and radiomics information to establish a more comprehensive and integrated survival analysis model, thereby improving the accuracy and reliability of survival prediction and providing doctors and patients with more reliable survival prediction results.

[0077] (3) The introduction of genomic information in this invention can delve deeper into the pathogenesis of bladder cancer and identify molecular biological features closely related to survival prognosis. This helps to better understand the pathophysiological process of bladder cancer and provides more valuable information for the formulation of treatment strategies.

[0078] (4) By introducing machine learning and artificial intelligence technologies, this invention automatically extracts key features from digital pathological slides, effectively utilizes radiomics information such as medical images and histological images, and mines the morphological features contained therein, providing more clinically significant data support for survival analysis. Attached Figure Description

[0079] Figure 1 This is a flowchart of the method for bladder cancer survival analysis based on digital pathological sections according to the present invention.

[0080] Figure 2 This is a system architecture diagram of the bladder cancer survival analysis based on digital pathological slides according to the present invention.

[0081] Figure 3 Survival curves plotted for models pre-trained based on genomic data, clinical data, and the ImageNet dataset.

[0082] Figure 4 Survival curves plotted for models pre-trained based on genomic data, clinical data, and the NCT-CRC-HE dataset.

[0083] Figure 5 Survival curves plotted for models pre-trained based on genomic data, clinical data, and the TCGA-BLCA dataset. Figure 6 This diagram illustrates the relative influence of each static variable on the prediction results. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0085] Depend on Figure 2As shown, the bladder cancer survival analysis system based on digital pathological slides includes the following system model:

[0086] The data embedding module is used to process genomic and clinical data into vectors of equal size as static variable vectors, and then transmit the static variable vectors to the static variable extractor SVS.

[0087] The Static Variable Extractor (SVS) is used to learn the static variable vector, output static variable selection features, and transmit the static variable selection features to the first gated residual network (GRN1) and the second gated residual network (GRN2), respectively.

[0088] The first gated residual network GRN1 is used to learn the selected features of static variables, output the static variable features, and transmit the static variable features to the feature fusion module.

[0089] The Multi-Instance Learning Module (MIL) is used to process the feature set of pathological slides to obtain the first feature of the pathological slides, and then transmit the first feature of the pathological slides to the third gated residual network (GRN3).

[0090] The second gated residual network GRN2 is used to learn the selected features of the static variables, output the intermediate features of the static variables, and transmit the intermediate features of the static variables to the third gated residual network GRN3.

[0091] The third gated residual network GRN3 is used to learn the first feature of the pathological section and the intermediate features of the static variable to obtain the second feature of the pathological section.

[0092] The feature fusion module is used to fuse static variable features and secondary features of pathological sections to obtain fused features.

[0093] The Long Short-Term Memory (LSTM) module is used to learn the fused features and output a value that reflects the survival risk.

[0094] Depend on Figure 1 As shown, a method for bladder cancer survival analysis based on digital pathological sections is presented, which includes the following steps:

[0095] S1, through the data embedding module, processes genomic data and clinical data into one-dimensional vectors of equal size, which serve as static variable vectors;

[0096] S2, using the Static Variable Extractor (SVS), learn the static variable vector obtained in step S1 and output static variable selection features;

[0097] Among them, the static variable selection features output by the static variable extractor SVS are used as common features. On the one hand, they are used for subsequent feature extraction of static variables, that is, for feature extraction of the first gated residual network GRN1. On the other hand, they are used for fusion learning with the first feature of the pathological slide output by the multi-instance learning module MIL.

[0098] S3, using the first gated residual network GRN1, learn the static variable selection features output in step S2 and output the static variable features;

[0099] S4. Using the Multi-Instance Learning (MIL) module, the feature set of the pathological slides is processed to obtain the first feature of the pathological slides.

[0100] S5. Using the second gated residual network GRN2, the static variable selection features output in step S2 are learned, and the intermediate features of the static variables are output.

[0101] S6. Using the third gated residual network GRN3, the second feature of the pathological slice is obtained by learning based on the first feature of the pathological slice and the intermediate features of the static variables.

[0102] S7. Using the feature fusion module, the static variable features output in step S3 and the second feature of the pathological slide output in step S6 are fused to obtain the fused features.

[0103] S8 uses a Long Short-Term Memory (LSTM) network to learn the fused features and outputs a value that reflects the survival risk.

[0104] In this invention, clinical data includes the patient's gender, age, diagnosis time, TNM stage (TNM is a staging system for tumors in oncology), TNM stage tumor spread, life status, tumor location, and treatment methods (whether drug therapy or radiotherapy was administered). Transcriptomics data includes oncogenes, tumor suppressor factors, and subtyping indicators. Radiomics data, i.e., the pathological slide feature set, includes N feature vectors derived from N pathological slide segmentation patches. Specifically, the patient's pathological slide is segmented into N patches, and the ResNet-50 model is used to extract features from each patch, converting them into 1×1024 vectors. Each pathological slide can be converted into an N×1024 vector matrix, thus forming the pathological slide feature set. In this embodiment, the ResNet-50 model is used to extract features from the pathological slide segmentation patches. The fully connected layers and the last residual layer of the pre-trained ResNet-50 model need to be removed, transforming each patch from a 224×224 image into a 1×1024 vector.

[0105] Data embedding module

[0106] Genomic and clinical data include both categorical and numerical data. The data embedding module uses a word embedding model for categorical data. First, the categorical data is preprocessed, with each categorical type using its own word embedding, generating corresponding word embedding vectors. Then, a mapping network is used to convert the word embedding vectors corresponding to the numerical and categorical data into n equal-sized one-dimensional vectors, i.e., static variable vectors. In the clinical data, TNM staging results, patient life status, whether drug treatment was administered, whether radiotherapy was administered, patient gender, TNM stage tumor spread, and tumor location information are categorical data, while patient diagnosis time is numerical data. All genomic data is numerical data.

[0107] Static variable extractor

[0108] The input to the static variable extractor SVS is S = {s1, ..., s2}. n},s i ∈R m This indicates that there are n static variable vectors input into the static variable extractor, each with dimension m. The static variable extractor (SVS) includes multiple gated residual networks (GRNs), and each GRN is used to compute the values ​​of each input s. i The eigenvalues ​​are determined, and the contribution of the input to the output value is calculated using a gated residual network (GRN). The specific method is as follows:

[0109] S21, the input n static variable vectors are respectively processed by n gated residual networks (GRNs) to perform gated residual operations, and the corresponding feature values ​​are output:

[0110] gOut i =GRN i (s i );

[0111] Among them, GRN i For the GRN feature converter corresponding to the i-th static variable vector, s i Let gOut be the i-th static variable vector. i The output result of the gated residual operation on the i-th static variable vector;

[0112] S22, summing the outputs of the gated residual operation on n static variable vectors:

[0113] gOut = concat([gOut1, ..., gOut) n ]);

[0114] Wherein, gOut is the superposition feature of n static variable vectors after gated residual operation;

[0115] S23, input the n static variable vectors into the gated residual network (GRN) for gated residual operation and softmax operation, and finally generate the coefficient values ​​a of the n static variable vectors:

[0116] a=softmax([aGRN1(s1),…,aGRN1(s n ,e n )]);

[0117] S23, multiply the coefficient value a and the superimposed feature gOut after the gated residual operation to generate a new feature Out:

[0118] Out = a·gOut;

[0119] Among them, the new feature Out is the static variable selection feature.

[0120] Gated Residual Network (GRN)

[0121] Gated residual networks (GRNs) are obtained by combining gated linear units (GRNs) with residual network structures. The input vector g of a GRN is a one-dimensional vector, g∈R. m m is the number of features of the vector; the extra input vector e of the gated residual network GRN is a one-dimensional vector, e∈R m The additional input vector e of the gated residual network GRN is optional. The calculation process of the gated residual network GRN is as follows:

[0122] S31, the input vector g and the additional input vector e, after passing through the first Dense layer and the ELU activation function, output...

[0123]

[0124] in, This refers to the first Dense layer that the input vector g passes through. The first Dense layer through which the additional input vector e passes. d is a parameter of the model. d is adjustable. In this embodiment, d is set to 64.

[0125] S32, Features output from step S31 After the second Dense layer W D2 Then, output

[0126]

[0127] Among them, W D2 ∈R d ;

[0128] S33, the second Dense layer W D2 Output feature D out2 They passed through two GLU networks respectively. and After the nonlinear transformation, the output

[0129]

[0130] in, σ is the Sigmoid activation function;

[0131] S34, features optimized by the GLU network. Add the input vector g, and then normalize it using the feature layer. The output is the output of the GRN network.

[0132]

[0133] Here, LayerNorm represents feature layer normalization.

[0134] In this invention, the gated residual network (GRN) in steps S2, S3, S5, and S6 is processed in the manner described above. Specifically, the gated residual network (GRN), the first gated residual network (GRN1), and the second gated residual network (GRN2) in the static variable extractor (SVS) have no additional input, while the third gated residual network (GRN3) has additional input, which is the intermediate features of the static variable.

[0135] Multi-Instance Learning Module (MIL)

[0136] The Multi-Instance Learning (MIL) module is used to process the feature set of pathological slides to obtain the first feature of the pathological slides. The specific method is as follows:

[0137] S41, the patient's pathological slide is divided into N patches. The ResNet-50 model is used to extract features from each patch and convert them into 1×1024 vectors. Each pathological slide can be converted into an N×1024 vector matrix, i.e., the pathological slide feature set. The pathological slide feature set is used as the input of the multi-instance learning module MIL, with the pathological slide as the bag and the patches obtained from the pathological slide segmentation as instances.

[0138] S42, in the Multi-Instance Learning Module (MIL), the first layer is a fully connected layer W. FC1 Dimension W FC1 ∈R 512×1024 This transforms each patch feature from a 1024-dimensional vector into a 512-dimensional vector:

[0139]

[0140] Among them, s m ∈R 1×1024 m=1,…,N,s m h is the feature vector of the m-th patch of the pathological section; m ∈R 1×512 This is the output of the first fully connected layer; the superscript T indicates matrix transpose.

[0141] S43, followed by two attention network layers W FC2-1 and W FC2-2 The dimensions are W. FC2-1 ∈R 256×512 and W FC2-2 ∈R 256×512 Each of the layers passing through the first fully connected layer W is respectively... FC1 The processed output vector h m From 512 dimensions to 256 dimensions:

[0142]

[0143]

[0144] Where, q m,1 ∈R 1×256 For the first attention network layer W FC2-1 The output, q m,2 ∈R 1×256 For the second attention network layer W FC2-2 The output;

[0145] S44 utilizes a fully connected layer W FC-a Transform the patch features into a scalar, namely the patch attention feature g. m Fully connected layer W FC-a ∈R 1×256 Then, the patch attention feature g is used. m Calculate the attention score for each instance, i.e., each tile:

[0146]

[0147]

[0148] Where tanh and sigmoid are activation functions; ⊙ is the Hadamard product, which is the one-to-one multiplication of two matrices; exp is the exponentiation operation with base e; g m The attention feature for the m-th patch; To calculate the sum of attention features for all patches; N is the number of patches segmented from the pathological slide; score m ∈R 1×1 Let m be the attention score for the m-th patch;

[0149] S45, using the attention score m With the first layer, i.e., the fully connected layer W FC1 The processed output vector h m To perform attention pooling, obtain new features h. slide :

[0150]

[0151] Among them, h slide ∈R 1×512 This is the output value of the Multi-Instance Learning Module (MIL), which is the first feature of the pathological slide.

[0152] Feature fusion module

[0153] Using the feature fusion module, the static variable selection feature output in step S3 and the second feature of the pathological slide output in step S5 are fused to obtain the fused feature. The specific process is as follows:

[0154] Out F =concat([F1,F2])

[0155] Where F1 is the output of the first gated residual network GRN1, i.e., the static variable feature; F2 is the output of the third gated residual network GRN3, i.e., the second feature of the pathological section; Out F These are the features after fusion.

[0156] Long Short-Term Memory Network (LSTM)

[0157] The Long Short-Term Memory (LSTM) network is used to learn the fused features and output a value reflecting survival risk. The LSTM includes a forget gate, an input gate, and an output gate. The input gate determines how much of the current time step's input is retained in the current time step's cell state; the forget gate determines how much of the previous time step's cell state is retained in the current time step's cell state; and the output gate determines how much of the current time step's cell state is output, as detailed below:

[0158] Input gates are used to control the input of new information, including forget gates that determine which old information to delete and which new information to add.

[0159] I t =σ(W i ·[h t-1 ,x t ]+b i )

[0160] Among them, h t-1 It is the hidden state at the current time step, xt W represents the fused features at the current time step. i Let b be the weight matrix of the input gate. i It is the bias of the input gate, σ is the sigmoid activation function, and I... t It is the output of the input gate, I t The range is from 0 to 1;

[0161] The forget gate is used to control which information in memory cells from the previous time step should be forgotten:

[0162] F t =σ(W f ·[h t-1 ,x t ]+b f )

[0163] Among them, W f This is the weight matrix of the forget gate, b f It is the bias of the forgetting gate, F t It is the output of the forget gate, F t The range is from 0 to 1;

[0164] Memory cells (Cell State) t It stores information about the current time step:

[0165] C t =F t ·C t-1 +I t ·tanh(W c ·[h t-1 ,x t ]+b c )

[0166] Among them, W c It is the weight matrix of memory cells, b c It is the bias of memory cells, C t-1 It is the memory cell from the previous time step, C t It is the memory cell at the current time step, and tanh is the hyperbolic tangent activation function;

[0167] The output gate determines which information is passed from the memory cell to the hidden state, becoming the output of the current time step.

[0168] O t =σ(W o ·[h t-1 ,x t ]+b o )

[0169] Among them, Wo It is the weight matrix of the output gate, b o It is the bias of the output gate, O t It is the output of the output gate, O t The range is from 0 to 1;

[0170] Hidden State t This is the final output, i.e., the final output of the LSTM, and can also be used as the input for the next time step:

[0171] h t =O t ·tanh(C t )

[0172] Among them, h t This is the final output of the LSTM;

[0173] The output layer (Dense) is specifically as follows:

[0174] R score =W s LSTM out +b s

[0175] Among them, W s It is the weight matrix of the output layer, b s It is the bias of the output layer.

[0176] In this invention, the model input for the bladder cancer survival analysis system includes the patient's pathological slide feature set, clinical data (TNM staging results, TNM stage tumor spread, whether drug treatment was administered, whether radiotherapy was administered, patient diagnosis time, tumor location information, etc.), and genomic data (19 numerical data points). The model predicts a survival risk value, using survival time for prediction; the higher the survival risk value, the shorter the corresponding survival time. In the sample data, the values ​​reflecting survival risk are the patient's overall survival (OS) and the patient's life / death status.

[0177] Example 2

[0178] In the experiment analyzing the survival of bladder cancer patients, this paper used the ImageNet dataset, the human colorectal cancer and healthy tissue image dataset, and the bladder cancer image dataset to pre-train a ResNet-50 model to extract features from digital pathological sections of bladder cancer. The ImageNet dataset, the human colorectal cancer and healthy tissue image dataset, and the bladder cancer image dataset were named ImageNet, NCT-CRC-HE, and TCGA-BLCA, respectively. 400 bladder cancer patients with TNM stages II–IV were randomly divided into training and test sets, and 10-fold cross-validation was performed. The mean, standard deviation, and maximum C-index (consistency index) of the three datasets are shown in Table 1.

[0179] Table 1. Experimental results for the three datasets.

[0180]

[0181] As can be seen from Table 1, the ResNet-50 model, i.e. the feature extraction model, pre-trained using the TCGA-BLCA bladder cancer image dataset is the best. During training, the results on all three datasets exceeded 0.70, while during testing, they all exceeded 0.60. The best result of the model during testing even reached 0.69.

[0182] The survival analysis model takes into account radiomics information (pathological slide feature set), clinical data (TNM staging results, TNM stage tumor spread, whether drug therapy was administered, whether radiotherapy was administered, patient diagnosis time, tumor location information, etc.), and genomics data (19 numerical data points). The survival analysis model predicts a survival risk value, using survival time for prediction; a higher survival risk value corresponds to a shorter survival time. In the sample data, the values ​​reflecting survival risk are the patient's overall survival (OS) and the patient's life-or-death status.

[0183] Survival analysis provides a clear overview of the entire process of an event, identifying key factors influencing survival time by constructing multiple survival curves. This study utilizes a proposed multi-feature fusion-based bladder cancer survival analysis model to predict survival risk values, categorizing patients into low-risk and high-risk groups, and plotting survival curves based on genomic data, clinical data, and radiomics information (ImageNet, NCT-CRC-HE, or TCGA-BLCA datasets). Figure 3-5 As shown.

[0184] Our method extracts the relative influence of each static variable on the prediction outcome from the model. Using a trained model, the influencing factors (i.e., static variables, composed of clinical and genomic data) for each patient are calculated, and the relative influence value of each factor is output during the calculation process, such as... Figure 6 As shown.

[0185] The top ten factors with the highest percentage of influence were selected as key influencing factors. In the entire sample, key influencing factors included stage, ARID1A, KDM6A, PTEN, CDKN2A, E2F3, KMT2D, ATD (Age at Diagnosis), KRT7, and stage_M; among them, stage, ATD, and stage_M represent clinical data; ARID1A, KDM6A, PTEN, CDKN2A, and KMT2D are tumor suppressor genes; and E2F3 and KRT7 are proto-oncogenes.

[0186] This invention, by comprehensively considering features from multiple aspects such as clinical data, genomic data, and radiomics information, can establish a more comprehensive and accurate survival analysis model. Such a model can more accurately predict the survival status of bladder cancer patients, providing doctors and patients with more reliable survival prediction results. Introducing genomic information allows for deeper exploration of the pathogenesis of bladder cancer, identifying molecular biological features closely related to survival prognosis. This contributes to a better understanding of the pathophysiological processes of bladder cancer, providing more valuable information for the formulation of treatment strategies.

[0187] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for bladder cancer survival analysis based on digital pathological sections, characterized in that, The method includes the following steps: S1, through the data embedding module, processes genomic data and clinical data into one-dimensional vectors of equal size, which serve as static variable vectors; S2, using the Static Variable Extractor (SVS), learn the static variable vector obtained in step S1 and output static variable selection features; S3, using the first gated residual network GRN1, learn the static variable selection features output in step S2 and output the static variable features; S4. Using the Multi-Instance Learning (MIL) module, the feature set of the pathological slides is processed to obtain the first feature of the pathological slides. S5. Using the second gated residual network GRN2, the static variable selection features output in step S2 are learned, and the intermediate features of the static variables are output. S6. Using the third gated residual network GRN3, the second feature of the pathological slice is obtained by learning based on the first feature of the pathological slice and the intermediate features of the static variables. S7. Using the feature fusion module, the static variable features output in step S3 and the second feature of the pathological slide output in step S6 are fused to obtain the fused features. S8 utilizes a Long Short-Term Memory (LSTM) network to learn the fused features and outputs a value reflecting survival risk. The gated residual network (GRN) is obtained by combining gated linear units and residual network structures; the input vector of the gated residual network (GRN) A one-dimensional vector , The number of features of the vector; the additional input vector of the gated residual network GRN. It is a one-dimensional vector. Additional input vectors for gated residual networks (GRN) Optional input; The calculation process of the gated residual network GRN is as follows: S31, Input Vector and additional input vectors After passing through the first Dense layer and the ELU activation function, the output is... : ； in, For input vectors The first Dense layer passed through, For additional input vectors The first Dense layer passed through, These are the parameters of the model; S32, Features output from step S31 After the second Dense layer Then, output : ； in, ; S33, the second Dense layer Output characteristics They passed through two GLU networks respectively. and After the nonlinear transformation, the output : ； in, , Use the Sigmoid activation function; S34, features optimized by the GLU network. With input vector Add them together, then normalize the feature layers; the output is the output of the GRN network. ； in, Indicates feature layer normalization; Neither the first gated residual network GRN1 nor the second gated residual network GRN2 has any additional input vectors. The third gated residual network GRN3 uses the intermediate features of the static variables output from step S5, i.e., the output of the second gated residual network GRN2, as an additional input vector. .

2. The method for bladder cancer survival analysis based on digital pathological sections according to claim 1, characterized in that, The specific method for step S4 is as follows: S41, the patient's pathological slide is divided into N patches. The ResNet-50 model is used to extract features from each patch and convert them into 1×1024 vectors. Each pathological slide can be converted into an N×1024 vector matrix, i.e., the pathological slide feature set. The pathological slide feature set is used as the input of the multi-instance learning module MIL, with the pathological slide as the bag and the patches obtained from the pathological slide segmentation as instances. S42, in the Multi-Instance Learning Module (MIL), the first layer is a fully connected layer. , dimension This transforms each patch feature from a 1024-dimensional vector into a 512-dimensional vector: ； in, , The first pathological section Feature vectors of each patch; This is the output of the first fully connected layer; superscript T Indicates matrix transpose; S43, followed by two attention network layers. and The dimensions are respectively and Each of the layers passing through the first layer, i.e., the fully connected layer, will be... The processed output vector From 512 dimensions to 256 dimensions: ； ； in, For the first attention network layer The output, For the second attention network layer The output; S44 utilizes a fully connected layer Transform the patch features into a scalar, namely the patch attention feature. Fully connected layer Then, the attention features of the map are utilized. Calculate the attention score for each instance, i.e., each tile: ； ； in, sigmoid is the activation function; The Hadamard product is the one-to-one multiplication of two matrices; exp is the exponentiation operation with base e. For the first m Attention features of each image patch; To calculate the sum of attention features for all tiles; The number of patches for dividing a pathological section; For the first m Attention score for each tile; S45, using attention scores With the first layer, i.e., the fully connected layer The processed output vector To perform attention pooling and obtain new features : ； in, This is the output value of the Multi-Instance Learning Module (MIL), which is the first feature of the pathological slide.

3. The method for bladder cancer survival analysis based on digital pathological sections according to claim 1, characterized in that, In step S1, clinical data includes the patient's gender, age, TNM stage of the patient's pathological section, and treatment method; genomic data includes oncogenes, tumor suppressor factors, and subtyping indicators.

4. The method for bladder cancer survival analysis based on digital pathological sections according to claim 1, characterized in that, In step S1, the genomic data and clinical data include categorical data and numerical data. The data embedding module uses a word embedding model for categorical data. First, the categorical data is preprocessed, and each type of categorical data uses a word embedding to generate the corresponding word embedding vector. Then, the mapping network is used to convert the word embedding vectors corresponding to the numerical data and categorical data into n one-dimensional vectors of equal size, i.e., static variable vectors.

5. The method for bladder cancer survival analysis based on digital pathological sections according to claim 1, characterized in that, In step S2, the input to the static variable extractor SVS is: , indicating that there is n A vector of static variables is input into the static variable extractor, and the dimension of each static variable vector is... m ; The Static Variable Extractor (SVS) comprises multiple gated residual networks (GRNs), each of which is used to compute the values ​​for each input. The eigenvalues ​​are calculated, and the contribution of the input to the output value is calculated using a gated residual network (GRN); the specific method is as follows: S21, Input n Each static variable vector is processed by gated residual operations, and the corresponding eigenvalues ​​are output: ； in, For the GRN feature converter corresponding to the i-th static variable vector, Let i be the vector of static variables. The output result of the gated residual operation on the i-th static variable vector; S22, for n The outputs of the gated residual operation on the vectors of static variables are superimposed: ； in, for n The superposition characteristics of a static variable vector after gated residual operation; S23, will n A vector of static variables is input into a gated residual network (GRN) for gated residual computation and softmax operation, finally generating coefficient values ​​for n static variable vectors. : ； S23, coefficient value Superposition features after gated residual calculation Multiplication generates new features. : ； Among them, new features This is the static variable selection feature.

6. The method for bladder cancer survival analysis based on digital pathological sections according to claim 1, characterized in that, In step S7, the feature fusion module is used to fuse the static variable selection features output in step S3 and the second feature of the pathological slide output in step S5 to obtain the fused features. The specific process is as follows: in, The output of the first gated residual network GRN1 is the static variable feature; The output of the third gated residual network GRN3 is the second feature of the pathological section; These are the features after fusion.

7. A bladder cancer survival analysis system based on digital pathological slides, characterized in that, The system model applicable to the bladder cancer survival analysis method based on digital pathological slides according to any one of claims 1-6 includes: The data embedding module is used to process genomic and clinical data into vectors of equal size as static variable vectors, and then transmit the static variable vectors to the static variable extractor SVS. The Static Variable Extractor (SVS) is used to learn the static variable vector, output static variable selection features, and transmit the static variable selection features to the first gated residual network (GRN1) and the second gated residual network (GRN2), respectively. The first gated residual network GRN1 is used to learn the selected features of static variables, output the static variable features, and transmit the static variable features to the feature fusion module. The Multi-Instance Learning Module (MIL) is used to process the feature set of pathological slides to obtain the first feature of the pathological slides, and then transmit the first feature of the pathological slides to the third gated residual network (GRN3). The second gated residual network GRN2 is used to learn the selected features of the static variables, output the intermediate features of the static variables, and transmit the intermediate features of the static variables to the third gated residual network GRN3. The third gated residual network GRN3 is used to learn the first feature of the pathological section and the intermediate features of the static variable to obtain the second feature of the pathological section. The feature fusion module is used to fuse static variable features and secondary features of pathological sections to obtain fused features. The Long Short-Term Memory (LSTM) module is used to learn the fused features and output a value that reflects the survival risk.

Images