Multi-mode hepatocellular carcinoma prognosis prediction method based on femtosecond laser pathological image

Abstract

The invention discloses a multi-mode hepatocellular carcinoma prognosis prediction method based on femtosecond laser pathological images. The method comprises the following steps: a) data acquisition and preprocessing; b) FLI feature extraction; according to the method, space and channel feature decoupling and denoising are achieved through multi-scale channel selection, channel self-attention and spatial filtering, FLI features, pathological report features and clinical data features are deeply fused through multi-modal residual feature fusion, and the accuracy and robustness of prognosis prediction are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and medical image processing, and in particular to a multimodal hepatocellular carcinoma prognosis prediction method based on femtosecond laser pathological images. This method combines femtosecond laser pathological images (FLI), pathology reports and clinical data, and achieves spatial and channel feature decoupling and noise reduction through multi-scale channel selection, channel self-attention and spatial filtering, and performs deep information fusion using a multimodal residual feature fusion process. Background Technology

[0002] Hepatocellular carcinoma (HCC) is the sixth most common cancer worldwide and the second leading cause of cancer-related death. Despite significant progress in the early diagnosis and treatment of HCC, the overall mortality rate remains high in most countries and regions, with a five-year survival rate of less than 20%. This is primarily due to the high heterogeneity and aggressiveness of HCC, its tendency to relapse and metastasize, leading to significant differences in clinical treatment outcomes. Therefore, postoperative prognostic prediction has become a crucial basis for physicians to develop individualized treatment plans and follow-up management. Accurate prognostic prediction not only helps assess a patient's survival and recurrence risk but also provides a reliable reference for the selection of subsequent surgical methods, the development of adjuvant therapy plans, and resource allocation.

[0003] Currently, with the development of deep learning technology, it is increasingly being applied to the diagnosis of HCC. However, due to the high heterogeneity of HCC, traditional imaging techniques (CT, MRI) and color pathological images (RGB) often fail to comprehensively represent the deeper tissue information and pathological features of the tumor. To address this limitation, femtosecond laser imaging technology has been applied to medical image analysis. Femtosecond laser pathological images utilize the response of femtosecond lasers at different structural levels in tissues to form multi-channel imaging patterns, thereby representing deeper tissue information. Existing research on the application of femtosecond laser pathological images in medical tasks includes a study (Research on Breast Tumor Neural Visualization and Prognostic Prediction Based on Multiphoton Microscopy, 2023.06.01) which compares and analyzes femtosecond laser pathological images with hematoxylin and eosin (H&E) stained images to visualize the nerves in the breast tumor microenvironment and constructs a prognostic prediction model based on neural pattern scoring. However, this approach has two limitations. First, the research simply compares femtosecond laser pathology images with ordinary HE-stained images, without utilizing deep learning technology. This requires researchers to possess relevant professional knowledge and expend significant manual effort. Second, if the image is a composite of multiple channels from a femtosecond laser pathology image, the multi-channel information often contains redundancy and noise, making it more difficult to accurately locate lesions with the naked eye. If it is a single-channel image from a femtosecond laser pathology image, the channel most relevant to tumor identification must be manually selected, resulting in the loss of information from other channels. This makes it difficult to dynamically select the optimal channel for different tasks, leading to reduced accuracy. The literature (Research on Liver Fibrosis Grading Based on Machine Learning and Multiphoton Imaging, 2023.04.17) inputs manually extracted features, deep learning features, and clinical information features from femtosecond laser pathology images into an MLP for fusion, ultimately outputting the grading results of liver fibrosis. This approach has the following limitations: the fusion method used is simple, lacking a deep cross-modal interaction mechanism, and easily losing complementary information between modalities.

[0004] Therefore, there is an urgent need for a method in the prognosis of HCC that can reduce the input of human resources and achieve automated assisted diagnosis; can also achieve automatic dynamic selection of key channels in multi-channel femtosecond laser pathological images to reduce redundancy and noise; and can make full use of complementary information between modes to achieve efficient fusion between multi-modal data, thereby solving the limitations of the above-mentioned research and improving the accuracy and stability of HCC prognosis prediction. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in existing technologies by proposing a multimodal HCC prognosis prediction method based on femtosecond laser pathological images (Multimodal HCC Prognosis Framework integrating FLI, MHPF-FLI). This method achieves decoupling and denoising of spatial and channel features through multi-scale channel selection with channel self-attention and spatial filtering (MCSCSASF). Then, it deeply fuses FLI features, pathology report features, and clinical data features through multimodal residual feature fusion (MRFFM), effectively improving the accuracy and robustness of prognostic prediction.

[0006] The specific technical solution for implementing this invention is as follows:

[0007] A multimodal prognostic prediction method for hepatocellular carcinoma based on femtosecond laser pathological images, the prediction method comprising the following steps: data acquisition and preprocessing, FLI feature extraction, multimodal data fusion, and prognostic prediction.

[0008] The data acquisition and preprocessing process refers to acquiring 5-channel FLI (femtosecond) images and preprocessing them to obtain sliced ​​femtosecond images, while simultaneously acquiring corresponding pathology reports and clinical data.

[0009] The 5-channel femtosecond image in the invention has 5 channels, each receiving a different signal band, thus displaying different tissue components. For example, the third harmonic generation (THG) channel receives signals in the 340-380nm band, characterizing cell membranes, nuclear membranes, etc.; the three-photon fluorescence (3PEF) channel receives signals in the 415-526nm band, characterizing melanin, hematoporphyrin, etc.

[0010] The sliced ​​femtosecond image refers to the process of normalizing the pixel values ​​of each channel of the acquired 5-channel femtosecond image, scaling the pixel values ​​of each channel to the range of [0,1]. Furthermore, a sliding window slicing method is used to divide the entire image into several small patches, and each patch retains 5 channels at the same time, forming an image of size 512x512x5, which is the sliced ​​femtosecond image, for subsequent feature extraction and multimodal data fusion.

[0011] The channel normalization process refers to performing a uniform linear transformation on all pixel values ​​of a certain channel in the entire image, scaling their numerical range to a specified interval [0,1].

[0012] The FLI feature extraction refers to the extraction of features from the preprocessed, sliced ​​femtosecond images, using multi-scale channel selection, channel self-attention, and spatial filtering methods.

[0013] The specific process of multi-scale channel selection, channel self-attention, and spatial filtering is as follows: The input sliced ​​femtosecond image is first passed through two CGRs in sequence, and then combined with the input sliced ​​femtosecond image to obtain three different scale image patches. The three different scale image patches are then entered into multi-scale channel selection. In multi-scale channel selection, the three image patches are independently and in parallel processed by CGR and channel saliency selection to obtain channel enhancement features at three different scales. The three are then processed by "upsampling", "invariance", and "downsampling" respectively, and then stitched together to reduce the dimensionality to obtain multi-scale fusion features. Finally, channel self-attention is calculated and spatial filtering is performed to obtain the extracted femtosecond image features, i.e., FLI features.

[0014] The CGR is a group normalized convolutional block, which consists of a 3x3 convolution (Conv3x3), group normalization (GN), and ReLU activation function. It is used to extract spatial features. This method maintains the independence of each channel and can process each channel.

[0015] The group normalization involves dividing the feature channels into several groups, and then performing normalization independently within each group and in the spatial dimension of each sample.

[0016] The input sliced ​​femtosecond image is first passed through two CGRs in sequence: the first CGR produces an image patch of one scale, and the second CGR produces an image patch of another scale. These two image patches, together with the sliced ​​femtosecond image, produce a total of three image patches of different scales. These three image patches of different scales are used as input to the multi-scale channel selection.

[0017] The channel saliency selection calculation includes: calculating the channel saliency score, and calculating the final channel selection weight based on the learnable weight, and then using the final channel selection weight to perform channel enhancement;

[0018] The channel saliency score is calculated using the following formula:

[0019]

[0020] in Let represent the channel saliency score of the i-th channel, reflecting its contribution to the task, where i represents different channels of FLI. Indicates global average pooling. Indicates global max pooling. This refers to calculating the average value of all pixel values ​​in the i-th channel of all input feature channels. This refers to finding the maximum value among all pixel values ​​in the i-th channel of all input feature channels;

[0021] The learnable weights refer to the calculated values ​​used to enable the model to dynamically learn the relative importance of each channel during training. The learnable weight calculation process introduces learnable channel parameters and uses the sigmoid activation function for normalization, ensuring that the learnable weight values ​​are within the [0,1] interval. The calculation method is as follows:

[0022]

[0023] in This represents the learnable channel parameter of the i-th channel, initialized using Xavier initialization. (·) represents the sigmoid function. This represents the learnable weight of the i-th channel. The learnable weight will be adaptively adjusted during training to improve the model's ability to respond to channel changes.

[0024] Xavier initialization is a commonly used parameter initialization method in deep learning. It sets the initial values ​​to values ​​that conform to the variance of 1 / 2. random distribution for Assign a value. This is the number of input nodes in this layer. Xavier initialization keeps the variance of the input and output of each layer of the network similar, thereby avoiding the problems of gradient vanishing or exploding in the early stage of training.

[0025] The final channel selection weight refers to the result obtained by multiplying the channel saliency score of each channel by its learnable weight, and then normalizing the result. The calculation method is as follows:

[0026]

[0027] Here, i and j represent the channel indices. This represents the channel significance score of the i-th channel. This represents the learnable weight of the i-th channel. Indicates the channel selection weight. It applies to all channels. and The weighted summation result is used to normalize the channel selection weights. The final channel selection weight calculation ensures that the importance of each feature channel is dynamically adjusted during training, effectively filtering and enhancing feature channels that are highly correlated with the model's prognosis prediction.

[0028] The channel enhancement refers to utilizing the final channel selection weight. This is applied to the original features represented by each feature channel to obtain the channel-enhanced features, calculated as follows:

[0029]

[0030] in This represents the original feature of the i-th channel; This represents the final channel selection weight for the i-th channel; This represents the enhanced output feature of the i-th channel, i.e., the channel enhancement feature;

[0031] The multi-scale channel selection takes as input three image patches of different scales after passing through CGR. These three image patches of different scales are passed through CGR in parallel, and then channel saliency selection is performed on each patch separately to obtain the channel enhancement features at the three different scales, represented as follows: , and ,Will Perform upsampling, Perform downsampling. Keeping the three elements unchanged, they are concatenated and combined, and then dimensionality is reduced using a 1x1 convolution (Conv1x1) to generate dimensionality-reduced features, i.e., multi-scale fusion features.

[0032] The aforementioned channel self-attention refers to extending the standard self-attention mechanism by using channel-aware projection and dynamic scaling attention mechanisms to make it suitable for channel feature processing. For input... ∈ ,in, The multi-scale fusion feature is represented by a tensor. Represents the real number space, meaning that all elements in the tensor are real numbers. This indicates the number of samples input into the model at one time. Indicates the number of channels. Indicates feature height, The feature width is represented by the channel self-attention, which is achieved through the following steps:

[0033] Step 1: Channel-aware projection. Channel-aware projection refers to the projection of the input... Through the weight matrix , , Generate query matrix ,key matrix Value matrix And it is activated by ReLU, calculated as follows:

[0034]

[0035] , , The three weight matrices are parameter matrices that are automatically learned during the training process;

[0036] Step 2: Dynamic scaling attention mechanism. The dynamic scaling attention mechanism refers to introducing a dynamic scaling factor when calculating attention weights. The methods for calculating attention weights and dynamic scaling factors are as follows:

[0037]

[0038] in scaling factor Combined with adaptive input, as shown in the formula on the right, here express Feature dimensions, It is a lightweight learnable function designed as a multi-layer MLP and activated by the sigmoid function, which can adaptively adjust the dynamic scaling factor according to the input features; Indicates will and Perform a dot product to obtain the original attention score; then use a dynamic scaling factor. Scale the original attention score and apply... The function yields the final matrix form of the attention weights. ;

[0039] Step 3: Multi-head aggregation. Multi-head aggregation refers to summing the attention outputs from multiple heads, calculated using the following formula:

[0040]

[0041] in This represents the attention weight matrix for the h-th head. Let represent the value matrix of the h-th head; their product represents the attention of the h-th head, which is also the output of the h-th head.

[0042] Multi-head attention refers to the parallel execution of the aforementioned channel perception projection, dynamic scaling attention mechanism, and attention computation on h "heads". Here, "head" refers to a set of independent computing units, each head learning different channel relationships; each head outputs an attention, and there will be h attention outputs in the end. Summing these h attentions is equivalent to summing the multi-head attention outputs.

[0043] Final output It is a global channel feature, and this process effectively captures long-range channel dependencies by dynamically adjusting the interaction weights between feature channels;

[0044] The spatial filtering refers to suppressing channel noise in femtosecond images by dynamically adjusting the variance using adaptive Gaussian variance adjustment technology, effectively reducing noise while maintaining channel integrity. The adaptive Gaussian variance adjustment technology is implemented through the following steps:

[0045] Step 1: Baseline Variance Setting. The baseline variance setting is an initial setting determined based on the global channel features of the input features, and the calculation method is as follows:

[0046]

[0047] in Represents the normalized global channel features The normalization method uses mean-variance standardization. The normalized variance representing the global channel characteristics provides a consistent starting point for subsequent adjustments;

[0048] Mean-variance standardization refers to transforming a feature into a new feature with a mean of 0 and a variance of 1.

[0049] Step 2: Adaptive Variance Adjustment Mechanism. This mechanism adaptively adjusts the variance based on the different characteristics of each feature channel, effectively reducing noise while preserving image details valuable for predicting the prognosis of hepatocellular carcinoma. The calculation method is shown in the formula:

[0050]

[0051] in This represents the normalized variance of channel i. This represents the normalized standard deviation of channel i. , It is a moderating factor that controls the degree of influence of local variance on the adjustment, 0≤ ≤1, =1− , Indicates the baseline variance. Indicates the baseline standard deviation;

[0052] Step 3: Gaussian Filtering. The Gaussian filtering process involves applying a Gaussian filter to each feature channel to smooth the data, effectively reducing noise while preserving channel information. Gaussian kernel function. The definition is as follows:

[0053]

[0054] in Represents pixels Its neighboring pixels The Euclidean distance between them For adaptive variance, The standard deviation is adaptive; Euclidean distance represents the straight-line distance between two points;

[0055] The Gaussian filtering process is implemented as shown in the formula:

[0056]

[0057] In the formula Represents pixels The spatial neighborhood range (usually including all adjacent pixels). Represents the position of the i-th input feature channel. The original eigenvalues, Then the i-th channel is at position The output value after spatial filtering;

[0058] The final output of spatial filtering is composed of all pixels in i channels. of The resulting features are the FLI features.

[0059] The FLI features obtained by employing the aforementioned multi-scale channel selection, channel self-attention, and spatial filtering methods effectively achieve decoupling and denoising of spatial and channel features.

[0060] The spatial and channel feature decoupling and denoising refers to processing the spatial information and characteristics of different channels in the femtosecond image independently to avoid mutual interference. On this basis, by dynamically selecting important channels and filtering out spatial noise, redundant channels and spatial noise are eliminated respectively, thereby extracting purer and more discriminative FLI features.

[0061] The multimodal data fusion process refers to performing cross-modal preliminary fusion of pathology report features and clinical data features to obtain preliminary fusion features, then concatenating the preliminary fusion features with the pooled and flattened FLI features to form composite features, and finally performing multimodal residual feature fusion of the preliminary fusion features, composite features and FLI features to obtain the multimodal residual feature fusion output features, i.e. fusion features;

[0062] The characteristics of the pathology report refer to the pathology report after... The features obtained from the model, their tokenization, and subsequent feature extraction methods are as follows:

[0063]

[0064]

[0065] In the first formula, express The word segmentation module in the model, This indicates the input pathology report text. Indicates The pathology report text processed by the word segmenter was converted into text. The sequence, i.e., the word segmentation operation; in the second formula, Indicates complete Model, Indicates the process Text processed by the word segmenter sequence, Indicates the process The feature vector obtained after model processing is of size 1xN, which is the pathology report feature;

[0066] The clinical data characteristics refer to the clinical data processed by a multilayer sensing machine. The resulting features demonstrate that multilayer perceptrons excel at learning nonlinear representations of input data, making them an ideal choice for extracting important features from clinical data. The feature extraction method is as follows:

[0067]

[0068] in This represents the input clinical data. for Model, Indicates The processed feature vector is of size 1xN, which is the clinical data feature.

[0069] The preliminary cross-modal fusion refers to the use of a cross-attention mechanism to dynamically enhance the interaction between pathology reports and clinical data modalities during the initial fusion phase. Clinical data features are used as query vectors. The features of the pathology report serve as key vectors. AND value vector The formula for calculating the cross-attention mechanism is as follows:

[0070]

[0071] in This is a scaling factor used for stabilization. calculate, This indicates that the query vector will be used. With key vector Perform a dot product, calculate the correlation between the two, scale the correlation result, and then... The attention weight matrix is ​​obtained through normalization, and a 1xN vector feature of length N is calculated through the cross-attention mechanism. The initial fusion features for the final output are the attention weight matrix and The product result, this method significantly enriches the fusion process by focusing the model on the features that contribute most to the prediction results across modalities. Pathology reports generated by pathologists through pathological image analysis and clinical data obtained from quantitative clinical tests can be effectively combined using a cross-attention mechanism. By guiding the features of the pathology report to indicate the features of the clinical data, the initial fusion of features ultimately achieves the organic integration of the unique characteristics of the two modalities.

[0072] The composite feature refers to the feature formed by splicing the preliminary fused feature with the pooled and flattened FLI feature. The pooled and flattened FLI feature is a 1x The feature vector, the composite feature is a 1x( Feature vectors of size (+N);

[0073] The multimodal residual feature fusion takes as input preliminary fusion features, composite features, and FLI features. The composite features and FLI features are processed as follows: the composite features are passed through a fully connected layer to obtain a feature vector of size 1xN; the FLI features are first passed through a ResNet residual network and then flattened by pooling to obtain a feature vector of size 1xN; then the processed composite features, FLI features, and the unprocessed preliminary fusion features are concatenated and fused to obtain a feature vector of size 3xN, i.e., the fused features.

[0074] The ResNet residual network is a standard ResNet residual block;

[0075] The fused features output by MRFFM are input into a fully connected layer, flattened into a 1xE vector of length E, and then fed into a fully connected layer and a sigmoid function. The cross-entropy loss function is used for training.

[0076] During model training, modal models with limited data (such as BERT and MLP) are pre-trained separately to stabilize their parameters; and the parameters of the pre-trained BERT and MLP are frozen, and only MCSCSASF, MRFFM and the last fully connected layer into which the fused features are trained are trained, thereby reducing the risk of overfitting in low-sample modal models.

[0077] The prognostic prediction process refers to inputting the fusion features output by MRFFM into a fully connected layer and performing the final prognostic prediction. The prognostic prediction process is a binary classification task, which divides patients into two groups based on whether they can live for more than three years: long-term survival group (>3 years) and short-term survival group (≤3 years).

[0078] The beneficial effects of this invention are:

[0079] 1) By adopting deep learning technology, the input of human resources is greatly reduced, automated assisted diagnosis is achieved, and the efficiency of HCC prognosis prediction is improved;

[0080] 2) By using MCSCSASF, multi-scale spatial and channel features are decoupled and denoised, improving the utilization of channel information, reducing channel redundancy and noise, and dynamically selecting key channels related to prognosis prediction.

[0081] 3) MRFFM fully integrates FLI features, pathology report features and clinical data features, and makes full use of complementary information between modalities to achieve deep fusion of multimodal data, thereby improving prediction accuracy and robustness. Attached Figure Description

[0082] Figure 1 This is an overall flowchart of the hepatocellular carcinoma prognostic prediction method MHPF-FLI based on multimodal femtosecond laser pathological image fusion constructed in this embodiment of the invention;

[0083] Figure 2 This is a flowchart of the multi-scale channel selection and channel self-attention and spatial filtering MCSCSASF in an example of the present invention;

[0084] Figure 3 This is a flowchart of the multi-scale channel selection MCS in an example of the present invention;

[0085] Figure 4 This is a flowchart of the Channel Saliency Selection (ChanS) process in an example of the present invention.

[0086] Figure 5 This is a flowchart of the multimodal residual feature fusion (MRFFM) process in an example of the present invention. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] To better understand the specific implementation methods, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0089] Example 1

[0090] This embodiment includes:

[0091] Data acquisition and preprocessing: This process yields sliced ​​femtosecond images, pathology reports, and clinical data.

[0092] FLI feature extraction process: This process obtains FLI features from sliced ​​femtosecond images;

[0093] Multimodal data fusion process: This process involves fusing multimodal data from the processed features.

[0094] Prognostic prediction process: This process yields the classification results for prognostic prediction of hepatocellular carcinoma;

[0095] As can be seen, the method proposed in this embodiment constructs a dataset by collecting femtosecond images, pathology reports and clinical data, and extracts features through a network to obtain a prognostic prediction model for hepatocellular carcinoma.

[0096] The following is in conjunction with the appendix Figures 1-5 The following details an example implementation of the present invention, along with the various processes of this method. Specifically, the specific function and operational behavior of each process are as follows:

[0097] The data acquisition and preprocessing process yields sliced ​​femtosecond images, pathology reports, and clinical data.

[0098] Sliced ​​femtosecond images refer to the process of normalizing the pixel values ​​of each channel of a 5-channel femtosecond image, scaling each channel pixel value to the [0,1] range, and then using a sliding window to slice the entire image into several small patches, ensuring that each patch retains 5 channels simultaneously, forming an image of size 512x512x5, i.e., a sliced ​​femtosecond image, which is used for subsequent feature extraction and multimodal data fusion.

[0099] Specifically, the femtosecond image in the invention has five channels, each receiving a different signal band, and thus displaying different tissue components (for example, the third harmonic THG channel receives signals in the 340-380nm band, characterizing cell membranes, nuclear membranes, etc.; the three-photon fluorescence 3PEF channel receives signals in the 415-526nm band, characterizing melanin, hematoporphyrin, etc.).

[0100] This method obtains information that is beneficial for subsequent model feature extraction and fusion operations through preprocessing of the collected data, which is helpful for subsequent use.

[0101] FLI feature extraction process: This process obtains FLI features from sliced ​​femtosecond images;

[0102] The FLI feature refers to the result of feature extraction from sliced ​​femtosecond images proposed in this invention, i.e., the FLI feature.

[0103] See Figure 2 , Figure 3 and Figure 4 The process of obtaining FLI feature results from the sliced ​​image requires that the input sliced ​​femtosecond image first undergoes CGR, then enters the multi-scale channel selection (MCS) for CGR and channel saliency selection, then performs channel self-attention calculation and spatial filtering to obtain the extracted FLI features.

[0104] In MCS, after the input is processed by CGR, it enters Channel Saliency Selection (ChanS). In ChanS, the channel saliency score is calculated, and the final channel selection weight is calculated based on the learnable weights. The final channel selection weight is then used for channel enhancement.

[0105] See Figure 2 and Figure 3 The CGR, as mentioned, consists of 3x3 convolution, group normalization, and ReLU activation function, used to extract spatial features. The sliced ​​femtosecond image is first passed through two CGRs sequentially, and then combined with the sliced ​​femtosecond image to obtain images of sizes [sizes to be filled in]. , and Three different image patches are input into the MCS in parallel. Each of the three image patches is then subjected to a CGR process to obtain images with the following dimensions: and The image blocks are then simultaneously input into the three Channels of the MCS. This method maintains the independence of each channel and can efficiently process each channel. =512, =512, =5.

[0106] See Figure 4 The channel saliency score refers to performing global average pooling on each input channel separately. and global max pooling Operation. In ChanS, the channel saliency score is calculated using the following formula:

[0107]

[0108] in This represents the channel saliency score of the i-th channel, reflecting its contribution to the task. This combined global pooling method can effectively capture the global importance of each channel in different scenarios.

[0109] The learnable weights refer to the values ​​calculated to enable the model to dynamically learn the relative importance of each channel during training. This calculation process introduces learnable channel parameters and uses the sigmoid activation function for normalization to ensure that the learnable weight values ​​are within the range of [0,1]. The calculation method is as follows:

[0110]

[0111] in This represents the learnable channel parameters for the i-th channel. (·) represents the sigmoid function. This represents the learnable weight of the i-th channel. The learnable weight will be adaptively adjusted during training to improve the model's ability to respond to channel changes.

[0112] The final channel selection weight refers to the result obtained by multiplying the channel saliency score of each channel by the learnable weight, and then normalizing the result. The calculation method is shown in the formula:

[0113]

[0114] Here, i and j represent the channel indices. The final channel selection weight calculation ensures that the importance of each feature channel is dynamically adjusted during training, effectively filtering and enhancing feature channels that are closely related to the model's prognostic prediction results in the task.

[0115] The channel enhancement refers to utilizing the final channel selection weight. The final channel selection weight is 1x1x , A value of 5 is applied to the original features of each feature channel to obtain the channel-enhanced features. The calculation method is shown in the formula:

[0116]

[0117] Finally, see Figure 3 After parallel ChanS, the dimensions obtained are as follows: , and The channel enhancement features are represented as , and ,Will Perform upsampling, Perform downsampling. Keeping the three elements unchanged, they are concatenated and combined, and then dimensionality reduced using a 1x1 convolution to generate dimensionality-reduced features, i.e., multi-scale fusion features, with a size of [missing value]. For use in subsequent processing. The default size is 8;

[0118] The aforementioned channel self-attention refers to extending the standard self-attention mechanism by using channel-aware projection and dynamic scaling attention mechanisms to make it suitable for channel feature processing. In the example, the input... ∈ ,in This represents multi-scale fusion features, i.e. Figure 2 Redundancy removal and feature extraction in the process =16, =8, =128, =128, the channel self-attention calculation requires the following method:

[0119] 1) Channel-sensing projection;

[0120] Specifically, channel-aware projection is the projection of input... Through the weight matrix , , generate , , And it is activated by ReLU, calculated as follows:

[0121]

[0122] 2) Implementation of the dynamic scaling attention mechanism;

[0123] Specifically, a dynamic scaling factor is introduced when calculating attention weights. The calculation methods for attention weights and dynamic scaling factors are as follows:

[0124]

[0125] in scaling factor Combined with adaptive input, as shown in the formula on the right, here express The dimension has a value of 512. It is a lightweight learnable function designed as a 3-layer MLP and activated by the sigmoid function, which can adaptively adjust the dynamic scaling factor according to the input features.

[0126] 1) Multi-head aggregation method;

[0127] Specifically, multi-head aggregation refers to summing the outputs of the attention from multiple heads, calculated using the formula:

[0128]

[0129] Final output It is a global channel feature, that is Figure 2 The channel self-attention mechanism extracts features, which effectively captures long-range channel dependencies by dynamically adjusting the interaction weights between feature channels.

[0130] The spatial filtering refers to suppressing channel noise in femtosecond images by dynamically adjusting the variance using adaptive Gaussian variance adjustment technology, effectively reducing noise while maintaining channel integrity. The relevant calculation method for adaptive Gaussian variance adjustment technology is as follows:

[0131] 1) Baseline variance settings;

[0132] Specifically, the baseline variance is an initial setting determined based on the global channel features of the input features, and is calculated using the following formula:

[0133]

[0134] in Indicates the normalized result The normalization method uses mean-variance standardization, which involves subtracting the mean of the global channel features from the global channel features and then dividing by the standard deviation of the global channel features. This represents the normalized variance of the global channel features, providing a consistent starting point for subsequent adjustments.

[0135] 2) Adaptive variance adjustment mechanism;

[0136] Specifically, the adaptive variance adjustment mechanism adaptively adjusts the variance based on the different characteristics of each feature channel, effectively reducing noise while preserving important image details. This represents the normalized variance of channel i. It is a moderating factor that controls the degree of influence of local variance on the adjustment, 0≤ ≤1, =1− As shown in the formula:

[0137]

[0138] 3) Gaussian filtering process:

[0139] Specifically, a Gaussian filter is applied to each feature channel to smooth the data, effectively reducing noise while preserving channel information. The Gaussian kernel function G is defined by the formula:

[0140]

[0141] in Represents pixels Its neighboring pixels The Euclidean distance between them For adaptive standard deviation, the Gaussian filtering process is calculated using the following formula:

[0142]

[0143] In the formula Represents pixels The neighborhood range (usually including all adjacent pixels). Represents the position of the i-th input feature channel. The original eigenvalues, Then the i-th channel is at position Output characteristics after spatial filtering;

[0144] The final output of spatial filtering is composed of all pixels from different channels. of The features constituted, namely the FLI features obtained, are... Figure 2 Denoising and feature extraction in [the context of the process].

[0145] Multimodal data fusion process: In this process, the pathological report features and clinical data features obtained by the following preprocessing method are first fused across modalities to obtain preliminary fused features. Then, the preliminary fused features are spliced ​​with the pooled and flattened FLI features to form composite features. Finally, the preliminary fused features, composite features and FLI features are fused with multimodal residual features to obtain the fused features output by multimodal residual feature fusion.

[0146] See Figure 1 The pathology report characteristics refer to the pathology report being processed by the pathology department. The features obtained from the model, pathology report (hidden=768) Generate context embeddings. This process first performs word segmentation to decompose the pathology report text into meaningful units, which are discrete data. Then, the discrete data is converted into low-dimensional vectors, i.e., feature vectors, with a length of 1xN, which are the features of the pathology report. The process is as follows:

[0147]

[0148]

[0149] The clinical data characteristics refer to the clinical data after... The characteristics obtained. Clinical data... Generate vector representations. This process transforms discrete clinical data into low-dimensional vectors, or feature vectors, with a length of 1xN, representing pathology report features. The process is as follows:

[0150]

[0151] See Figure 1 The aforementioned cross-modal preliminary fusion refers to the use of a cross-attention mechanism to dynamically enhance the interaction between pathology reports and clinical data modalities during the initial fusion stage, with clinical data features serving as query vectors. The features of the pathology report serve as key vectors. AND value vector The calculation method for the cross-attention mechanism is shown in the formula:

[0152]

[0153] in This is a scaling factor used for stabilization. The method focuses the model on the features that contribute most to prognostic prediction across modalities to capture the correlation between clinical data and pathology reports, ultimately generating a 1xN feature vector of length N, which is the preliminary fusion feature.

[0154] Before feature fusion via MRFFM, such as Figure 1 As shown, the preparation process for input features is as follows:

[0155] Input 1 comes from the preliminary fused features calculated by the cross-attention mechanism, while input 3 originates from the FLI features output by MCSCSASF. For input 2, the FLI features need to undergo average pooling and flattening operations to obtain the pooled and flattened FLI features, i.e., features of length [length missing]. 1x The feature vector is then concatenated with the preliminary fused features, and the resulting composite feature is used as the input to MRFFM, with a size of 1x( eigenvectors of (+N);

[0156] To optimize multimodal data fusion, MRFFM is used for combination and enhancement while minimizing information loss.

[0157] Specifically, refer to Figure 5 The processing of the multimodal residual feature fusion MRFFM is as follows:

[0158] Input 2 passes through a fully connected layer to obtain a feature vector of size 1xN; Input 3 first passes through a ResNet residual network and then undergoes pooling and flattening to obtain a feature vector of size 1xN; The processed inputs 2 and 3 are then concatenated with input 1 to obtain a feature vector of size 3xN, which is the fused feature.

[0159] The fused features output by MRFFM are input into a fully connected layer, flattened into a 1xE vector of length E, and then trained using a cross-entropy loss function through a fully connected layer and a sigmoid function.

[0160] During training, modal models with limited data (such as BERT and MLP) are pre-trained separately to stabilize their parameters. The BERT model is pre-trained on PubMed literature summaries, a large medical text corpus, and the MLP is pre-trained on publicly available clinical datasets. The parameters of the pre-trained BERT and MLP are frozen, and only MCSCSASF, MRFFM, and the last fully connected layer with fused features are trained. This strategy can significantly reduce the risk of overfitting in low-sample modal models.

[0161] Prognostic prediction process: This process yields the classification results for prognostic prediction of hepatocellular carcinoma;

[0162] After the model is trained, the prognosis of hepatocellular carcinoma can be predicted. After the femtosecond images of the tested patients are passed through the model for prediction, if the patient's survival time is less than or equal to 3 years, they are classified as a short-term survival group and the output is 0; if the survival time is greater than 3 years, they are classified as a long-term survival group and the output is 1.

[0163] In the specific implementation of this application, 836 femtosecond images, 152 pathology reports, and 3034 clinical data were obtained from 152 patients in two hospitals. Five-fold cross-validation was performed based on the patient identifiers, and the accuracy rate exceeded 92%, verifying the effectiveness of the method of this invention.

[0164] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal prognostic prediction method for hepatocellular carcinoma based on femtosecond laser pathological images, characterized in that, The method includes the following steps: Step 1: Data Acquisition and Preprocessing Femtosecond laser pathology images (FLI) are acquired and preprocessed to obtain sliced ​​femtosecond images; pathology reports and clinical data are acquired. Step 2: FLI Feature Extraction Feature extraction is performed on the preprocessed sliced ​​femtosecond images using a multi-scale channel selection, channel self-attention, and spatial filtering method. Specifically, the input sliced ​​femtosecond images are first processed through group normalized convolutional blocks (CGR) to obtain image patches of three different scales. These three image patches are then processed through multi-scale channel selection. In multi-scale channel selection, the three image patches are processed independently and in parallel through CGR and channel saliency selection to obtain channel enhancement features at three different scales. After "upsampling," "invariance," and "downsampling" processing, the three features are concatenated and dimensionality reduced to obtain multi-scale fusion features. Finally, channel self-attention is calculated and spatial filtering is performed to obtain the extracted femtosecond image features, i.e., FLI features. Step 3: Multimodal data fusion The collected pathology reports and clinical data are processed to obtain pathology report features and clinical data features. The pathology report features and clinical data features are then subjected to cross-modal preliminary fusion to obtain preliminary fused features. The preliminary fused features are then spliced ​​with the pooled and flattened FLI features to form composite features. Finally, the preliminary fused features, composite features and FLI features are fused with multimodal residual features to obtain the multimodal residual feature fusion output features. Step 4: Prognostic Prediction The features output from the fusion of multimodal residual features are input into the fully connected layer for final prediction.

2. The multimodal hepatocellular carcinoma prognostic prediction method according to claim 1, characterized in that, Step 1, the sliced ​​femtosecond image, refers to the channel normalization processing of the pixel values ​​of each channel of the acquired 5-channel femtosecond image, scaling the pixel values ​​of each channel to the [0,1] range; using sliding window slicing, the entire image is divided into several small patches, and each patch retains 5 channels at the same time, forming an image of size 512x512x5, i.e., the sliced ​​femtosecond image, which is used for subsequent feature extraction and multimodal data fusion; The channel normalization process refers to performing a uniform linear transformation on all pixel values ​​of a certain channel in the entire image, scaling their numerical range to a specified interval [0,1].

3. The multimodal hepatocellular carcinoma prognostic prediction method according to claim 1, characterized in that, Step 2, the CGR is a group normalized convolutional block, which consists of 3x3 convolution, group normalization and ReLU activation function, used to extract spatial features, maintain the independence of each channel and can process each channel; Group normalization involves dividing the feature channels into several groups and then performing normalization independently within each group and in the spatial dimension of each sample. The feature channels are the channels of the femtosecond image; The channel saliency selection includes: calculating the channel saliency score, and calculating the final channel selection weight based on the learnable weight, and then using the final channel selection weight to perform channel enhancement; The channel saliency score is calculated using the following formula: ； in Let represent the channel saliency score of the i-th channel, reflecting its contribution to the task, where i represents different channels of FLI. This refers to calculating the average value of all pixel values ​​in the i-th channel of all input feature channels. This refers to finding the maximum value among all pixel values ​​in the i-th channel of all input feature channels; The learnable weights refer to the calculated values ​​used to enable the model to dynamically learn the relative importance of each channel during training. The learnable weight calculation process introduces learnable channel parameters and uses the sigmoid activation function for normalization to ensure that the learnable weight values ​​are within the range of [0,1]. The calculation method is as follows: ； in This represents the learnable channel parameter of the i-th channel, initialized using Xavier initialization. (·) represents the sigmoid function. This represents the learnable weight of the i-th channel. The learnable weight will be adaptively adjusted during training to improve the model's ability to respond to channel changes. Xavier initialization is achieved by setting the initial value to conform to the variance. random distribution for Assign a value. This is the number of input nodes in this layer; The final channel selection weight refers to the result obtained by multiplying the channel saliency score of each channel by its learnable weight, and then normalizing the result. The calculation method is as follows: ； Here, i and j represent the channel indices. This represents the channel significance score of the i-th channel. This represents the learnable weight of the i-th channel. Indicates the channel selection weight. It applies to all channels. and The weighted summation result is used to normalize the channel selection weights. The final channel selection weight calculation ensures that the importance of each feature channel is dynamically adjusted during training, effectively filtering and enhancing feature channels that are strongly correlated with the model's prognosis prediction results in the task. The channel enhancement refers to using the final channel selection weight. This is applied to the original features of each channel to obtain the channel enhancement features, calculated as follows: ； in This represents the original feature of the i-th channel. This represents the final channel selection weight for the i-th channel. This represents the enhanced output feature of the i-th channel, i.e., the channel enhancement feature.

4. The multimodal hepatocellular carcinoma prognostic prediction method according to claim 1, characterized in that, Step 2, which involves entering multi-scale channel selection, first passes the input sliced ​​femtosecond image through two CGRs sequentially, then adds the input sliced ​​femtosecond image to the CGR, resulting in image patches of three different scales. The multi-scale channel selection refers to passing these three different scale image patches through CGRs in parallel, and then performing the channel saliency selection as described in claim 3 on each patch, to obtain channel enhancement features at three different scales, represented as follows: , and ;Will Perform upsampling, Perform downsampling. Keeping the three elements unchanged, they are concatenated and combined, and 1x1 convolution is used to reduce the dimensionality, generating the dimensionality-reduced features, i.e., multi-scale fusion features.

5. The multimodal hepatocellular carcinoma prognostic prediction method according to claim 1, characterized in that, The channel self-attention mentioned in step 2 refers to extending the standard self-attention mechanism by using channel-aware projection and dynamic scaling attention mechanisms to make it suitable for channel feature processing. For input... ∈ ,in, The multi-scale fused features are represented by a tensor. Represents the real number space, meaning that all elements in the tensor are real numbers. This indicates the number of samples input into the model at one time. Indicates the number of channels. Indicates feature height, The feature width is represented by the channel self-attention, which is achieved through the following steps: i) Channel-aware projection, for input Through the weight matrix , , Generate query matrix ,key matrix Value matrix And it is activated by ReLU, calculated as follows: ； ii) Dynamic scaling attention mechanism: When calculating attention weights, a dynamic scaling factor is introduced. The calculation methods for attention weights and dynamic scaling factors are as follows: ； in scaling factor Combined with adaptive input, as shown in the formula on the right, here express Feature dimensions, It is a lightweight, learnable function designed as a multi-layer MLP and activated by the sigmoid function. It can adaptively adjust the dynamic scaling factor based on the input features. Indicates will and Perform a dot product to obtain the raw attention score, then use a dynamic scaling factor. Scale the original attention score and apply... The function yields the final matrix form of the attention weights. ; iii) Multi-head aggregation: summing the multi-head attention outputs, calculated as follows: ； in This represents the attention weight matrix for the h-th head. This represents the value matrix of the h-th head. Multiplying these two matrices represents the output of the h-th head. The final output is... It is a global channel feature, and this process effectively captures long-range channel dependencies by dynamically adjusting the interaction weights between feature channels.

6. The multimodal hepatocellular carcinoma prognostic prediction method according to claim 1, characterized in that, The spatial filtering described in step 2 refers to suppressing channel noise in the femtosecond image by using adaptive Gaussian variance adjustment technology to adjust the variance, thereby maintaining channel integrity while effectively reducing noise. The adaptive Gaussian variance adjustment technology specifically includes: a) Baseline variance setting: The initial setting is determined based on the global channel features of the input features, and is calculated as follows: ； in Represents the normalized global channel features , The normalized variance representing the global channel characteristics provides a consistent starting point for subsequent adjustments; b) Adaptive variance adjustment mechanism: This mechanism adaptively adjusts the variance based on the different characteristics of each feature channel, effectively reducing noise while preserving image details valuable for predicting the prognosis of hepatocellular carcinoma. The calculation method is as follows: ； in This represents the normalized variance of channel i. This represents the normalized standard deviation of channel i. , It is a moderating factor that controls the degree of influence of local variance on the adjustment, 0≤ ≤1, =1− , Indicates the baseline variance. Indicates the baseline standard deviation; c) Gaussian filtering process: Gaussian filtering is applied to each feature channel to smooth the data, effectively reducing noise while preserving channel information. (Gaussian kernel function) Defined as follows: ； in Represents pixels Its neighboring pixels The Euclidean distance between them For adaptive variance, For adaptive standard deviation; The Gaussian filtering process is implemented as follows: ； In the formula Represents pixels The spatial neighborhood range includes all adjacent pixels. Represents the position of the i-th input feature channel. The original eigenvalues, Then the i-th channel is at position The output value after spatial filtering; The final output of spatial filtering is composed of all pixels. of The resulting features are the FLI features.

7. The multimodal hepatocellular carcinoma prognostic prediction method according to claim 1, characterized in that, Step 3, the pathology report characteristics, refers to the pathology report after... The features obtained from the model, their tokenization, and subsequent feature extraction methods are as follows: ； in, express The word segmentation module in the model, This represents the text of the input pathology report; Indicates The pathology report text processed by the word segmenter was converted into text. sequence; ； in, Indicates complete Model, Indicates the process Text processed by the word segmenter sequence, This represents a feature vector of size 1xN obtained after model processing, i.e., pathology report features; The clinical data features refer to the features obtained from clinical data through MLP, and the feature extraction method is as follows: ； in This represents the input clinical data. for Model, Indicates The processed feature vector is of size 1xN, which is the clinical data feature. The preliminary cross-modal fusion refers to the use of a cross-attention mechanism to dynamically enhance the interaction between pathology reports and clinical data modalities during the initial fusion phase; clinical data features serve as query vectors. The features of the pathology report serve as key vectors. AND value vector The cross-attention mechanism is calculated as follows: ； in This is a scaling factor used for stabilization. calculate; This indicates that the query vector will be used. With key vector Perform a dot product, calculate the correlation between the two, scale the correlation result, and then... Normalization yields the attention weight matrix. The initial fusion features for the final output are the attention weight matrix and The product result, this method focuses the model on the features that contribute the most to the prediction results across modalities, and finally achieves the fusion of the unique characteristics of the two modalities by initially fusing the features; The composite feature refers to the feature formed by splicing the preliminary fusion feature with the FLI feature after pooling and flattening; The multimodal residual feature fusion takes as input preliminary fusion features, composite features, and FLI features. The composite features and FLI features are processed as follows: the composite features are passed through a fully connected layer, the FLI features are first passed through a ResNet residual network and then pooled and flattened, and finally the above-processed composite features, FLI features, and unprocessed preliminary fusion features are concatenated and fused to obtain the features output by multimodal residual feature fusion, thereby preserving modality-specific information and enhancing complementarity.

Images