HER2 positive breast cancer neoadjuvant therapy response prediction system based on multi-omics deep fusion network

Abstract

The invention relates to the technical field of data processing, and provides an HER2 positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network, and the system comprises a data collection and preprocessing module which is used for obtaining and preprocessing the multi-omics data of a patient, and generating a standardized multi-mode feature set; the multi-omics deep fusion network module comprises a single-mode deep sub-network group which is used for extracting deep features corresponding to each mode from the standardized multi-mode feature set; the cross-modal interaction fusion module is used for distributing fusion weights by calculating feature similarity among modals and performing redundancy filtering on the features to generate cross-modal interaction features; the global information fusion module is used for carrying out global modeling on the cross-modal interaction features and outputting global attention features; and the prediction output module is used for generating a prediction result aiming at the neoadjuvant therapy response based on the global attention features. According to the method, the accurate prediction of the pCR state, the treatment sensitivity and the risk stratification can be realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a system for predicting the response to neoadjuvant therapy for HER2-positive breast cancer based on a multi-omics deep fusion network. Background Technology

[0002] HER2-positive breast cancer, an important subtype of breast cancer, accounts for approximately 20%-30% of all breast cancer cases. It is characterized by rapid disease progression and a high risk of recurrence, making neoadjuvant therapy a crucial means of improving patient prognosis. However, there are significant individual differences in the response to neoadjuvant therapy among patients, with some failing to achieve pathological complete response (pCR) or even developing drug resistance. Therefore, accurate preoperative prediction of treatment response is essential for optimizing treatment strategies.

[0003] Currently, the prediction of response to neoadjuvant therapy in breast cancer mainly relies on single-modality data (such as clinicopathological features and single gene detection), which suffers from limited information dimensions and insufficient prediction accuracy. With the development of multi-omics technologies, multi-dimensional data such as pathological images, proteomics, and genomics have made accurate prediction possible, but existing technologies still face three core challenges: First, missing values ​​are common in multi-omics data (such as insufficient sensitivity of serum protein detection and differences in sequencing data quality). Traditional missing value imputation methods (such as mean imputation and single autoencoders) cannot balance data continuity and accuracy, easily leading to feature distortion. Second, the optimization of feature extraction models for modalities such as pathological images relies on traditional algorithms (such as gradient descent and basic genetic algorithms). These algorithms struggle to dynamically balance the exploration (global search) and development (local optimization) stages, easily getting trapped in local optima and affecting feature discrimination ability. Third, feature learning for time-related data such as proteomics often adopts continuous masking strategies, focusing only on the mapping relationship between historical and future data, failing to fully explore global feature associations, resulting in limited model generalization ability. Furthermore, multimodal data is highly heterogeneous and contains a lot of redundant information, making it difficult for existing fusion models (such as early feature splicing and late result fusion) to effectively integrate information, which further restricts the improvement of prediction performance.

[0004] Therefore, developing a deep fusion prediction system that can efficiently handle missing values, optimize modal feature extraction, and fully mine multi-omics association information has become a key technical requirement for accurately predicting the response to neoadjuvant therapy in HER2-positive breast cancer. Summary of the Invention

[0005] This invention addresses the problems in existing technologies, such as poor handling of missing values ​​in multi-omics data, insufficient optimization of modal feature extraction, low efficiency of multi-modal fusion, and insufficient mining of key features. It proposes a neoadjuvant therapy response prediction system for HER2-positive breast cancer based on a multi-omics deep fusion network. By designing a multi-omics data processing flow, a single-modal optimization sub-network, and a cross-modal deep fusion architecture, it achieves high-precision prediction of treatment response.

[0006] To achieve the above objectives, the following technical solution is provided: This invention provides a HER2-positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network, comprising:

[0007] The data acquisition and preprocessing module is used to acquire and preprocess the patient's multi-omics data, including pathological imaging data, proteomic data, genomic data and clinical baseline data, to generate a standardized multimodal feature set;

[0008] The multi-omics deep fusion network module includes: a single-modality deep sub-network group, comprising dedicated sub-networks corresponding to pathological images, proteomics, genomics, and clinical baseline data, each dedicated sub-network being used to extract deep features corresponding to each modality from the standardized multi-modal feature set; a cross-modal interaction fusion module, used to receive deep features from all modalities, allocate fusion weights by calculating the feature similarity between modalities, and perform redundancy filtering on features to generate cross-modal interaction features; and a global information fusion module, used to perform global modeling of the cross-modal interaction features based on an attention mechanism and output global attention features.

[0009] The prediction output module generates prediction results for neoadjuvant therapy response based on the global attention features, including at least: pathological complete remission (pCR) status, treatment sensitivity classification, and risk stratification.

[0010] Furthermore, the data acquisition and preprocessing includes:

[0011] The preprocessing of pathological image data includes: using a semantic segmentation model to identify tumor lesion regions in digital pathological images, generating lesion masks, and normalizing the lesion region images based on the masks;

[0012] Preprocessing of proteomic data includes: imputing and standardizing missing values ​​in serum protein expression data using a deep autoencoder based on iterative median update;

[0013] Preprocessing of genomic data includes: differential gene screening and spatial co-expression analysis of spatial transcriptome data to identify core gene sets associated with treatment response;

[0014] Preprocessing of clinical baseline data includes data cleaning and standardization, resulting in standardized clinical feature vectors.

[0015] Furthermore, in the single-modal deep subnetwork group, the dedicated subnetwork for processing pathological image data is configured to perform the following operations:

[0016] Extracting various initial pathological features, including morphological, texture, and intensity information, from preprocessed lesion images;

[0017] A superpixel perturbation verification mechanism is introduced to verify the stability of the initial pathological features, and stable features are selected to form a feature set.

[0018] The weight parameters of this dedicated subnetwork are optimized using the fuzzy whale optimization algorithm to extract deep features from pathological images.

[0019] Furthermore, in the single-modal deep sub-network group, the dedicated sub-network for processing proteomic data is configured to perform the following operations:

[0020] The standardized protein expression features were processed using a random point-level masking strategy, and the masked features were reconstructed using a deep autoencoder to learn the intrinsic relationships between protein features.

[0021] Based on the changes in protein expression levels before and after treatment, combined with statistical tests and protein-protein interaction network analysis, a set of core protein features related to treatment response was screened.

[0022] The prediction sub-network is trained using the core protein feature set to extract deep proteome features.

[0023] Furthermore, within the single-modal deep subnetwork group, the dedicated subnetwork for processing genomic data is configured to perform the following operations:

[0024] Based on patient response to neoadjuvant therapy, differential expression analysis was performed on spatial transcriptome data to screen for genes significantly associated with treatment response;

[0025] Spatial co-expression analysis was performed on the screened genes to identify gene modules with coordinated spatial expression patterns in tumor tissues;

[0026] The expression features of the gene modules are learned using a convolutional neural network to extract deep genomic features.

[0027] Furthermore, within the single-modal deep subnetwork group, the dedicated subnetwork for processing clinical baseline data includes:

[0028] The feature interaction layer is used to perform non-linear transformations on standardized clinical baseline features to generate enhanced features;

[0029] A feature filtering layer is used to weight and filter the enhanced features through an attention mechanism to obtain refined features; and

[0030] A deep mapping layer is used to map the refined features to a high-dimensional space to extract deep features from the clinical baseline.

[0031] Furthermore, the cross-modal interaction fusion module is configured to perform the following operations:

[0032] Calculate the cosine similarity between deep features of any two different modalities;

[0033] Based on the calculated cosine similarity of each modality to all other modalities, the importance weight of each modality during fusion is determined;

[0034] Learnable sparse gating is introduced into the deep features of each modality to filter out feature information that is judged to be redundant and output the filtering result.

[0035] Based on the importance weights and the filtering results of the sparse gating, the deep features of all modalities are weighted and fused to generate cross-modal interaction features.

[0036] Furthermore, the global information fusion module, adopting the Transformer architecture, is configured to perform the following operations:

[0037] Positional encoding is added to the cross-modal interaction features to preserve the position and order information of the features;

[0038] A multi-head self-attention mechanism is used to perform global feature interaction and fusion on the encoded features to generate global attention features.

[0039] Furthermore, the prediction output module includes at least three parallel branches, which independently generate prediction results for pathological complete remission (pCR) status, treatment sensitivity classification, and risk stratification based on the global attention features.

[0040] Furthermore, during the training process, the total loss function of the multi-omics deep fusion network module is constructed as a weighted sum of the cross-entropy loss function and the mean squared error loss function, and the parameters are updated using an optimizer that incorporates weight decay.

[0041] Compared with the prior art, the present invention achieves the following beneficial effects:

[0042] 1. This invention designs a deep autoencoder (AM-DAE) based on an adaptive learning median imputation mechanism. It fills missing values ​​in proteomics and other multi-omics data through an iterative process of "input-reconstruction-median update", which solves the problems of discontinuous information transmission and insufficient accuracy of traditional imputation methods, and ensures data integrity and reliability.

[0043] 2. This invention designs a fuzzy whale optimization algorithm (FWOA), which integrates a Mamdani-type fuzzy system with the whale optimization algorithm (WOA) to optimize the weight parameters of the pathological image subnetwork. The boundary between the exploration phase parameter α and the development phase parameter C is dynamically adjusted to solve the problem that traditional optimization algorithms have difficulty balancing global search and local optimization, thereby improving the extraction accuracy and discrimination ability of high-order features of pathological images.

[0044] 3. This invention designs a random point-level masking strategy for the standardized protein features. Random masking with a 75% masking rate forces the model to learn the global intrinsic relationships between proteins, replacing the limitation of traditional continuous masking which only focuses on local temporal mapping, and enhancing the generalization ability of proteome feature learning.

[0045] 4. This invention constructs a three-level architecture of "single-modal feature mining - cross-modal interaction - Transformer global fusion". Through cross-modal attention mechanism and sparse gating, it realizes dynamic weight allocation and redundancy filtering of multi-omics information, solves the problem of modal heterogeneity, and finally achieves accurate prediction of pCR status, treatment sensitivity and risk stratification.

[0046] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0048] Figure 1 This is a schematic diagram of a module of a HER2-positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the system architecture and workflow of a HER2-positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network according to an embodiment of the present invention.

[0050] Figure 3 This is a schematic diagram of the core architecture of the multi-omics deep fusion network according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the workflow of the Fuzzy Whale Optimization Algorithm (FWOA) according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the random point-level masking strategy and feature learning in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0055] Figure 1 This is a schematic diagram of a module of a HER2-positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the system architecture and workflow of a HER2-positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network, according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a HER2-positive breast cancer neoadjuvant therapy response prediction system 100 of the present invention includes: a data acquisition and preprocessing module 110, a multi-omics deep fusion network module 120 and a prediction output module 130 connected in sequence.

[0056] The data acquisition and preprocessing module 110 is used to acquire and preprocess the patient's multi-omics data, including pathological imaging data, proteomic data, genomic data and clinical baseline data, to generate a standardized multimodal feature set.

[0057] Step 1: Data Acquisition and Preprocessing

[0058] Step 1.1: Multi-omics data collection

[0059] For HER2-positive early or locally advanced breast cancer patients (clinical stage II-III, according to AJCC criteria), multidimensional biomedical data were collected to form a complete dataset. The specific steps are as follows:

[0060] Step 1.1.1: Acquire pathological imaging data

[0061] Obtain pretreatment HE-stained and HER2 immunohistochemical slides from the patient, and perform full-slide scanning using a digital pathology scanner to generate digital pathology images (denoted as...). ).

[0062] Digital pathological images, generated from HE-stained sections or HER2 immunohistochemical sections using a digital pathology scanner (resolution...). (Scanned and generated, dimension is) (H is the image height, W is the image width, and 3 corresponds to the RGB three channels). The height of a digital pathological image (unit: pixels), which is the number of pixels in the vertical direction of the image. Width of a digital pathological image (unit: pixels), which is the number of pixels in the horizontal direction of the image.

[0063] Step 1.1.2: Collect proteomic data

[0064] Baseline serum samples were collected from patients before treatment and preoperatively after 6 cycles of neoadjuvant therapy. 1034 serum protein markers (denoted as ) were detected using a high-sensitivity protein detection technology (such as Olink Reveal). ), and record the changes in the expression levels of each protein.

[0065] : A collection of serum protein indicators, containing 1034 serum proteins detected by high-sensitivity protein detection technologies (such as OlinkReveal), with each element corresponding to the expression level data of a protein. : No. Expression levels of serum proteins ( The range of values ​​is The study recorded the changes in the patient's expression levels before treatment baseline and after 6 cycles of neoadjuvant therapy.

[0066] Step 1.1.3: Collect genomic data

[0067] Obtain tumor tissue samples from patients and perform spatial transcriptome sequencing (each sample contains...) Each spatial site is detected. The expression level of each gene, denoted as Meanwhile, PCR verification was performed on breast cancer-specific genes (such as ERBB2 and BRCA1 / 2).

[0068] Spatial transcriptome sequencing data matrix, dimension 1 Each row corresponds to a spatial location, and each column corresponds to the expression level of a gene. The total number of spatial sites in a spatial transcriptome sequencing sample, where each site represents a specific location on the tumor tissue. The total number of genes detected by spatial transcriptome sequencing, covering tumor-related genes and breast cancer-specific genes (such as ERBB2, BRCA1 / 2).

[0069] Step 1.1.4: Collect clinical baseline data

[0070] Collect patient age, ECOG performance status score, tumor information. Clinical information such as staging, HR (estrogens receptor / progesterone receptor) status, and Ki-67 index is used to form a clinical feature set. .

[0071] The clinical feature set contains eight clinical baseline information related to patient treatment response, which are used to assist in the construction of predictive models. : No. Clinical features ( The range of values ​​is Specifically, this includes age, ECOG performance status score, and tumors. Installment payment, HR status, Ki-67 index, etc.

[0072] Step 1.2: Multi-omics data preprocessing

[0073] Step 1.2.1: Pathological image preprocessing

[0074] The preprocessing of pathological image data includes: identifying tumor lesion regions in digitized pathological images using a semantic segmentation model, generating lesion masks, and normalizing the lesion region images based on the masks. The specific implementation steps are as follows:

[0075] Step 1.2.1.1: Lesion segmentation

[0076] A deep learning-based semantic segmentation model (such as U-Net) is used to segment tumor regions and non-tumor regions (stromal, necrosis, and normal tissue) in digital pathological images, generating lesion masks. (1 represents the lesion area, and 0 represents the non-lesion area).

[0077] : Lesion mask matrix, dimension 1 The element takes a value of 0 or 1, where 1 indicates that the corresponding pixel belongs to the tumor lesion area, and 0 indicates that it belongs to the non-lesion area (stromal, necrosis or normal tissue).

[0078] Step 1.2.1.2: Image Normalization

[0079] The grayscale of the segmented lesion area image is normalized to eliminate brightness differences caused by different scanning devices. The formula is as follows:

[0080]

[0081] The normalized pixel values ​​of the pathological image correspond to the coordinates in the original image. The result of grayscale normalization of the pixels has a value range of . . The coordinates in the digital pathological image are The original pixel grayscale value ( For pixel row index, (Indexed by pixel column). The minimum grayscale value of all pixels in a digital pathological image, used for normalization calculations to eliminate brightness differences. The maximum grayscale value of all pixels in a digital pathological image, used for normalization calculations to eliminate brightness differences. The row index of a pixel in a pathological image, with a value range of... , which corresponds to the position in the vertical direction of the image. : Column index of pixels in a pathological image, with a value range of This corresponds to the horizontal position in the image.

[0082] Step 1.2.2: Proteomic Data Preprocessing

[0083] Preprocessing of proteomic data includes: imputing and standardizing missing values ​​in serum protein expression data using a deep autoencoder based on iterative median update. The specific steps are as follows:

[0084] Step 1.2.2.1: Missing value imputation

[0085] To address missing values ​​in serum protein detection caused by detection sensitivity or sample quality, a deep autoencoder (AM-DAE) based on an adaptive learning median imputation mechanism is designed. This mechanism imputes missing values ​​through an iterative "input-reconstruction-median update" process. In this iteration, protein features containing missing values ​​are first... Input AM-DAE to obtain reconstructed values Then calculate the median of the two. ,Will As the first The input for each iteration continues until the missing value is filled and convergence is achieved (the difference in median values ​​between adjacent iterations). ).

[0086] : The number of iterations for missing value imputation, a positive integer (starting from 1), used to mark the round of the "input-reconstruction-median update" process. : No. The protein feature input matrix at the next iteration contains expression data for 1034 serum proteins, some of which have unfilled missing values. A deep autoencoder based on an adaptive learning median imputation mechanism, a neural network designed specifically for imputing missing values ​​in proteomics data, with core functions of "input feature encoding - missing value reconstruction - median iterative update". : No. The reconstructed output matrix of AM-DAE in the next iteration has dimensions and Consistent, providing reconstructed estimates for protein features containing missing values. : No. The median protein feature matrix of the next iteration, through and The average of the corresponding elements is used as the first... The input of each iteration achieves gradual convergence and filling of missing values. After the filling convergence, the first The final median expression level of each serum protein ( The range of values ​​is ), which is the protein expression data after missing values ​​have been filled in.

[0087] Step 1.2.2.2: Standardization Processing

[0088] The imputed protein data were Min-Max normalized and mapped to... The interval, the formula is:

[0089]

[0090] : No. The expression levels of each serum protein after Min-Max normalization, with values ​​mapped to [value range]. The interval eliminates the dimensional differences between different protein indicators. : Fill in the minimum of the final median expression level of all serum proteins (1034) after convergence, and use it as the lower bound for normalization calculation calibration. : Fill in the maximum value of the final median expression level of all serum proteins (1034) after convergence, and use it as the upper bound for normalization calculation calibration. : Index of serum proteins, with a value range of This corresponds to 1034 serum protein indicators detected.

[0091] Step 1.2.3: Genomic Data Preprocessing

[0092] Preprocessing of genomic data includes differential gene screening and spatial co-expression analysis of spatial transcriptome data to identify core gene sets associated with treatment response. The specific implementation steps are as follows:

[0093] Step 1.2.3.1: Sequencing data filtering

[0094] Removal of expression levels in spatial transcriptome sequencing Low-quality genes are discarded, while high-reliability gene sets are retained.

[0095] Step 1.2.3.2: Spatial Clustering and Cellular Deconvolution

[0096] Clustering algorithms (such as Seurat) were used to cluster spatial sites (resolution set to 0.8) to divide them into functionally similar spatial regions; cell type deconvolution tools (such as CIBERSORTx) were used to calculate the proportion of cell types at each spatial site. .

[0097] The first step in space transcriptome sequencing At the spatial location, the th The percentage of each cell type, with a range of values, is as follows: This reflects the cellular composition ratio at that site. Spatial site index for spatial transcriptome sequencing, with values ​​ranging from [value range missing]. ( 1: Total number of spatial sites in a single sample, representing a specific physical location on the tumor tissue. 2: Cell type index, representing the different cell types present in the tumor tissue (such as tumor cells, immune cells, stromal cells, etc.).

[0098] Step 1.2.4: Clinical baseline data preprocessing

[0099] Preprocessing of clinical baseline data includes data cleaning and standardization, resulting in standardized clinical feature vectors. Specifically:

[0100] The collected clinical baseline feature set Preprocessing is performed to generate standardized feature vectors that can be directly used for model fusion.

[0101] Data cleaning: Inspecting and handling obvious outliers or recording errors.

[0102] Standardization: All eight clinical features were standardized to eliminate dimensional differences. Standardization included Z-score standardization of continuous features (such as age and Ki-67 index) and normalization of categorical features (such as HR status and T / N stage) after one-hot encoding. Finally, all processed features were concatenated to form a unified standardized clinical feature vector. ,in This represents the total dimension of the features.

[0103] Step 1.2.5: Standardize Data Format

[0104] All modality data were converted to tensor format and stored indexed by "patient ID - modality type - feature dimension" to construct a standardized multi-omics dataset. .

[0105] The standardized multi-omics dataset is an integrated tensor format dataset containing preprocessed data from all modalities, stored indexed by "patient ID - modality type - feature dimension". The dataset contains a subset of pathological image modalities, including preprocessed digitized pathological images, lesion masks, and extracted pathological feature data. : A subset of proteomic modalities in the dataset, containing standardized serum protein expression data for 1034 proteins (missing values ​​have been filled). : A subset of genomic modalities in the dataset, containing filtered, spatially clustered, and cellularly deconvolutioned spatial transcriptome sequencing data. The dataset contains a subset of clinical modalities, which includes standardized data for eight clinical baseline features (age, ECOG score, etc.).

[0106] The multi-omics deep fusion network module 120 includes: a single-modality deep sub-network group, comprising dedicated sub-networks corresponding to pathological images, proteomics, genomics, and clinical baseline data, each dedicated sub-network being used to extract deep features corresponding to each modality from the standardized multi-modal feature set; a cross-modal interaction fusion module, used to receive deep features from all modalities, allocate fusion weights by calculating the feature similarity between modalities, and perform redundancy filtering on features to generate cross-modal interaction features; and a global information fusion module, used to perform global modeling of the cross-modal interaction features based on an attention mechanism and output global attention features.

[0107] Step 2: Construct a multi-omics deep fusion network

[0108] Figure 3 This is a schematic diagram of the core architecture of a multi-omics deep fusion network according to an embodiment of the present invention, as shown below. Figure 3As shown, this network adopts a three-level architecture of "single-modal feature mining - cross-modal interaction coordination - Transformer global fusion", and the overall network is denoted as . The input is a multi-omics dataset. The global attention features are output. The specific construction steps are as follows:

[0109] Step 2.1: Construct a single-modal deep subnetwork

[0110] To address the structural characteristics and biological significance of data from each modality, a dedicated deep sub-network was designed. A high-order key feature was mined through an optimization mechanism. The specific steps are as follows:

[0111] Step 2.1.1: Constructing the pathology imaging subnetwork ( )

[0112] Pathology Imaging Group Sub-Network ( As a dedicated sub-network for processing pathological image data, this sub-network performs the following operations: extracting various initial pathological features, including morphological, texture, and intensity information, from the preprocessed lesion images; introducing a superpixel perturbation verification mechanism to verify the stability of the initial pathological features and selecting stable features to form a feature set; and using the fuzzy whale optimization algorithm to optimize the weight parameters of this dedicated sub-network to extract deep features from the pathological images. Specifically, the following steps are included:

[0113] Step 2.1.1.1: Feature Extraction

[0114] Using DenseNet-121 as the basic architecture, multi-dimensional pathological features are extracted from the preprocessed lesion images. The specific steps are as follows:

[0115] Step 2.1.1.1.1: Extract morphological features

[0116] Calculate the volume of the lesion area equivalent diameter sphericity There are 12 morphological parameters.

[0117] The pathology imaging subnetwork is a branch of the single-modal deep subnetwork. It is built based on the DenseNet-121 architecture and is used to mine high-order key features of pathology images. Volume of the pathological lesion region (unit: pixels), obtained by applying a lesion mask matrix. The summation of all elements gives the total number of pixels reflecting the lesion. The pixel row index of the pathological image (same as step 1.2.1.2) has a value range of [value range missing]. (H is the image height), used to locate the specific pixel position in the lesion mask matrix. The pixel column index of the pathological image (same as step 1.2.1.2) has a value range of [value range missing]. (where is the image width) is used to locate the specific pixel position in the lesion mask matrix. Coordinates in the lesion mask matrix The element value at the location can be 0 or 1, where 1 indicates that the pixel belongs to the lesion area and 0 indicates that it is not a lesion area. Equivalent diameter of the lesion region (unit: pixels), based on lesion volume. The calculated value is used to quantify the overall size of the lesion. Sph: sphericity of the lesion region, dimensionless, with a range of values... The closer the value is to 1, the closer the lesion shape is to a sphere, which is used to describe the regularity of the lesion's shape.

[0118] Step 2.1.1.1.2: Extract texture features

[0119] Contrast Con is calculated based on the Gray-Level Co-occurrence Matrix (GLCM). ,energy Correlation Corr Simultaneously, multi-scale texture features were extracted using wavelet transform, resulting in a total of 64 texture features. Con: Texture contrast calculated based on the gray-level co-occurrence matrix (GLCM), dimensionless; a larger value indicates a more obvious difference in image gray levels, reflecting the clarity of lesion texture. The total number of gray levels in a pathological image, i.e., the quantization level of the image pixel gray values ​​(e.g., 8-bit image). ), used for GLCM texture feature calculation. The first grayscale index in GLCM, with a value range of 1. , represents the previous gray level in the gray level pair. The second grayscale index in GLCM, with a value range of [value range missing]. , represents the next gray level in the gray-scale pair. Gray-scale pairs in GLCM The probability of occurrence, i.e., the frequency proportion of the grayscale pair appearing in the image texture, takes a value range of 1. Eng: Texture energy calculated based on GLCM, unitless; a higher value indicates a more uniform and regular image texture, reflecting the consistency of lesion texture. Corr: Texture correlation calculated based on GLCM, unitless, with a value range of... The closer the value is to 1, the stronger the linear correlation of the image grayscale, reflecting the regularity of the lesion texture. Grayscale value The mean of all values ​​containing grayscale values. The average pixel grayscale value is used for GLCM correlation calculation. Grayscale value The standard deviation of all values ​​containing grayscale values. The degree of dispersion of pixel grayscale is used for GLCM correlation calculation.

[0120] Step 2.1.1.1.3 Extract intensity features

[0121] Calculate the mean value of pixels in the lesion region. Standard deviation skewness Eight intensity statistics, etc.

[0122] : The normalized pixel mean of the lesion area, unitless, with a value range of 100%. This reflects the overall brightness level of the lesion area. Coordinates in pathological images The normalized pixel values ​​range from 1 to 10. This serves as the foundational data for calculating intensity characteristics. : Standard deviation of pixels after normalization of the lesion area, unitless, with a range of values ​​of 100%. This reflects the dispersion of pixel brightness in the lesion area. Skew: The normalized pixel skewness of the lesion area, dimensionless, with a value range of... Positive values ​​indicate a right-biased brightness distribution (more low-brightness pixels), while negative values ​​indicate a left-biased distribution (more high-brightness pixels), reflecting the asymmetry of the brightness distribution.

[0123] Step 2.1.1.2: Feature stability screening

[0124] Introducing a superpixel perturbation verification mechanism—dividing the lesion image into Superpixel blocks, randomly perturbed The location of the superpixel is determined, and the coefficient of variation of each feature is calculated after each perturbation. ,reserve The stable features form the initial feature set. Specifically, in this step, the mechanism simulates non-rigid deformation of the image by artificially changing the spatial position of local regions (superpixel blocks), thereby quantitatively evaluating the stability of the extracted pathological features under this perturbation.

[0125] The coefficient of variation of a feature is dimensionless and is used to assess the stability of a feature. It is calculated as the ratio of the feature's standard deviation to its mean. The standard deviation of a certain pathological feature in the superpixel perturbation experiment reflects the degree of fluctuation of the feature after perturbation. The mean value of a certain pathological feature in the superpixel perturbation experiment reflects the average level of that feature. The initial feature set of pathological images is a stable feature set retained after stability screening, with dimensions of [missing information]. . Initial feature set of pathological images The dimensions are fixed at 48, meaning that 48 stable pathological features are retained after screening. Initial feature set of pathological images The vector space representation of is that the feature set is a 48-dimensional real vector.

[0126] Step 2.1.1.3: Construct a network optimization mechanism

[0127] A fuzzy whale optimization algorithm (FWOA) was designed to optimize the weight parameters of the pathological imaging subnetwork. This addresses the problem that traditional optimization algorithms struggle to dynamically balance the exploration and development phases. For example... Figure 4 The diagram shown is a schematic representation of the workflow of the Fuzzy Whale Optimization Algorithm (FWOA) according to an embodiment of the present invention.

[0128] Step 2.1.1.3.1: Fuzzy System Design

[0129] A Mamdani-type fuzzy system is used, with the input being "normalized fitness". "Current exploration parameters" "Current Development Parameters" ":

[0130] Calculation: Loss based on model prediction (MSE loss), according to Normalization to This reflects the current optimization state of the model.

[0131] Fuzzy subset partitioning: Fuzzy rules adopt the form "IF (condition) THEN (conclusion)", where the condition part is the input variable. The fuzzy subset combinations are all divided into three fuzzy subsets: "Low", "Medium", and "High". The output variable is the parameter adjustment amount. The fuzzy subsets are divided into three fuzzy subsets: "NE (negative), ZE (zero), and PO (positive)".

[0132] Develop a fuzzy rule base: Construct 18 fuzzy rules, such as "If NFV is Low and..." If it is High, then "For NE", "If NFV is High and..." If it is Low, then for This enables adaptive decision-making for parameter adjustment. Specifically, the 18 fuzzy rules are as follows:

[0133] 1. If NFV is Low AND α is High THEN Δα is NE (Negative, negative adjustment) AND is ZE (Zero, zero adjustment).

[0134] 2. If NFV is Low AND α is Medium THEN Δα is NE AND is ZE.

[0135] 3. If NFV is Low AND α is Low THEN Δα is ZE AND is PO (Positive, positive adjustment).

[0136] 4. If NFV is Medium AND α is High THEN Δα is NE AND is ZE.

[0137] 5. If NFV is Medium AND α is Medium THEN Δα is ZE AND is ZE.

[0138] 6. If NFV is Medium AND α is Low THEN Δα is ZE AND is PO.

[0139] 7. If NFV is High AND α is High THEN Δα is ZE AND is NE.

[0140] 8. If NFV is High AND α is Medium THEN Δα is PO AND is ZE.

[0141] 9. If NFV is High AND α is Low THEN Δα is PO AND is ZE.

[0142] 10. If NFV is Low AND is High THEN Δα is ZE AND is NE.

[0143] 11. IF NFV is Low AND is Medium THEN Δα is ZE AND is NE.

[0144] 12. IF NFV is Low AND is Low THEN Δα is ZE AND is PO.

[0145] 13. IF NFV is Medium AND is High THEN Δα is ZE AND is NE.

[0146] 14. IF NFV is Medium AND is Medium THEN Δα is ZE AND is ZE.

[0147] 15. IF NFV is Medium AND is Low THEN Δα is ZE AND is PO.

[0148] 16. IF NFV is High AND is High THEN Δα is ZE AND is NE.

[0149] 17. IF NFV is High AND is Medium THEN Δα is ZE AND is ZE.

[0150] 18. If NFV is High AND is Low THEN Δα is PO AND is ZE.

[0151] The output (Δα, of the fuzzy inference system) The fuzzy subset needs to be converted into specific values ​​for updating the WOA parameters through a defuzzification process. In a preferred embodiment of the invention, the centroid method is used for defuzzification. The output variable Δα and The membership functions of the fuzzy subsets NE (negative), ZE (zero), and PO (positive) are defined using symmetric triangular functions, with their universes of discourse set to [-0.2, 0], [-0.1, 0.1], and [0, 0.2], respectively. Finally, the system calculates a precise adjustment value based on all activated rules and their membership degrees to update α and... .

[0152] FWOA: Fuzzy Whale Optimization Algorithm, is a network optimization mechanism that integrates a Mamdani-type fuzzy system with the Whale Optimization Algorithm (WOA) to optimize the weight parameters of the pathological imaging subnetwork during the dynamic equilibrium exploration and development phase. .

[0153] The set of weight parameters for the pathological imaging subnetwork, including the connection weights of each layer in the DenseNet-121 architecture, is optimized using FWOA to obtain the optimal weights. .

[0154] NFV: Normalized fitness, with a value range of [value missing]. Loss based on model prediction The value is obtained by normalization and is used to reflect the current optimization state of the model (the smaller the value, the better the model performance).

[0155] fit: Model prediction loss, calculated using mean squared error (MSE) loss, measures the difference between the prediction results of the pathological image sub-network and the true labels.

[0156] The minimum value of the model prediction loss during multiple iterations is used for the normalization calculation of NFV and serves as a reference for the lower bound of the loss.

[0157] The maximum value of the model's predicted loss during multiple iterations is used for the normalization calculation of NFV and serves as a reference for the upper bound of the loss.

[0158] : Core parameters of the WOA exploration phase, used to control the size of the search range. It entered the exploration phase. It then entered the development stage.

[0159] : The core parameter in the development phase of WOA, used to control the fineness of local search and help the model focus on the optimal region for parameter adjustment.

[0160] Exploration parameters The adjustment amount is the output variable of the Mamdani-type fuzzy system, taking values ​​of NE (negative), ZE (zero), or PO (positive), and is used for dynamic updates. .

[0161] Development parameters The adjustment amount is the output variable of the Mamdani-type fuzzy system, taking values ​​of NE (negative), ZE (zero), or PO (positive), and is used for dynamic updates. .

[0162] Step 2.1.1.3.2: Whale Optimization Iteration

[0163] Based on the output of the fuzzy system Update WOA's core parameters:

[0164] Exploration phase :pass reduce Expand the search scope to uncover the global optimum;

[0165] Development phase :pass optimization Focusing on specific areas for precise adjustments.

[0166] : The number of iterations of FWOA, a positive integer (starting from 1), marking the round in which the parameters are updated. : No. Exploration parameters at the next iteration The value is used as the basic parameter for the current iteration. : No. Exploration parameters at the next iteration Update value, via The calculations show that the scope of the exploration phase can be adjusted. : No. Development parameters during the next iteration The value is used as the basic parameter for the current iteration. : No. Development parameters during the next iteration Update value, via The calculations allow for fine-tuning during the development phase.

[0167] Step 2.1.1.3.3: Weight Update

[0168] The optimized parameters are incorporated into the sub-network training and updated via backpropagation. The final output is the deep features of the pathological image.

[0169] The optimal set of weight parameters for the pathological imaging subnetwork is the final weight obtained after FWOA optimization, which is used for feature extraction and prediction of the subnetwork. : Deep feature vectors of pathological images, with a dimension of 256, composed of pathological image subnetworks Input initial feature set and optimal weight The output contains high-level key features of the pathological image. The mapping function of the pathological imaging sub-network represents the sub-network's processing of input features (feature extraction, optimization calculation), with the input being the initial feature set. and weight parameters The output is deep features. . Initial feature set of pathological images, with dimensions of This is a set of stable pathological features retained after stability screening. : Deep feature vector of pathological images The vector space representation of is that the eigenvectors are 256-dimensional real vectors.

[0170] Step 2.1.2: Constructing a proteomics subnetwork ( )

[0171] Proteomics Student Network ( As a dedicated sub-network for processing proteomic data, it performs the following operations: Standardized protein expression features are processed using a random point-level masking strategy, and the masked features are reconstructed using a deep autoencoder to learn the intrinsic relationships between protein features; based on changes in protein expression levels before and after treatment, a core protein feature set related to treatment response is selected by combining statistical tests and protein-protein interaction network analysis; the core protein feature set is then used to train a prediction sub-network to extract deep proteomic features. Specifically, the following steps are included:

[0172] Step 2.1.2.1: Mask Feature Learning

[0173] A random point-level masking strategy was designed to mask the standardized protein features. Perform masking – by Mask rate randomly generates mask matrix (1 indicates preserved features, 0 indicates masked features), will Inputting a deep autoencoder forces the model to learn the intrinsic relationships between proteins. Random point-level masking is a feature learning technique in self-supervised learning. Its core lies in independently and randomly zeroing (masking) each dimension (i.e., "points") of the input feature vector with a preset probability, forcing the model to reconstruct the complete feature vector based on the unmasked features through a deep autoencoder. This allows the model to learn the global dependencies between features, rather than being limited to traditional continuous sequence mappings. For example... Figure 5 The diagram shown is a schematic diagram of the random point-level masking strategy and feature learning in an embodiment of the present invention.

[0174] Encoder: Employs a 3-layer fully connected network with LeakyReLU activation function to compress input features into a low-dimensional space and output hidden features. .

[0175] Decoder: Employs a 3-layer fully connected network symmetrical to the encoder, with the sigmoid activation function, to reconstruct mask features. .

[0176] The Proteomics Subnetwork, a branch of the unimodal deep subnetwork, is constructed through a process of "mask feature learning - differential protein screening - MLP training" and is used to mine high-order association features of proteomics data. The standardized protein feature matrix, with dimensions of ( (Number of samples), containing expression levels of 1034 serum proteins after Min-Max normalization (missing values ​​have been filled). : Protein feature masking matrix, with dimensions of 1034 (corresponding to 1034 protein indicators). Elements take values ​​of 0 or 1; 1 indicates that the protein feature is preserved, and 0 indicates that the protein feature is masked. The masking rate is fixed. . Element-wise multiplication (Hadamard product) is used to normalize protein features. With mask matrix Element-wise multiplication achieves the following: Masking of protein features. The hidden feature vector of the proteome encoder has a dimension of 128 and is obtained by compression encoding through a 3-layer fully connected network. It contains low-dimensional abstract information about protein features. The encoder's activation function is LeakyReLU, with a negative slope of 0.2, which introduces a non-linear transformation into the encoding process to enhance feature representation. The encoder's weight parameter matrix, whose dimensions match those of the input and output features, is used to perform a linear transformation on the masked protein features. The encoder's bias parameter vector, and the hidden features Dimensional consistency is used to adjust the offset of encoded features. : The reconstructed output matrix of masked protein features, with dimension . The dimensions are consistent with the original protein feature matrix, and the hidden features are determined by the decoder. It was rebuilt. The decoder's activation function uses the sigmoid algorithm, mapping the reconstructed output to... The interval is consistent with the range of standardized protein feature values. The decoder's weight parameter matrix, and the encoder weights Symmetrical design for hiding features Perform inverse transformation reconstruction. The bias parameter vector of the decoder is consistent with the original protein feature dimension and is used to adjust the feature offset after reconstruction. Hidden features The vector space representation of the hidden features is that they are 128-dimensional real vectors. Rebuild output The vector space representation is that the reconstructed features are 1034-dimensional real vectors (consistent with the dimensions of the original protein features).

[0177] Step 2.1.2.2: Differential protein screening

[0178] Calculate the change in protein expression levels before and after treatment. Differentially expressed proteins were screened using t-tests (significance level 0.05), and protein-protein interaction network (PPI) analysis was combined to retain differentially expressed proteins located in key pathways (such as PI3K-Akt and MAPK), forming a core protein feature set. Protein-protein interaction network analysis refers to constructing a network graph based on the connection information of physical or functional interactions between proteins in known biological databases (such as STRING and BioGRID). By analyzing the node centrality (such as degree centrality and betweenness centrality) of differentially expressed proteins in this network and their corresponding functional modules or pathways, proteins that are located at the network hubs or are closely related to key disease pathways can be screened from a systems biology perspective, thereby improving the biological significance and reliability of the screening features.

[0179] : No. Changes in the expression levels of serum proteins before and after treatment ( The range of values ​​is The change in protein expression is reflected by calculating the absolute difference between the expression level after treatment and the expression level before treatment. : No. Expression levels of serum proteins after 6 cycles of neoadjuvant therapy (normalized + missing values ​​imputed). : No. The expression levels of each serum protein at baseline before treatment (normalized + missing values ​​imputed). Core protein feature set, with dimensions of , which is the set of differentially expressed protein features selected through t-test (significance level 0.05) and PPI network analysis. Core protein feature set The dimensions are fixed at 64, meaning that 64 key differential protein features are retained after screening. Core protein feature set The vector space representation of is that the feature set is a 64-dimensional real vector.

[0180] Step 2.1.2.3 Subnetwork Training

[0181] A 2-layer MLP (128 hidden layers) was constructed as a proteomics sub-network, using mean squared error (MSE) loss as the loss function, expressed as follows:

[0182]

[0183] The training loss function for the proteomics subnetwork uses mean squared error (MSE) loss to measure the difference between the subnetwork's predictions and the true labels. Total number of training samples, i.e., the number of patient samples participating in the proteomics sub-network training. Sample index, with a value range of: This is used to iterate through all training samples and calculate the total loss. : No. The true treatment response labels for each sample are consistent with the dimensions of the predicted results (including pCR status, sensitivity classification, and risk stratification). Proteomics sub-network for the first The predicted output for each sample is given by the core protein features of that sample. The output is the predicted treatment response result. The square of the Euclidean norm is used to calculate the squared error between the predicted output and the true label.

[0184] Using the AdamW optimizer (learning rate) Training with weight decay of 0.1 outputs deep proteomic features. .

[0185] : Deep proteome feature vector, with a dimension of 256, output by the trained proteomics sub-network, containing high-order correlation features of proteome data. Deep feature vectors of the proteome The vector space representation of is that the eigenvectors are 256-dimensional real vectors.

[0186] Step 2.1.3: Constructing a genomic subnetwork ( )

[0187] Genomics Subnetwork ( As a dedicated subnetwork for processing genomic data, it performs the following operations: Based on patient response to neoadjuvant therapy, it performs differential expression analysis on spatial transcriptome data to screen genes significantly associated with treatment response; it performs spatial co-expression analysis on the screened genes to identify gene modules with synergistic spatial expression patterns in tumor tissue; and it uses a convolutional neural network to learn the expression characteristics of these gene modules to extract deep genomic features. Specifically, it includes the following steps:

[0188] Step 2.1.3.1: Spatial Gene Feature Mining

[0189] Step 2.1.3.1.1: Differential gene screening

[0190] pCR group With non-pCR group Spatial transcriptome data were used for differential expression analysis using the MAST algorithm, with the following selection criteria: And after adjustment A preliminary set of differentially expressed genes was obtained.

[0191] The genomics subnetwork, a branch of the single-modality deep subnetwork, is constructed through a "spatial gene feature mining - 1D-CNN feature extraction" process to mine high-order pattern features related to genomic data and treatment response. The pCR status label for the sample takes a value of 1 or 0. 1 indicates that the patient has achieved pathological complete remission (ypT0 / isN0) after neoadjuvant therapy, and 0 indicates that it has not been achieved (non-pCR). It is used for grouping and differential gene screening. : Logarithmic change in gene expression levels, used to quantify the magnitude of the difference in gene expression between the pCR group and the non-pCR group. This indicates a significant difference of 2-fold or more in gene expression levels. The statistical significance P-value for differential gene screening, after multiple test correction (adjusted P-value), is set as the screening criterion. This is used to exclude random differences and retain genes that are truly associated with treatment response.

[0192] Step 2.1.3.1.2: Gene Spatial Colocalization Analysis

[0193] Gene spatial co-expression analysis specifically refers to calculating the correlation of expression vectors of different genes at all spatial sites (such as the Pearson correlation coefficient in this step) within the context of spatial transcriptomics data. Its purpose is to identify gene pairs or gene modules with highly coordinated spatial expression patterns in the tumor tissue microenvironment. These genes often participate in the same biological processes or reside in the same cell type / functional region, thus enabling the screening of gene features with stronger functional associations and better reflecting tumor heterogeneity from a spatial perspective.

[0194] Calculate the spatial co-expression coefficient between genes Preserve colocation coefficient A 0.6 gene module was used to remove isolated genes.

[0195] : No. The gene and the first The spatial co-expression coefficients of each gene, with values ​​ranging from [value range missing]. The closer the value is to 1, the more similar the expression patterns of the two genes at spatial sites, which is used to screen for functionally related gene modules. : No. The gene and the first The covariance of gene expression levels is used to measure the degree of synergy between changes in the expression of two genes and is a core indicator for calculating the co-expression coefficient. The first in the spatial transcriptome sequencing data matrix The expression vector of each gene has a dimension of . ( This represents the total number of spatial sites (with the same meaning as in step 1.1.3), which includes the expression level data of the gene at all spatial sites. The first in the spatial transcriptome sequencing data matrix The expression vector of each gene has a dimension of . ,and Dimensional consistency is used for co-expression analysis. : No. The standard deviation of the gene expression vector reflects the degree of dispersion of gene expression at different spatial sites and is used for the normalization calculation of co-expression coefficients. : No. The standard deviation of the gene expression vector, its effect on —Conformity, used to eliminate the influence of differences in gene expression levels on the co-expression coefficient. The index of the first gene, with a value range of [value range missing]. ( This represents the total number of genes detected in the spatial transcriptome (same meaning as in step 1.1.3), used to specify the genes involved in the co-expression analysis. The index of the second gene, with a value range of [value missing]. ,and Reproducible (for self-correlation validation) and used for paired co-expression analysis.

[0196] Step 2.1.3.1.3: Verify PCR

[0197] The differentially expressed genes were verified using quantitative PCR to ensure primer efficiency. Within this scope, 30 core genes strongly associated with breast cancer treatment response (such as S100A8, ERBB2, and MMP1) were ultimately retained to form a genomic feature set. .

[0198] The genome feature set, with a dimension of 30, is a set of 30 core gene expression level features retained after differential screening, colocalization analysis, and PCR validation, which are directly related to the treatment response of breast cancer. : Genome feature set The vector space representation is that the feature set is a 30-dimensional real vector, with each dimension corresponding to the expression level of a core gene.

[0199] Step 2.1.3.2: Constructing Subnetworks

[0200] A 1D-CNN architecture (kernel size 3, stride 1, number of kernels 64) is used to extract gene expression pattern features. Global average pooling (GAP) is then used to compress feature dimensions, outputting deep genome features. .

[0201] : Genome deep feature vector, with a dimension of 256, is obtained by extracting gene expression pattern features using a 1D-CNN architecture and then compressing them using global average pooling (GAP). It contains high-order abstract features of genomic data. : Deep feature vectors of the genome The vector space representation, i.e., the feature vector is a 256-dimensional real vector, is adapted to the dimensionality requirements of cross-modal fusion.

[0202] Step 2.1.4 Constructing the clinical baseline sub-network ( )

[0203] Clinical baseline data contains the patient's basic physiological state and macroscopic tumor characteristics, serving as important prior knowledge for treatment response. To fully leverage this information, a dedicated clinical baseline subnetwork (…) is constructed. The dedicated subnetwork for processing clinical baseline data includes: a feature interaction layer, which receives standardized clinical feature vectors. Through a fully connected layer (input dimension) Output dimension Perform nonlinear transformation to output enhanced features. Feature filtering layer, used to enhance features. For each dimension of the feature, the weights are calculated through the attention weight generation module. Then, a weighted summation is performed to output a refined set of clinical characteristics. ; and a deep mapping layer for refining clinical features. Inputting a two-layer MLP, mapping it to a high-dimensional space, and finally outputting clinical baseline deep features with a dimension of 256. Specifically, the steps include the following:

[0204] Step 2.1.4.1: Structuring and Enhanced Coding of Clinical Features

[0205] First, categorical features (such as HR status and T / N stage) are one-hot encoded, while continuous features (such as age and Ki-67 index) are standardized. Then, a clinical feature interaction layer is designed. This layer uses a shallow fully connected network to learn nonlinear combinations and higher-order interaction terms between features, simulating the thought process of a clinician making judgments based on multiple indicators. This layer outputs an enhanced clinical feature vector. .

[0206] Step 2.1.4.2: Construct a feature selection module based on attention weights

[0207] Not all clinical features contribute equally to the prediction. A feature-level attention mechanism is introduced to compute a learnable attention weight for each enhanced clinical feature. After normalization using the Softmax function, the features are weighted and summed to highlight key features and suppress noisy or irrelevant features, resulting in a refined representation of clinical features. .

[0208] Step 2.1.4.3: Subnetwork Construction and Output

[0209] Will The input is a multilayer perceptron (MLP), which undergoes further nonlinear transformation and dimension mapping to ultimately output deep clinical baseline features consistent with other modalities. The training of this subnetwork can be performed using a similar regression or classification loss as other subnetworks, either through pre-training or end-to-end training.

[0210] Step 2.2: Construct a cross-modal interaction fusion module ( )

[0211] To address the issues of strong heterogeneity and excessive redundant information in multimodal features, a cross-modal interactive fusion module was designed. This paper proposes a method based on cross-modal attention (CMA) combined with sparse gating to achieve dynamic coordination and redundant filtering between modalities. The process includes: calculating the cosine similarity between deep features of any two different modalities; determining the importance weight of each modality during fusion based on the calculated cosine similarity between each modality and all other modalities; introducing learnable sparse gating to the deep features of each modality to filter out redundant features and outputting the filtering results; and weighted fusion of deep features from all modalities based on the importance weights and the filtering results of the sparse gating to generate cross-modal interaction features. The specific steps are as follows:

[0212] Step 2.2.1: Modal similarity calculation

[0213] Let the deep feature vectors of each mode be... ( For modal identification, ), calculate the cosine similarity between any two modal features:

[0214]

[0215] The cross-modal interaction fusion module adopts the process of "modal similarity calculation - sparse gating filtering - interactive feature generation". It solves the heterogeneity and redundancy problem of multimodal features through the cross-modal attention mechanism (CMA) and realizes dynamic coordination between modalities. : Deep feature vectors of each single modality, with dimension . These correspond to the deep features of pathological images. Deep proteomic features Deep features of the genome Clinical baseline deep features ;path: one of the modal identifiers, representing pathological imaging modalities, corresponding to deep features prot: one of the modality identifiers, representing proteomic modalities and corresponding to deep features. ;gen: one of the modality identifiers, representing genomic modality, corresponding to deep features clin: one of the modality identifiers, representing the clinical baseline modality, corresponding to deep features. . : The vector space representation of the deep feature vectors of each modality, that is, all single-modal deep features are 256-dimensional real vectors, ensuring the dimensionality consistency of cross-modal fusion. : No. The modality and the first The cosine similarity of deep features of each modality, with values ​​ranging from 1 to 2. The closer the value is to 1, the stronger the feature correlation between the two modes, which is used to measure the correlation between modes. : No. The modality and the first The vector dot product of deep features of each modality is used to calculate the numerator of cosine similarity, reflecting the degree of linear correlation between features. : No. The 2-norm (Euclidean norm) of each modality's deep feature vector is used for the normalization calculation of cosine similarity, eliminating the influence of feature vector length on similarity. : No. The 2-norm of the deep eigenvectors of each modality, acting on... Consistency is ensured, guaranteeing that the similarity value is within the range of... Interval. : Identifier of the first mode It is used to calculate the similarity between modes by pairing. : Identifier of the second modality ,and It is repeatable and can be used to comprehensively measure the correlation between modes.

[0216] Importance weights for each modality are calculated based on similarity. A larger weight indicates a stronger contribution of that mode to the prediction.

[0217] : No. The importance weights of each modality, with values ​​ranging from [value range missing]. The values ​​are obtained by normalization using the Softmax function. The larger the weight, the stronger the contribution of the mode to the prediction of treatment response. The exponential function is used to convert the sum of modal similarities into a non-negative value, providing a basis for weight calculation. : No. The sum of the similarities of a mode to all other modes reflects the overall correlation between that mode and the global modes. Modal traversal index ( ), used to traverse modes other than the current mode All other modalities besides. Modal normalization index ( This is used to iterate through all modalities and implement Softmax normalization of the weights.

[0218] Step 2.2.2: Sparse Gated Filtering

[0219] Design learnable gating vectors for each modality. ,when If the modal feature is deemed redundant, it is masked out.

[0220] : No. A sparse gate vector of modalities, with dimension 256, is output by the Sigmoid activation function, and its values ​​range from 1 to 2. It is used to filter non-redundant features. The Sigmoid activation function maps the gating calculation results to... Intervals enable probabilistic determination of feature redundancy. : Gating weight matrix, dimension 1 Used for deep modal features Perform linear transformations to learn the redundant decision rules for modal features. : Gating bias vector, with a dimension of 256, used to adjust the offset of the gating linear transformation and optimize the accuracy of redundancy determination.

[0221] Step 2.2.3: Interaction Feature Generation

[0222] By fusing the weighted features with the gated filtered features, cross-modal interaction features are obtained. :

[0223]

[0224] The cross-modal interaction feature vector, with a dimension of 256, is obtained by summing the weights of each modality, gating filtering, and integrating key non-redundant information from multiple modalities. The feature summation operation is performed on all modal m features to achieve the final interactive fusion of multimodal features.

[0225] Step 2.3: Constructing the deep fusion module ( )

[0226] Global information fusion module ( This method employs a Transformer architecture and utilizes a "positional encoding - multi-head self-attention - parallel prediction output" process to achieve global fusion of cross-modal features and accurate prediction of treatment responses. It addresses the problem of traditional fusion models struggling to capture long-distance dependencies. This includes: adding positional encoding to cross-modal interaction features to preserve their position and order information; and using a multi-head self-attention mechanism to perform global feature interaction and fusion on the encoded features, generating globally attention features. The specific steps are as follows:

[0227] Step 2.3.1: Location Encoding

[0228] For cross-modal interaction features Add sinusoidal positional encoding to preserve the temporal and modal order information of features:

[0229]

[0230] The cross-modal interaction feature vector, with a dimension of 256, serves as the input to this module and integrates key non-redundant information from pathological images, proteomics, and genomes. The even-numbered dimension values ​​of the positional encoding are calculated using a sine function and are used to add positional information to the even-numbered dimensions of the interactive features, while preserving modal order and feature temporal attributes. : Odd-numbered dimension values ​​of the location encoding are calculated using a cosine function, complementing the even-numbered dimension sine encoding and enhancing the discriminative power of location information. pos: Feature location index, with a value range of... (corresponding interaction features) (256 dimensions) are used to mark the position of each element in the feature vector. : Dimensional grouping index of positional encoding, with a value range of This is used to divide 256-dimensional features into 128 pairs of odd and even dimensions. Cross-modal interaction features The dimension is fixed at 256, which is the core parameter for positional encoding calculation, ensuring that the encoding matches the feature dimension. : The scaling factor for positional encoding, used to control the period of the sine and cosine functions, so that the encoded values ​​at different positions have distinguishability. : The sine function is used to calculate the positional encoding value in even-numbered dimensions, mapped to Interval. The cosine function is used to calculate the position encoding value in odd-numbered dimensions. It complements the output of the sine function and enhances the ability to express position information.

[0231] Step 2.3.2 Introduce a multi-head self-attention mechanism

[0232] Four attention heads are set up to split the encoded features into four parallel sub-features (each sub-feature has a dimension of 64). Attention to size The calculation is as follows:

[0233]

[0234]

[0235] Attention head index, with a value range of: (A total of 4 attention heads are set up) to mark the attention branches of parallel computation. : No. The output features of each attention head are 64-dimensional and are obtained through attention calculation on the encoded features, capturing local and global dependencies of the features. Softmax(): The Softmax activation function is used to normalize the attention score to... The interval is used to obtain the attention weight distribution among features. : No. The query vector of each attention head, with dimension [missing information]. It is obtained by multiplying the encoded features by the query weight matrix. : No. The key vector of each attention head, with dimensions of... The value is obtained by multiplying the encoded features by the key weight matrix and is used to calculate the attention score. : No. A vector of values ​​for each attention head, with dimensions of [missing information]. The value weight matrix is ​​obtained by multiplying the encoded features by the value weight matrix and is used to generate the attention output. The cross-modal interaction features with added location encoding still have a dimension of 256, preserving the original interaction feature information and location information. : No. A query weight matrix for each attention head, with dimensions [missing information]. This is used to map encoded features into query vectors. : No. The key weight matrix of each attention head has dimensions of . It is used to map encoded features into key vectors. : No. The weight matrix of each attention head has dimensions of . This is used to map encoded features into value vectors. The sub-feature dimension of a single attention head is fixed. This is used to scale the attention score to avoid the value being too large. The square root of the sub-feature dimension is used to normalize the attention score and alleviate the gradient vanishing problem of the Softmax function. : No. The matrix multiplication result of the query and key of each attention head, with dimension 1. This yields the raw attention scores between features.

[0236] The outputs of the four attention heads are concatenated to obtain the global attention feature Attn. .

[0237] Attn The concatenated features from the outputs of four attention heads, with dimensions of [missing information]. It integrates the different dependencies captured by multiple attention heads to form a global attention feature. Global attention features The vector space representation of the features is a 256-dimensional real vector, which is suitable for the input requirements of the subsequent prediction layer.

[0238] The prediction output module 130 generates prediction results for neoadjuvant therapy response based on the global attention features, including at least: pathological complete remission (pCR) status, treatment sensitivity classification, and risk stratification.

[0239] Step 3: Construct the prediction output module 130

[0240] Three parallel fully connected branches are designed, each independently outputting three key dimensions of treatment response (including pCR status, treatment sensitivity classification, and risk stratification) based on global attention features. The overall result is then used as the prediction of treatment response. The specific steps are as follows:

[0241] Step 3.1 Predict pCR status

[0242] A two-layer fully connected network (128 hidden layers) is used to output binary classification probabilities. This indicates that the patient achieved complete pathological remission after neoadjuvant therapy.

[0243] pCR state prediction results, binary classification probability distribution (range of values) ), This indicates that the patient has achieved pathological complete remission (ypT0 / isN0). This indicates that the target was not met. The weight matrix of the pCR state prediction branch, with dimensions of [missing information]. (Input dimension 256, output dimension 2, corresponding to binary classification). : The bias vector of the pCR state prediction branch, with a dimension of 2, used to adjust the offset of the prediction output.

[0244] Step 3.2: Treatment sensitivity classification

[0245] A two-layer fully connected network (128 hidden layers) is used to output the probability of three classifications. Attn The classification results include "sensitive", "moderately sensitive", and "insensitive".

[0246] Treatment sensitivity classification prediction results, three-category probability distribution (range of values) The classification results were "sensitive", "moderately sensitive", and "insensitive". The weight matrix of the treatment sensitivity classification branch, with dimension 1. (Input dimension 256, output dimension 3, corresponding to three categories). : The bias vector of the treatment sensitivity classification branch, with a dimension of 3, is used to adjust the bias of the predicted output of this branch.

[0247] Step 3.3: Risk Stratification

[0248] A two-layer fully connected network (128 hidden layers) is used to output binary classification probabilities. The stratification results include "high-response population" and "low-response population". Risk stratification prediction results, binary classification probability distribution (range of values) The stratification results were "high-response population" and "low-response population". The weight matrix of the risk stratification branch, with dimensions of (Input dimension 256, output dimension 2, corresponding to binary classification). : The bias vector of the risk stratification branch, with a dimension of 2, is used to adjust the bias of the predicted output of this branch.

[0249] Step 4: Model Training and Optimization

[0250] Furthermore, during training, the total loss function of the multi-omics deep fusion network module is constructed as a weighted sum of the cross-entropy loss function and the mean squared error (MSE) loss function to balance classification and regression performance, and an optimizer incorporating weight decay is used for parameter updates. In a preferred embodiment, the total loss function is:

[0251]

[0252] The total loss function of the model is a weighted sum of cross-entropy loss and MSE loss (each weighted at 0.5; it can be understood that the weight coefficients can be adjusted according to the actual task), which balances the probability distribution fitting of the classification task with the minimization of the numerical error of the regression task. Cross-entropy loss is used to measure the prediction results. With real labels Differences in probability distribution can be used to adapt to classification tasks. Mean squared error (MSE) loss is used to measure the prediction result. With real labels The numerical difference improves the regression accuracy of the prediction results.

[0253] The overall set of prediction results from the model, including The three dimensions are the predictive inputs for calculating total loss. : The set of true labels for the samples, and The dimensions correspond one-to-one, including the actual pCR status, treatment sensitivity classification, and risk stratification, and are the actual input for total loss calculation.

[0254] AdamW optimizer (initial learning rate) The weight decays by 0.1, and the early stopping strategy (if the validation set loss does not decrease for 5 consecutive rounds, training is stopped) ensures the model's generalization ability.

[0255] According to the embodiments of the present invention, a neoadjuvant therapy response prediction system for HER2-positive breast cancer firstly collects and preprocesses four types of data: pathological images, proteomics, genomics, and clinical baseline. Secondly, a dedicated sub-network is designed, incorporating a fuzzy whale optimization algorithm (FWOA), a random point-level masking strategy, and an attention filtering mechanism, to mine high-order deep features from each modality. Then, heterogeneous features are dynamically integrated through a cross-modal attention and sparse gating fusion module, and global modeling is performed via a Transformer architecture. Finally, predictions are output in three aspects: pCR status, treatment sensitivity classification, and risk stratification, through parallel branching. This effectively solves the problems of poor handling of missing values ​​in multi-omics data, low modality fusion efficiency, and insufficient feature mining. Through the above-mentioned deep fusion architecture, the system can more accurately mine the associations between biomarkers of different modalities, ultimately achieving better predictive performance than traditional methods, providing clinicians with an efficient, automated, and accurate decision support tool for developing individualized treatment plans.

[0256] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0257] It should also be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0258] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in the embodiments of this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed in the embodiments of this application.

Claims

1. A HER2-positive breast cancer neoadjuvant therapy response prediction system based on a multi-omics deep fusion network, characterized by, The method comprises the following steps: a data acquisition and preprocessing module is configured to acquire and preprocess multi-omics data of a patient, wherein the multi-omics data comprises pathological image data, proteomic data, genomic data and clinical baseline data, so as to generate a standardized multi-modal feature set; a multi-omics deep fusion network module comprises a single-modal deep sub-network group, wherein the single-modal deep sub-network group comprises a dedicated sub-network corresponding to each of the pathological image data, the proteomic data, the genomic data and the clinical baseline data, and each dedicated sub-network is configured to extract deep features corresponding to each modality from the standardized multi-modal feature set; a cross-modal interaction fusion module is configured to receive the deep features of all modalities, assign fusion weights by calculating the similarity of the features between modalities, and filter the features to generate cross-modal interaction features; and a global information fusion module is configured to model the cross-modal interaction features based on an attention mechanism and output global attention features; a prediction output module is configured to generate a prediction result for a neoadjuvant therapy response based on the global attention features, wherein the prediction result comprises at least a pathological complete response (pCR) state, a therapy sensitivity classification and a risk stratification.

2. The system of claim 1, wherein, In the data acquisition and preprocessing, the preprocessing of the pathological image data comprises the following steps: a semantic segmentation model is used to identify a tumor lesion area in a digitized pathological image, a lesion mask is generated, and the lesion area image is normalized based on the mask; a deep autoencoder based on iterative median update is used to fill in missing values in serum protein expression data and perform standardization processing; differential gene screening and gene space co-expression analysis are performed on spatial transcriptome data to identify a core gene set related to therapy response; data cleaning and standardization processing are performed on the clinical baseline data to form a standardized clinical feature vector.

3. The system of claim 1, wherein, In the single-modal deep sub-network group, the dedicated sub-network processing the pathological image data is configured to perform the following operations: a plurality of initial pathological features including morphological, textural and intensity information are extracted from the preprocessed lesion image; a superpixel perturbation verification mechanism is introduced to verify the stability of the initial pathological features, and stable features are selected to form a feature set; a fuzzy whale optimization algorithm is used to optimize the weight parameters of the dedicated sub-network to extract deep pathological image features.

4. The system of claim 1, wherein, In the single-modal deep sub-network group, the dedicated sub-network processing the proteomic data is configured to perform the following operations: a random point-level mask strategy is used to process the standardized protein expression features, and a deep autoencoder is used to reconstruct the masked features to learn the internal correlation between protein features; based on the changes in protein expression before and after treatment, statistical tests and protein interaction network analysis are combined to screen a core protein feature set related to therapy response; the core protein feature set is used to train a prediction sub-network to extract deep proteomic features.

5. The system of claim 1, wherein, In the single-modal deep sub-network group, the dedicated sub-network processing the genomic data is configured to perform the following operations: Based on the grouping of patients according to the response to neoadjuvant therapy, differential expression analysis is performed on the spatial transcriptome data to screen genes significantly related to treatment response; Spatial co-expression analysis is performed on the screened genes to identify gene modules with coordinated spatial expression patterns in tumor tissues; Convolutional neural networks are used to learn the expression characteristics of the gene modules to extract deep genomic features.

6. The system of claim 1, wherein, In the single-modal deep subnetwork group, the dedicated subnetwork for processing clinical baseline data includes: a feature interaction layer for performing nonlinear transformation on standardized clinical baseline features to generate enhanced features; a feature screening layer for weighting and screening the enhanced features through an attention mechanism to obtain refined features; and a deep mapping layer for mapping the refined features to a high-dimensional space to extract deep clinical baseline features.

7. The system of claim 1, wherein, The cross-modal interaction fusion module is configured to perform the following operations: calculate the cosine similarity between the deep features of any two different modalities; determine the importance weight of each modality when fused based on the calculated cosine similarity of each modality with all other modalities; introduce a learnable sparse gating for the deep features of each modality to filter out redundant feature information, and output the filtering result; weight and fuse the deep features of all modalities according to the importance weight and the filtering result of the sparse gating, to generate cross-modal interaction features.

8. The system of claim 1, wherein, The global information fusion module adopts a Transformer architecture and is configured to perform the following operations: add position encoding to the cross-modal interaction features to preserve the position and order information of the features; use a multi-head self-attention mechanism to perform global feature interaction and fusion on the encoded features to generate global attention features.

9. The system of claim 1, wherein, The prediction output module includes at least three parallel branches that independently generate prediction results of pathological complete remission (pCR) status, treatment sensitivity classification, and risk stratification based on the global attention features.

10. The system of claim 1, wherein, During the training process of the multi-omics deep fusion network module, the total loss function is constructed as the weighted sum of cross-entropy loss function and mean square error loss function, and the optimizer combined with weight decay is used for parameter update.

Images