A pathological image classification method based on dyeing perception intelligent feature modeling

Abstract

The application discloses a kind of pathological image classification methods based on dyeing perception intelligent feature modeling, belong to pathological image classification field, this method includes collecting the multi-staining full section image group data of different objects;Multi-staining full section image group data includes H&E dyeing image and several immunohistochemical dyeing images;Each multi-staining full section image group data is respectively preprocessed, obtains the several equal-size image blocks of each object;Based on the image block of each object, graph is constructed, and the multi-modal heterogeneous pathology graph of each object is obtained;With the multi-modal heterogeneous pathology graph of each object as training data, train pathological image classification model;The pathological image classification model of training completion is used to classify the pathological image to be measured.The application solves the problems of the existing method in processing digital pathology graph data, such as node category characteristics, edge information utilization, cross-staining interaction and non-perfect data processing.

Description

Technical Field

[0001] This invention belongs to the field of pathological image classification, and in particular relates to a pathological image classification method based on staining-sensory intelligent feature modeling. Background Technology

[0002] With the development of digital pathology, the analysis of whole-slide images (WSI) using deep learning has become an important means of assisting diagnosis. Especially in multiple staining scenarios such as immunohistochemistry (IHC), by combining the analysis of H&E staining with multiple IHC markers (such as WT-1, Vim, ER, etc.), it is possible to more comprehensively reveal the tumor microenvironment and immune infiltration patterns.

[0003] Graph neural networks, as a powerful tool for processing non-Euclidean spatial data (such as graph-structured data), have demonstrated enormous potential in fields such as biomedical image analysis, social networks, and recommender systems. In the field of digital pathology, the entire pathological slide (WSI) is typically modeled as a large graph structure, where tissue patches are considered nodes, and the spatial or semantic relationships between patches are modeled as edges. By learning and analyzing this graph, key tasks such as histological feature classification and abnormal region identification of pathological images can be achieved.

[0004] Currently, graph neural network (GNN)-based methods have become the mainstream paradigm for multi-stain pathological image analysis. This approach typically employs the following workflow: First, whole-slide images (WSI) are segmented into tissue regions, removing background areas and preserving tissue regions at a specific magnification. Second, the tissue regions are seamlessly divided into fixed-size patches, with each patch defined as a node in the graph. Subsequently, a pre-trained deep learning model (such as UNI) is used to extract the visual features of the patches as node features. Finally, a graph topology is typically constructed based on Euclidean distance (KNN) in the feature space or geometric adjacency relationships in the physical space, and graph convolutional layers are used to aggregate local neighborhood features for classification prediction. However, while this general workflow performs well in single-modality scenarios, existing GNN models still have many limitations when processing complex clinical multi-modal (multi-stain) data.

[0005] On the one hand, existing multi-stain analysis models typically use a shared feature extractor to process image patches of all staining types. However, different staining modalities (e.g., nuclear / cytoplasmic staining in H&E versus specific antigen-labeled staining in IHC) exhibit significant domain shifts in color, texture, and intensity distribution. Existing methods often ignore these essential differences between modalities, directly splicing or mixing features from different sources, making it difficult for the model to distinguish the biological origin of features, resulting in confusion and inaccuracy in feature representation. Furthermore, edges in graphs often carry rich semantic information (e.g., in pathological images, edges can indicate whether two patches belong to the same tissue type, are spatially adjacent, or have similar morphologies). Existing techniques either completely ignore edge attributes or only use simple, coupled embedding methods (e.g., splicing all edge attributes and embedding them uniformly), failing to finely decouple and model multi-source, heterogeneous edge information. This coarse processing makes it difficult for the model to effectively capture complex relationship patterns between nodes, thus losing important structural information. On the other hand, existing multimodal graph network architectures typically employ a "local aggregation followed by global fusion" late fusion strategy. In this architecture, after the model extracts features from its various deep networks, it only integrates multimodal information at the very end through a large attention or pooling layer. This architecture results in intermediate layer nodes lacking a global perspective, failing to perceive the contextual information of other staining modalities in the early stages of feature extraction (e.g., feature extraction of tumor regions should be guided by information from the surrounding immune microenvironment, but in the current architecture, this interaction occurs too late), and the terminal fusion module often has redundant parameters and low computational efficiency.

[0006] In real-world clinical settings, patients are often unable to complete a full set of staining procedures due to limitations in sample quality or testing costs (e.g., missing ER or Ki67 slides). However, existing supervised contrastive learning or multimodal fusion classifiers typically have fixed input topologies, requiring strict alignment of input modalities in the feature dimension. Faced with missing data, existing methods cannot dynamically adapt to the input dimension, often resorting to discarding the sample or forcibly filling in features using zero-padding / mean-padding techniques. This not only wastes valuable incomplete clinical data, but the padded pseudo-values ​​also introduce severe noise perturbation into the feature space, compromising the authenticity of the feature distribution and ultimately leading to a significant decline in the model's classification performance on incomplete datasets.

[0007] In summary, existing graph neural network models have significant shortcomings when processing graph data with multi-source attributes, complex structures, and rich semantics (such as digital pathology images), particularly in terms of node type features, utilization of edge information, cross-stain interactions, and processing of incomplete data. Therefore, there is an urgent need in this field for a novel graph neural network solution that can comprehensively address these issues to achieve more accurate, robust, and interpretable graph representation learning and classification. Summary of the Invention

[0008] To address the aforementioned shortcomings in existing technologies, this invention provides a pathological image classification method based on staining-aware intelligent feature modeling. This method solves the problems of existing methods in processing digital pathological image data, particularly in terms of node type features, utilization of edge information, cross-stain interaction, and processing of incomplete data.

[0009] To achieve the aforementioned objectives, the technical solution adopted by this invention is: a pathological image classification method based on staining-sensing intelligent feature modeling, comprising: A dataset of multi-stain whole-section pathological images was collected; the dataset included multi-stain whole-section image sets of different subjects; the multi-stain whole-section image sets included H&E staining images and several immunohistochemical staining images. Preprocess the data of each multi-stained whole-section image group to obtain several image blocks of equal size corresponding to each object; Based on the image patches of each object, a map is constructed to obtain a multimodal heterogeneous pathological map of each object; A pathological image classification model is trained using multimodal heterogeneous pathological images of various objects as training data; the pathological image classification model includes: The edge attribute embedding module is used to embed edge attributes of multi-source decoupling multimodal heterogeneous pathological maps to obtain enhanced edge features; The L-layer feature modulation modules are all used to enhance node features based on metadata-based feature modulation mechanisms to obtain modulated node features; the input data of the first layer feature modulation module is a multimodal heterogeneous pathological image; the input data of the 2nd to Lth layer feature modulation modules is the final node features output by the previous deep hierarchical feature extraction network. The L-layer deep hierarchical feature extraction network is used to obtain the global context features across slices corresponding to each coloring type based on the enhanced edge features and the modulated node features output by the feature modulation module of the corresponding layer; the global context features across slices corresponding to each coloring type are broadcast back to the nodes with the corresponding coloring attribute labels, and combined with the gating coefficients to obtain the final node features. The MLP classification head is used to obtain a graph-level feature representation by splicing the global context features across slices corresponding to each staining type output by the L-layer deep hierarchical feature extraction network, and to obtain the pathological classification prediction result. The trained pathological image classification model is used to classify the pathological images to be tested.

[0010] Furthermore, obtaining several image blocks of equal size corresponding to each object specifically involves: For the multi-stained whole-section image data of each object, the following operations were performed: For each stained image in the multi-stained whole-section image group of the current object, the following operations are performed: the stained image is converted to the HSV color space to obtain the HSV stained image; a binary mask of the HSV stained image is generated using the Otsu method, and the background area is removed based on the binary mask of the HSV stained image to obtain the effective area containing only biological tissue; at a preset magnification, the effective area is seamlessly cut into several non-overlapping image blocks of fixed size; By integrating the image blocks corresponding to each stained image in the multi-stained whole-slice image group data of the current object, several image blocks of the same size corresponding to the current object are obtained.

[0011] Furthermore, the process of obtaining the multimodal heterogeneous pathological images of each object specifically involves: For each object's several image blocks, perform the following operations respectively: Each image patch of the current object is input into the pre-trained UNI deep learning pathology basic model, and the output of the penultimate layer of the UNI deep learning pathology basic model is extracted as the initial visual feature vector corresponding to each image patch. Determine the coloring attribute labels for each image patch; Using image patches as nodes, connections are established based on the spatial location of each node to determine the set of physical spatial adjacency edges. Specifically: when the... The node and the first Nodes originating from the same multi-stained whole-slice image and whose Euclidean distance on the 2D grid is less than a preset threshold, or the nodes originating from the same multi-stained whole-slice image and whose Euclidean distance on the 2D grid is less than a preset threshold, or the ... nodes origin The node and the first The nodes come from different multi-stained whole-slice images but their corresponding coordinates are completely consistent. The node and the first The first node establishes a connection; otherwise, the second node... The node and the first No connection is established between any nodes; Using image patches as nodes, connections are established based on the initial visual feature vectors corresponding to each node to determine the semantic feature nearest neighbor set. Specifically, for each node, the nearest feature distance is selected. A connection is established between the nodes; the feature distance is the cosine similarity or Euclidean distance calculated based on the initial visual feature vectors of each node; Edges are determined using the set of adjacent edges in physical space and the set of nearest edges in semantic features, thus obtaining the graph structure; Define edge attributes and generate position encoding vectors for each node based on random walks. Introduce the position encoding vectors, initial visual feature vectors, and staining attribute labels of each node into the graph structure to obtain the multimodal heterogeneous pathological map of the current object.

[0012] Furthermore, the expression for the edge attribute is:

[0013] in, For the first The node and the first The edge attributes of the connecting edges of each node; For coloring associated attributes, when the first The coloring attribute labels of each node With the The coloring attribute labels of each node If the values ​​are the same, a value of 0 indicates a homogeneous connection; otherwise, a value of 1 indicates a heterogeneous connection. For physical space adjacency attributes, when the first The node and the first If the connecting edge of a node belongs to the set of adjacent edges in the physical space, the value is 1; otherwise, the value is 0. For semantic features, the nearest neighbor attribute, when the... The node and the first If the connecting edges of a node belong to the semantic feature nearest neighbor edge set, the value is 1; otherwise, the value is 0. The expression for the position encoding vector of each node is:

[0014]

[0015] in, For the first The position encoding vector of each node; For the first Random walk of nodes The probability of self-returning after a step; For random walk The probability distribution matrix after the step.

[0016] Furthermore, the edge attribute embedding module includes: The attribute separation unit is used to separate the edge attributes of multimodal heterogeneous pathology maps into corresponding staining-related attributes. Staining interaction components Corresponding physical space adjacency attribute Region adjacency components and corresponding semantic features nearest neighbor attributes Feature nearest neighbor components ; The first embedding layer is used to capture the coloring interaction components. The latent semantics are used to obtain continuous low-dimensional vectors corresponding to the coloring interactions. ; The second embedding layer is used to encode region adjacency components. Connectivity information in physical space yields a continuous low-dimensional vector corresponding to that physical space. ; The third embedding layer is used to encode the feature nearest neighbor components. Connectivity information from feature similarity yields a continuous low-dimensional vector in the feature space. ; The edge feature fusion unit is used to convert the continuous low-dimensional vectors corresponding to the coloring interactions. Continuous low-dimensional vectors corresponding to physical space The continuous low-dimensional vectors corresponding to the feature space Concatenate along the feature channel dimension to generate enhanced edge features. .

[0017] Furthermore, each of the aforementioned feature modulation modules includes: The node embedding layer is used to map the node coloring attributes of the input data to coloring embedding vectors corresponding to each coloring type, based on the coloring attribute labels. The linear projection layer is used to perform linear projection on the color embedding vectors corresponding to each color type to obtain the projection parameters corresponding to each color type. The parameter slicing layer is used to divide the projection parameters corresponding to each coloring type into a scaling factor vector and a translation factor vector on the last dimension respectively. The affine transformation layer is used to perform element-wise affine transformations on the current node features based on the coloring attribute labels, according to the scaling factor vector and translation factor vector of each coloring type, to obtain the modulated node features:

[0018] in, For the first The modulated node features output by the layer feature modulation module; For the first The scaling factor vector of the coloring type corresponding to the coloring attribute label of the current node in the layer feature modulation module; For Hadamah accumulation; Features of the current node; For the first The translation factor vector of the coloring type corresponding to the coloring attribute label of the current node in the layer feature modulation module.

[0019] Furthermore, each of the aforementioned deep hierarchical feature extraction networks includes: The graph attention aggregation unit is used to concatenate the position encoding vector of each node with the modulated node features output by the feature modulation module of the corresponding level along the channel dimension to obtain the enhanced node input matrix; based on the enhanced node input matrix and enhanced edge features, the attention weights of each node and its neighboring nodes are calculated; and neighborhood information is aggregated according to the attention weights of each node and its neighboring nodes to obtain the updated node features; the expression for the attention weights of each node and its neighboring nodes is as follows:

[0020] in, For the first The node and the first Attention weights of each neighboring node; Use the Softmax activation function; For leaky modified linear unit activation functions; This is a vector of learnable attention mechanism parameters; The feature projection matrix to be learned; To enhance the number of nodes in the input matrix The feature vector of each node; To enhance the number of nodes in the input matrix Feature vectors of neighboring nodes; For the first The node and the first The enhanced edge features between each neighbor node are taken from the overall enhanced edge features. ; For splicing; For transpose; The coloring-aware attention pooling and prototype extraction unit is used to calculate the importance attention score of each node using self-attention-based graph pooling; and retain the K nodes with the highest importance attention scores to obtain a sparse graph structure; the updated node features of each node in the sparse graph structure are multiplied by the normalized importance attention score of the node to obtain the weighted features of each node in the sparse graph structure; the nodes in the sparse graph structure are grouped according to the coloring attribute label to obtain node groups under each coloring attribute; for each node group under each coloring attribute, the weighted features of each node in the group are aggregated to obtain the coloring prototype vector of the overall state of each coloring attribute at the current level.

[0021] in, For the first Deep Hierarchical Feature Extraction Network The coloring prototype vector of the overall state of each coloring attribute; For splicing; For the first The group of coloring attribute nodes corresponding to each coloring attribute The number of nodes; For node indexing; For coloring attribute node groups No. The weighted features of each node; Coloring attribute node group The maximum value of the weighted feature of the middle node; The HSIT cross-coloring global interaction unit is used to arrange the coloring prototype vectors of the overall state of each coloring attribute in the current level into a coloring prototype sequence; the coloring prototype sequence is enhanced by position encoding using the coloring embedding vectors corresponding to each coloring type output by the feature modulation module of the corresponding level, resulting in a coloring prototype enhanced sequence; the Transformer encoder is used to extract features from the coloring prototype enhanced sequence to obtain the cross-slice global context features corresponding to each coloring type. The adaptive gating feedback mechanism unit broadcasts the global context features across slices corresponding to each staining type back to the nodes with the corresponding staining attribute labels in the sparse graph structure, and updates the node features by combining the gating coefficients to obtain the final node features:

[0022] in, For the sparse graph structure, the first The final node characteristics of each node; To integrate the gating coefficients; For Hadamah accumulation; For the sparse graph structure, the first The updated node characteristics of each node; This is a feature projection function used to map and align the dimensions of global context features to a coordinate system. Same dimensions; For the sparse graph structure, the first Each node is determined by its assigned coloring attribute label. Received global context features across slices corresponding to the staining type; Sigmoid is the activation function; This is the learnable weight matrix of the fully connected layer inside the gated unit; For splicing.

[0023] Furthermore, the expression for the pathological classification prediction result is as follows:

[0024]

[0025]

[0026]

[0027] in, For pathological classification prediction results; This is the predicted final disease classification result; Index for disease categories; For input multimodal heterogeneous pathological images; The first unnormalized pathological prediction score vector Predicted component values ​​for each category; The total number of categories; The first unnormalized pathological prediction score vector Predicted component values ​​for each category; The unnormalized pathological prediction score; This is the learnable weight matrix for the second fully connected classification network; The hidden layer classification feature vector is obtained after the graph-level feature representation undergoes dimensionality reduction and nonlinear transformation in the first fully connected network layer. This is the learnable bias vector for the second fully connected classification network; For random inactivation; To modify the activation function of the linear unit; This is the learnable weight matrix of the first fully connected classification network; For layer normalization; This represents graph-level features. This is the learnable bias vector for the first layer of the fully connected classification network; For splicing; The global context features across slices corresponding to each staining type are extracted from the Lth layer deep hierarchical feature extraction network output.

[0028] Furthermore, the loss function of the pathological image classification model is:

[0029]

[0030]

[0031] in, The loss function for the pathological image classification model; Cross-entropy loss; This is an unnormalized pathological prediction score vector; The actual disease classification labels corresponding to the current multimodal heterogeneous pathology diagram; Weights for conditional contrastive loss; For conditional contrast loss; For the first A set of staining types present in a multimodal heterogeneous pathological map; for Projection features after projection head mapping and L2 normalization; To extract the first layer of the L-layer deep hierarchical feature extraction network output The deep feature representation is obtained by concatenating the global context features across slices corresponding to each staining type; The positive sample prototype vector retrieved from the structured 2D prototype mesh library, i.e., the coloring type index is... Furthermore, the disease category index is the true label of the current multimodal heterogeneous pathology map. The ideal feature center; Temperature coefficient; Index for color type; Index for disease categories; The total number of categories; The prototype vector retrieved from the structured 2D prototype mesh library, i.e., the coloring type index is... And the disease category index is Same-staining characteristic centers; For the characteristic projection function; For L2 normalization; each mesh element in the structured two-dimensional prototype mesh library Indicates the first The staining type in the first Ideal feature centers for each disease category; This is the color type index corresponding to the total set of color types.

[0032] The beneficial effects of this invention are as follows: In scenarios involving different staining modalities (such as H&E and IHC), an adaptive feature affine modulation mechanism based on metadata (i.e., dynamically generating scaling and translation parameters to recalibrate features according to modality type) and multi-source decoupled edge attribute embedding technology are employed to achieve adaptive alignment of heterogeneous features and refined modeling of complex tissue associations, thereby improving the model's accuracy in perceiving multimodal data. A hierarchical cross-staining interaction and gated feedback mechanism is designed (i.e., extracting the global staining prototype using Transformer at each layer and injecting contextual information back into local nodes through gating units), breaking the limitations of traditional graph neural networks that only perform local aggregation or rely on inefficient end-point fusion, thus meeting the needs of efficient interaction in deep semantics. This design constructs a "local-global-local" feature update closed loop, enabling deep node features to naturally integrate the multimodal semantics of the entire graph, enhancing the model's ability to discriminate the tissue microenvironment. Finally, considering the incomplete data problem caused by missing slides in clinical applications, and the need to maintain the stability of model training under incomplete modalities, a prototype-guided learning based on two-dimensional structured grids combined with a dynamic masking strategy was used to achieve "on-demand" supervised matching of samples containing only partial stained slides, avoiding noise interference caused by data imputation and maximizing the utilization value of incomplete clinical data. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the method framework of the present invention.

[0034] Figure 2 This is a diagram of the HSIT cross-stain global interaction framework of the present invention.

[0035] Figure 3 This is a schematic diagram of the robust conditionalization prototype learning process of the dynamic masking strategy of the present invention.

[0036] Figure 4 This is a flowchart of the method of the present invention. Detailed Implementation

[0037] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0038] like Figure 1 and Figure 4 As shown, in one embodiment of the present invention, it includes: A dataset of multi-stain whole-section pathological images was collected; the dataset included multi-stain whole-section image sets of different subjects; the multi-stain whole-section image sets included H&E staining images and several immunohistochemical staining images. Preprocess the data of each multi-stained whole-section image group to obtain several image blocks of equal size corresponding to each object; Based on the image patches of each object, a map is constructed to obtain a multimodal heterogeneous pathological map of each object; A pathological image classification model is trained using multimodal heterogeneous pathological images of various objects as training data; the pathological image classification model includes: The edge attribute embedding module is used to embed edge attributes of multi-source decoupling multimodal heterogeneous pathological maps to obtain enhanced edge features; The L-layer feature modulation modules are all used to enhance node features based on metadata-based feature modulation mechanisms to obtain modulated node features; the input data of the first layer feature modulation module is a multimodal heterogeneous pathological image; the input data of the 2nd to Lth layer feature modulation modules is the final node features output by the previous deep hierarchical feature extraction network. The L-layer deep hierarchical feature extraction network is used to obtain the global context features across slices corresponding to each coloring type based on the enhanced edge features and the modulated node features output by the feature modulation module of the corresponding layer; the global context features across slices corresponding to each coloring type are broadcast back to the nodes with the corresponding coloring attribute labels, and combined with the gating coefficients to obtain the final node features. The MLP classification head is used to obtain a graph-level feature representation by splicing the global context features across slices corresponding to each staining type output by the L-layer deep hierarchical feature extraction network, and to obtain the pathological classification prediction result. The trained pathological image classification model is used to classify the pathological images to be tested.

[0039] The overall analysis process of this invention is mainly divided into five closely linked processing stages: Phase 1: Data Preprocessing and Multi-Source Graph Construction. In this phase, the transformation from raw images to graph-structured data is completed. First, the input multi-stained whole-slice (WSI) images are segmented to remove background regions, and the regions of interest are cut into fixed-size image patches. Second, a pre-trained feature extractor is used to extract the initial visual feature vector for each image patch. Next, a graph structure is constructed based on the image patches, defining multi-source edge information, including region adjacency edges (RAG) connecting physically adjacent nodes, K-nearest neighbor edges (KNN) connecting similar nodes in the feature space, and stained edges identifying the slice source of a node. Finally, a random walk operation is performed to generate positional codes to initialize the graph data.

[0040] Phase Two: Preparation Before Inputting the Model – Feature Modulation and Edge Embedding. In this phase, the initialized graph data undergoes feature space alignment and enrichment. For node features, metadata-based linear feature modulation is performed. Staining type IDs are used to generate stain-modality-specific scaling and translation factors. Affine transformations and recalibration are applied to the initial features to eliminate domain offsets between different stain slices. For edge features, stain associations, spatial adjacency, and feature similarity indices are mapped to continuous vectors through three independent embedding layers and then fused to generate edge features rich in multiple semantics.

[0041] The third stage: Hierarchical deep feature extraction and fusion interaction stage. In this stage, a graph neural network backbone is used for deep feature learning. After each graph convolution operation, a hierarchical cross-coloring interaction and gating feedback mechanism (HSI-Gating) is embedded. This mechanism first clusters nodes into coloring prototypes according to coloring type, then uses a Transformer encoder to capture long-distance global dependencies across slices, and finally uses a Sigmoid gating unit to dynamically "feed back" global context information and inject it into local node features, achieving closed-loop feature update.

[0042] Phase Four: Model Training and Supervision. This phase focuses on addressing the problem of training with incomplete data. A robust Conditional SPC learning strategy is introduced, maintaining a structured two-dimensional grid prototype library. During training, a dynamic masking mechanism automatically identifies the actual set of colored modalities in the input samples, retrieving only the positive and negative prototypes corresponding to existing modalities for contrastive loss calculation, automatically ignoring missing modalities, thus achieving effective supervision of incomplete data.

[0043] Fifth stage: Classification and prediction stage. In this stage, the graph node features after deep interaction are aggregated into a graph-level representation vector through a readout function, and then input into a fully connected classification head to output the final pathological image classification result.

[0044] (I) Data Preprocessing and Multi-Source Graph Construction The main task of this stage is to convert clinically acquired multimodal pathological images into computer-processable graph-structured data rich in semantic information. Specific implementation steps include whole-slice image preprocessing, initial feature extraction, node attribute definition, multi-source heterogeneous edge construction, and graph initialization based on random walks.

[0045] Step 1.1: Multimodal slice acquisition and tissue region extraction First, acquire a set of multi-stain whole-slide images (WSI) of the same object. This set of images typically consists of one H&E (hematoxylin-eosin) stained slide and multiple immunohistochemical (IHC) stained slides (e.g., ER, PR, WT-1, Ki67, etc.) to reflect different cell morphologies and the expression distribution of specific antigens.

[0046] Because WSI images are enormous (typically reaching billion pixels) and contain a large amount of blank background with no information, tissue segmentation is performed first. Specifically, the WSI image is converted to the HSV color space, and a binary mask is generated using Otsu's method or a preset saturation threshold to remove the background area, retaining only the effective area containing biological tissue. Subsequently, at a specific magnification (e.g., 20×), the effective tissue area is seamlessly segmented into a series of fixed-size segments (e.g., ...). Non-overlapping image patches (pixels).

[0047] The specific steps for obtaining several image blocks of equal size corresponding to each object are as follows: For the multi-stained whole-section image data of each object, the following operations were performed: For each stained image in the multi-stained whole-section image group of the current object, the following operations are performed: the stained image is converted to the HSV color space to obtain the HSV stained image; a binary mask of the HSV stained image is generated using the Otsu method, and the background area is removed based on the binary mask of the HSV stained image to obtain the effective area containing only biological tissue; at a preset magnification, the effective area is seamlessly cut into several non-overlapping image blocks of fixed size; By integrating the image blocks corresponding to each stained image in the multi-stained whole-slice image group data of the current object, several image blocks of the same size corresponding to the current object are obtained.

[0048] Step 1.2: Initial Feature Extraction Based on Pre-trained Model To convert image patches into mathematical vectors, this invention employs a pre-trained deep learning pathology fundamental model (UNI model) as a feature extractor. However, the directly extracted features suffer from severe domain bias, therefore feature recalibration is required subsequently through the feature modulation module of this invention.

[0049] Input all the image patches obtained in step 1.1 into the feature extractor, and extract the output of its penultimate layer (i.e., the features input to the UNI model classification head) as the initial visual feature vector. (See step 1.) The collection of slides from each patient contains Each image patch, after passing through the extractor, yields a feature matrix. ,in The feature dimension is (e.g., 1024 dimensions). In this case, each image patch is treated as a node in the graph.

[0050] Step 1.3: Node Attribute Tagging and Color ID Assignment Define the biological origin attribute explicitly for each node. This will automatically be applied to each node based on the slice name. Extract the corresponding coloring type and use it as a discrete coloring attribute label. ,in It represents the total number of coloring types contained in the dataset.

[0051] For example, unstained markers are encoded as index 0, and H&E-stained markers are encoded as index 1, etc.

[0052] This attribute It is not directly used in graph neural network operations, but is stored as metadata in node information for subsequent modulation features and differentiation of coloring types.

[0053] Step 1.4: Construction of ternary heterogeneous edges Traditional methods often confuse spatial adjacency with feature similarity. This invention designs a multi-source decoupling edge construction strategy to construct a graph. For any edge in the graph It not only records the connection relationships, but also constructs a three-dimensional attribute vector. The specific construction logic is as follows: 1. Coloring-related attributes: Construction logic: Explicitly determine whether the two endpoints of an edge belong to the same coloring mode.

[0054] Attribute encoding: settings This is the ID for the coloring interaction type. For example, if... ,but Represents homogeneous connectivity; if ,but This represents heterogeneous connections.

[0055] 2. Physically adjacent edges: Construction logic: Based on the physical coordinates recorded in step 1.1 If two nodes and From the same slice (i.e.) If two nodes... and Sections from different staining (i.e.) If the coordinates are exactly the same, then a connection is established.

[0056] For this type of connection, set the edge attribute components. =Indicates spatial adjacency, otherwise 0.

[0057] 3. Semantic feature nearest neighbor edge: Construction logic: Based on the feature vectors extracted in step 1.2 Calculate the cosine similarity or Euclidean distance between all node pairs in the feature space. For each node... Select the feature closest to 1 node (usually) Establish connections. These connections are based on all sections from the same patient and are designed to capture potential biological homology between different stained sections (e.g., tumor regions in H&E and highly expressed regions in IHC may have similar texture features).

[0058] For this type of connection, set the edge attribute components. If the characteristics are similar, the value is 0; otherwise, it is 0.

[0059] 4. Final edge set generation: Final graph edge set It is the union of the RAG edge set and the KNN edge set: For each edge, its final attribute is a vector combining the above three components: .

[0060] This will serve as the input index for the three independent embedding layers in the subsequent graph neural network, ultimately becoming the edge features that enter the graph attention network.

[0061] Step 1.5: Initialization of graph position encoding based on random walk To address the complex non-Euclidean topological characteristics of pathological tissue diagrams, and to enhance the model's ability to perceive local subgraph structures, this step employs a location encoding strategy based on random walks to capture topological dependencies within the pathological diagram structure. The specific calculation logic is as follows: 1. Calculation of the transition matrix: For the constructed graph First, calculate its adjacency matrix. Matrix and degree matrix Define the random walk transition matrix. for: ; Where, it represents from node Single-step jump to neighbor node The probability of.

[0062] 2. K-step landing probability calculation: Calculate random walk The probability distribution matrix after the step For location encoding, the primary focus is on the landing probability, i.e., the probability of loops from the node. Departure, Passing through After one random walk, it returns to the node exactly. The probability of itself.

[0063] This probability is determined by the matrix. The diagonal elements are given: .

[0064] From a graph theory perspective Able to reflect nodes The density of the local subgraph structure. For example, nodes in the center of a dense community (clique) usually have a higher probability of self-returning, while nodes on the edge of a chain structure have a lower probability.

[0065] 3. Location encoding vector generation: Select a set of step size sequences (In this invention, the parameter walk_length is, for example, 20). The self-return probabilities at each step size are concatenated to form a node. Position encoding vector : .

[0066] 4. Data Integration: generated This will be appended to the graph data object as a separate structural feature tensor. Finally, each graph data object's `data` object contains: data.x: Initial visual features ; data.edge_index: Graph connectivity relationships ; data.edge_attr: Attributes of a ternary edge ; data.node_attr: dyeing ID ; data.random_walk_pe: Position encoding ; This completes the construction of graph data from raw multimodal pathological images to graph data containing rich structural and semantic information, laying a solid data foundation for subsequent training and classification.

[0067] The obtained multimodal heterogeneous pathological images of each object are specifically as follows: For each object's several image blocks, perform the following operations respectively: Each image patch of the current object is input into the pre-trained UNI deep learning pathology basic model, and the output of the penultimate layer of the UNI deep learning pathology basic model is extracted as the initial visual feature vector corresponding to each image patch. Determine the coloring attribute labels for each image patch; Using image patches as nodes, connections are established based on the spatial location of each node to determine the set of physical spatial adjacency edges. Specifically: when the... The node and the first Nodes originating from the same multi-stained whole-slice image and whose Euclidean distance on the 2D grid is less than a preset threshold, or the nodes originating from the same multi-stained whole-slice image and whose Euclidean distance on the 2D grid is less than a preset threshold, or the ... nodes origin The node and the first The nodes come from different multi-stained whole-slice images but their corresponding coordinates are completely consistent. The node and the first The first node establishes a connection; otherwise, the second node... The node and the first No connection is established between any nodes; Using image patches as nodes, connections are established based on the initial visual feature vectors corresponding to each node to determine the semantic feature nearest neighbor set. Specifically, for each node, the nearest feature distance is selected. A connection is established between the nodes; the feature distance is the cosine similarity or Euclidean distance calculated based on the initial visual feature vectors of each node; Edges are determined using the set of adjacent edges in physical space and the set of nearest edges in semantic features, thus obtaining the graph structure; Define edge attributes and generate position encoding vectors for each node based on random walks. Introduce the position encoding vectors, initial visual feature vectors, and staining attribute labels of each node into the graph structure to obtain the multimodal heterogeneous pathological map of the current object.

[0068] The expression for the edge attribute is:

[0069] in, For the first The node and the first The edge attributes of the connecting edges of each node; For coloring associated attributes, when the first The coloring attribute labels of each node With the The coloring attribute labels of each node If the values ​​are the same, a value of 0 indicates a homogeneous connection; otherwise, a value of 1 indicates a heterogeneous connection. For physical space adjacency attributes, when the first The node and the first If the connecting edge of a node belongs to the set of adjacent edges in the physical space, the value is 1; otherwise, the value is 0. For semantic features, the nearest neighbor attribute, when the... The node and the first If the connecting edges of a node belong to the semantic feature nearest neighbor edge set, the value is 1; otherwise, the value is 0. The expression for the position encoding vector of each node is:

[0070]

[0071] in, For the first The position encoding vector of each node; For the first Random walk of nodes The probability of self-returning after a step; For random walk The probability distribution matrix after the step.

[0072] (ii) Feature modulation and edge embedding based on metadata This phase aims to address two key issues in multi-stain pathological image analysis: first, the differences in feature distributions between different staining modalities (such as H&E and IHC); and second, the obfuscation of different types of connectivity relationships in the graph in traditional methods. To address these issues, this invention employs an independent decoupling embedding mechanism based on multi-source edge attributes and an adaptive modulation mechanism for node features based on metadata before the graph convolution operation.

[0073] Step 2.1: Edge attribute embedding for multi-source decoupling In order to enable graph neural networks to accurately distinguish the different semantics of edges in a graph (i.e., distinguish which edges represent physical organizational continuity and which edges represent potential biological homology), this invention abandons the traditional approach of directly concatenating edge attributes or processing them through a single embedding layer, and instead designs a parallel independent embedding architecture.

[0074] 1. Attribute separation: Receive the ternary edge attribute matrix from the graph data constructed in the first stage. For each edge in the graph Its properties are separated into three independent discrete components: Staining interaction components ( ):correspond This indicates whether the nodes connected by the edge are same-colored or cross-colored.

[0075] Region adjacency component ( ):correspond This indicates whether the edge is a physical neighbor (RAG).

[0076] Feature nearest neighbor components ( ):correspond This indicates whether the edge is a nearest neighbor (KNN) in the feature space.

[0077] 2. Independent Embedding Mapping: The system initializes three independent embedding layers, corresponding to the three components mentioned above: : Used to capture the latent semantics of coloring interactions.

[0078] Used to encode connectivity information in physical space.

[0079] : Connectivity information used to encode feature similarity.

[0080] For each edge, perform the following mapping operation to map discrete indices to continuous low-dimensional vectors (with dimensions set to edge_part_dim, e.g., 10-dimensional): ; ; .

[0081] 3. Generation of fused edge features: The three independent vectors are concatenated along the feature channel dimension to generate the final high-dimensional edge feature vector. : .

[0082] Technical effect: Through this decoupling-re-fusion mechanism, the generated edge features retain the independence of each attribute while achieving information integration. In the subsequent Graph Attention Network (GAT), this feature will explicitly participate in the calculation of attention coefficients, enabling the model to learn complex aggregation strategies such as "prioritizing physically adjacent nodes with similar features, but reducing attention to nodes that are far apart across colorings".

[0083] The edge attribute embedding module includes: The attribute separation unit is used to separate the edge attributes of multimodal heterogeneous pathology maps into corresponding staining-related attributes. Staining interaction components Corresponding physical space adjacency attribute Region adjacency components and corresponding semantic features nearest neighbor attributes Feature nearest neighbor components ; The first embedding layer is used to capture the coloring interaction components. The latent semantics are used to obtain continuous low-dimensional vectors corresponding to the coloring interactions. ; The second embedding layer is used to encode region adjacency components. Connectivity information in physical space yields a continuous low-dimensional vector corresponding to that physical space. ; The third embedding layer is used to encode the feature nearest neighbor components. Connectivity information from feature similarity yields a continuous low-dimensional vector in the feature space. ; The edge feature fusion unit is used to convert the continuous low-dimensional vectors corresponding to the coloring interactions. Continuous low-dimensional vectors corresponding to physical space The continuous low-dimensional vectors corresponding to the feature space Concatenate along the feature channel dimension to generate enhanced edge features. .

[0084] Step 2.2: Feature linear modulation based on metadata To eliminate the significant differences in feature space distribution between different stained sections (such as H&E staining and staining with different IHC markers), this step designs a feature recalibration module based on metadata. This module adopts an element-wise affine transformation strategy, using staining type embedding to generate specific scaling and translation factors, and dynamically modulates node features according to staining type.

[0085] 1. Metadata embedding: Using the node coloring attributes recorded in the first stage Through a shared node embedding layer Map it to a colored embedding vector This vector contains the global statistical characteristics of this coloring type.

[0086] 2. Dynamic parameter generator: In each layer of a graph neural network Each of these layers is configured with an independent linear projection layer. This projection layer receives the coloring embedding vector. As input, the output is a parameter vector with twice the feature dimension: ; 3. Parameter segmentation and feature recalibration: The generated parameter vector It is divided into two parts on average along the last dimension: the scaling factor vector. Translation factor vector : ; Subsequently, the nodes of this layer are input with features. Perform an element-wise affine transformation:

[0087] in, This represents the Hadamard product (element-by-element multiplication).

[0088] Each of the aforementioned feature modulation modules includes: The node embedding layer is used to map the node coloring attributes of the input data to coloring embedding vectors corresponding to each coloring type, based on the coloring attribute labels. The linear projection layer is used to perform linear projection on the color embedding vectors corresponding to each color type to obtain the projection parameters corresponding to each color type. The parameter slicing layer is used to divide the projection parameters corresponding to each coloring type into a scaling factor vector and a translation factor vector on the last dimension respectively. The affine transformation layer is used to perform element-wise affine transformations on the current node features based on the coloring attribute labels, according to the scaling factor vector and translation factor vector of each coloring type, to obtain the modulated node features:

[0089] in, For the first The modulated node features output by the layer feature modulation module; For the first The scaling factor vector of the coloring type corresponding to the coloring attribute label of the current node in the layer feature modulation module; For Hadamah accumulation; Features of the current node; For the first The translation factor vector of the coloring type corresponding to the coloring attribute label of the current node in the layer feature modulation module.

[0090] Step 2.3: Multimodal Feature Assembly and Input Construction After completing the edge feature construction and node feature modulation as described above, the data from each part is assembled into the standard input of the graph convolutional layer.

[0091] 1. Node feature splicing: Visual features modulated by color information Compared with the random walk position code generated in the first stage The channel dimensions are concatenated to form an enhanced node input matrix.

[0092] 2. Input transmission: Assembled node features The graph connection index edge_index and the fused edge features generated in step 2.1 Together they are fed into the next stage of the graph attention convolutional layer (GATv2Conv).

[0093] At this point, the data has completed the transformation from "original heterogeneous state" to "aligned and structured state", which is fully prepared for accurate reasoning in deep networks.

[0094] (III) Hierarchical Deep Feature Extraction and Fusion Interaction Stage This stage is the core inference process of the entire graph neural network. It employs stacking multiple (e.g.) The hierarchical architecture of the layer processing module. Unlike traditional methods that only perform unidirectional feature aggregation, this invention constructs a closed-loop update mechanism (HSI-Gating Loop) in each layer of the network, which includes "local aggregation, prototype extraction, global interaction, and gating feedback", to ensure that the features of deep nodes can be dynamically fused with cross-modal context information of the entire graph.

[0095] Step 3.1: Graph Attention Aggregation Based on Dual Optimization In the In the initial stage of the layer processing module, local feature aggregation is first performed using a graph attention network (GATv2).

[0096] 1. Input reception: Receive the modulated node feature matrix from the second stage (or the previous layer). and multi-source decoupling edge features .

[0097] 2. Attention mechanism: For each node Calculate its relationship with neighboring nodes Attention coefficient In this calculation, edge features It is explicitly incorporated into the attention scoring function, enabling the model to dynamically adjust the aggregation weights based on the type of edges (such as physical adjacency or semantic similarity).

[0098] 3. Message passing: Aggregate neighborhood information based on the calculated attention weights and update node features.

[0099] Step 3.2: Colorimetric Attention Pooling and Prototype Extraction To reduce computational complexity and extract higher-order semantics, the system then performs stain-aware SAGPooling, which includes the following three sub-steps: 1. Node Importance Assessment and Selection: The importance attention score for each node is calculated using graph convolutional layers. Based on a preset retention ratio (e.g., 0.7), the top-K nodes with the highest scores are retained from the entire graph, while redundant background nodes with lower scores are removed, resulting in a sparse graph structure.

[0100] 2. Attention-Guided Feature Reweighting: Using the attention score calculated in step 1, perform an element-wise weighted summation on the retained node features. For each retained node, multiply its feature vector by its corresponding normalized attention score.

[0101] This operation not only filters nodes using scores, but also modulates the feature amplitude using the numerical value of the scores. The feature vectors of nodes with high attention scores (usually corresponding to regions with significant pathological features) are further amplified, while the features of edge nodes are relatively suppressed, thereby significantly enhancing the signal-to-noise ratio of the features.

[0102] 3. Stain Prototype Construction: After feature weighting, nodes are grouped according to their coloring attributes. For each existing coloring type, the weighted features of all nodes to which that coloring belongs are aggregated (using a concatenation of the mean and the maximum value) to generate a coloring prototype vector representing the overall state of the current level of that coloring slice.

[0103] Since the input is attention-weighted features, the resulting prototype vector can highly condense the most valuable biological information for classification in the stained slice.

[0104] Step 3.3: HSIT cross-stain global interaction To capture long-range, non-local dependencies between different stained sections (e.g., the correlation between morphological features of tumor regions and expression levels of distant immune cells), this invention employs the HSI-Transformer module (Hierarchical Stain Interaction Transformer). Figure 2 The hierarchical cross-chromosome interaction and gated feedback module's hierarchical cross-chromosome interaction component.

[0105] 1. Serialization Modeling: Arrange all the colorimetric prototype vectors extracted in step 3.2 into a sequence. At the same time, positional encoding enhancement is performed on the sequence using learnable colorimetric type embedding.

[0106] 2. Self-attention interaction: Input the sequence into the Transformer encoder layer. Utilize the multi-head self-attention (MHSA) mechanism to calculate the association strength between each pair of chromatid prototypes in the sequence.

[0107] This employs a self-attention mechanism, enabling each chromatid prototype to dynamically aggregate semantic information from other related chromatid prototypes. For example, the prototype vector of the ER can extract its associated organizational context from the H&E prototype through the attention mechanism.

[0108] 3. Context Generation: The Transformer outputs the enhanced sequence after interaction. At this point, each vector... It no longer represents information with a single coloring, but rather a global semantic representation that incorporates the entire stack context.

[0109] Step 3.4: Adaptive Gating Feedback Mechanism To inject the global context information obtained in step 3.3 into local micro-nodes, this invention designs a "context broadcasting and gating fusion" mechanism to form a closed loop for feature updates. Figure 2 The gating feedback component in the hierarchical cross-coloring interaction and gating feedback module.

[0110] 1. Context broadcasting: The global context vector output by the Transformer is then broadcast backwards, based on the coloring ID index, to every local node belonging to that coloring type. In other words, all nodes belonging to the H&E coloring will receive the same H&E context vector after global interaction. .

[0111] 2. Gating coefficient calculation: To prevent global information from overshadowing local details, a learnable sigmoid gating unit is introduced. For each node, its current features are concatenated with the received global context and input into the fully connected layer to calculate the fusion gating coefficients.

[0112] 3. Feature closed-loop update: The node features are then updated based on the gating coefficient.

[0113] This mechanism enables adaptive input of local node features to the global cross-modal context. If the global context determines that the current sample is high-risk (e.g., by detecting multi-slice collaboration anomalies through Transformer), the gating mechanism will automatically increase. This forces local node features to gravitate towards abnormal patterns; conversely, it preserves the original attributes of local features.

[0114] Step 3.5: Hierarchical Iteration and Readout Steps 3.1 to 3.4 above constitute a complete network layer (Layer Block). Through stacking... These layers (e.g., 2-4 layers) progressively extract features from low-level texture to high-level semantics.

[0115] Finally, the global colorimetric prototypes (Layer Readouts) of each layer's output are collected, and then the multi-layer readouts are concatenated to form a long vector representing all layers, generating a graph-level representation vector of the entire patient, which serves as the basis for final classification. Simultaneously, intermediate features for conditional prototype learning (SPC) are collected for use in the next stage of training.

[0116] Each of the aforementioned deep hierarchical feature extraction networks includes: The graph attention aggregation unit is used to concatenate the position encoding vector of each node with the modulated node features output by the feature modulation module of the corresponding level along the channel dimension to obtain the enhanced node input matrix; based on the enhanced node input matrix and enhanced edge features, the attention weights of each node and its neighboring nodes are calculated; and neighborhood information is aggregated according to the attention weights of each node and its neighboring nodes to obtain the updated node features; the expression for the attention weights of each node and its neighboring nodes is as follows:

[0117] in, For the first The node and the first Attention weights of each neighboring node; Use the Softmax activation function; For leaky modified linear unit activation functions; This is a vector of learnable attention mechanism parameters; The feature projection matrix to be learned; To enhance the number of nodes in the input matrix The feature vector of each node; To enhance the number of nodes in the input matrix Feature vectors of neighboring nodes; For the first The node and the first The enhanced edge features between each neighbor node are taken from the overall enhanced edge features. ; For splicing; For transpose; The coloring-aware attention pooling and prototype extraction unit is used to calculate the importance attention score of each node using self-attention-based graph pooling; and retain the K nodes with the highest importance attention scores to obtain a sparse graph structure; the updated node features of each node in the sparse graph structure are multiplied by the normalized importance attention score of the node to obtain the weighted features of each node in the sparse graph structure; the nodes in the sparse graph structure are grouped according to the coloring attribute label to obtain node groups under each coloring attribute; for each node group under each coloring attribute, the weighted features of each node in the group are aggregated to obtain the coloring prototype vector of the overall state of each coloring attribute at the current level.

[0118] in, For the first Deep Hierarchical Feature Extraction Network The coloring prototype vector of the overall state of each coloring attribute; For splicing; For the first The group of coloring attribute nodes corresponding to each coloring attribute The number of nodes; For node indexing; Coloring attribute node group No. The weighted features of each node; Coloring attribute node group The maximum value of the weighted feature of the middle node; The HSIT cross-coloring global interaction unit is used to arrange the coloring prototype vectors of the overall state of each coloring attribute in the current level into a coloring prototype sequence; the coloring prototype sequence is enhanced by position encoding using the coloring embedding vectors corresponding to each coloring type output by the feature modulation module of the corresponding level, resulting in a coloring prototype enhanced sequence; the Transformer encoder is used to extract features from the coloring prototype enhanced sequence to obtain the cross-slice global context features corresponding to each coloring type. The adaptive gating feedback mechanism unit broadcasts the global context features across slices corresponding to each staining type back to the nodes with the corresponding staining attribute labels in the sparse graph structure, and updates the node features by combining the gating coefficients to obtain the final node features:

[0119] in, For the sparse graph structure, the first The final node characteristics of each node; To integrate the gating coefficients; For Hadamah accumulation; For the sparse graph structure, the first The updated node characteristics of each node; This is a feature projection function used to map and align the dimensions of global context features to a coordinate system. Same dimensions; For the sparse graph structure, the first Each node is determined by its assigned coloring attribute label. Received global context features across slices corresponding to the staining type; Sigmoid is the activation function; This is the learnable weight matrix of the fully connected layer inside the gated unit; For splicing.

[0120] (iv) Robust Conditioned Prototype Learning This phase aims to address the missing modality problem prevalent in practical clinical applications. Traditional deep learning methods typically require input data with a fixed number of channels. When dealing with missing segments (e.g., patients who underwent H&E but not PR), zero-padding or mean imputation is often used. This introduces non-biological noise distributions into the feature space, severely interfering with classification boundaries.

[0121] To address this, this invention proposes a training strategy of "Structured 2D Grid with Dynamic Masking" to achieve "on-demand supervision." Corresponding to... Figure 3 A schematic diagram of a robust conditional prototyping learning process combined with a dynamic masking strategy.

[0122] Step 4.1: Construct a structured 2D prototype mesh Unlike traditional contrastive learning (such as MoCo, SimCLR) which maintains an unordered, first-in-first-out queue of negative samples, this invention constructs and maintains a two-dimensional learnable parameter matrix (prototype library) with a clear semantic structure, denoted as... .

[0123] 1. Mesh Definition: Prototype Library The dimension is defined as ,in: The maximum number of colorimetric species in the dataset (e.g., H&E, ER, PR, etc.).

[0124] : Number of disease classification categories (e.g., category A, category B, etc. tissue subtypes).

[0125] The number of prototypes for each class (usually set to 1).

[0126] : Feature dimension after projection.

[0127] 2. Physical meaning: Each cell in the grid Represents "the first The staining type in the first "Ideal feature center under each disease category" This is the color type index corresponding to the total set of color types. For example, This represents the "typical characteristic pattern that ER slices of patients with ovarian endometrioid carcinoma should have." This matrix is ​​updated end-to-end along with the network parameters using gradient updates.

[0128] Step 4.2: Existence Detection and Dynamic Masking of Incomplete Data During the training iteration, for each input patient sample (and its true label), dynamic masking logic is first executed to ensure the data integrity of the sample.

[0129] 1. Existence Detection: Scan the patient's current set of actual chromosome embeddings. Assume the complete set is... The patient lacked PR. A set was identified. .

[0130] 2. Prototype Dynamic Retrieval: based on and tags Generate a dynamic index mask, only from the prototype library. Retrieve relevant prototypes from the model and use them in the calculation, completely ignoring the prototypes corresponding to the missing modes: Positive Sample Prototype Set Search all And the category is The prototype. That is... ; negative sample prototype set Search all And the category is not The prototype. That is... ; For missing PR modalities, neither feature imputation nor the corresponding PR prototype is retrieved. The computation process bypasses the missing data, effectively avoiding noise interference introduced into the feature space by traditional interpolation or zero-padding operations.

[0131] Step 4.3: Calculation of Conditional Contrast Loss Based on the retrieved valid prototype subset, the robust conditional contrastive loss is calculated.

[0132] 1. Feature projection: The extracted global context features across slices for each staining type (from the third-stage output) are mapped to the same dimensional space as the prototype through a projection head and then L2 normalized.

[0133] in, The normalized projected features are used in all subsequent loss calculations. .

[0134] 2. Similarity matrix calculation: Only the dot product similarity between the existing coloring embedding and the retrieved prototype is calculated.

[0135] 3. Loss function construction: The loss function in the form of InfoNCE aims to bring patients' staining features closer to their corresponding "same-same-staining prototypes" and push them further away from "different-staining prototypes".

[0136] Step 4.4: Multi-task joint optimization The final training objective function consists of the classification main task loss and the auxiliary SPC loss:

[0137] in: It is a cross-entropy loss calculated based on the full graph representation vector (Graph Readout) and the output of the fully connected layer, used to supervise the overall classification accuracy.

[0138] The conditional contrastive loss calculated above is used to constrain the distribution of intermediate layer features, making them more compact within classes and separable between classes, and forcing the model to learn decoupled coloring feature representations.

[0139] Through this training method, even when the model encounters samples with missing slices during the inference stage (the next stage), it can still match the learned robust prototypes based on the existing coloring features, demonstrating extremely strong robustness.

[0140] The loss function of the pathological image classification model is:

[0141]

[0142]

[0143] in, The loss function for the pathological image classification model; Cross-entropy loss; This is an unnormalized pathological prediction score vector; The actual disease classification labels corresponding to the current multimodal heterogeneous pathology diagram; Weights for conditional contrastive loss; For conditional contrast loss; For the first A set of staining types present in a multimodal heterogeneous pathological map; for Projection features after projection head mapping and L2 normalization; To extract the first layer of the L-layer deep hierarchical feature extraction network output The deep feature representation is obtained by concatenating the global context features across slices corresponding to each staining type; The positive sample prototype vector retrieved from the structured 2D prototype mesh library, i.e., the coloring type index is... Furthermore, the disease category index is the true label of the current multimodal heterogeneous pathology map. The ideal feature center; Temperature coefficient; Index for color type; Index for disease categories; The total number of categories; The prototype vector retrieved from the structured 2D prototype mesh library, i.e., the coloring type index is... And the disease category index is Same-staining characteristic centers; For the characteristic projection function; For L2 normalization.

[0144] (V) Multi-level feature aggregation and classification prediction This stage is the output terminal. After processing by the aforementioned hierarchical graph neural network, the model has extracted deep features containing multi-coloring semantics and cross-slice global context. The task of this stage is to aggregate these features distributed at different network depths at multiple scales and map them to the classification space of clinical diseases, outputting the final classification result.

[0145] Step 5.1: Aggregation of Multi-Scale Full-Graph Representations To comprehensively capture pathological information from microscopic textures (shallow network features) to macroscopic organizational patterns (deep network features), a patient-level final representation vector is constructed using a hierarchical concatenation strategy.

[0146] 1. Hierarchical read-out collection: Looking back at the third stage, in each layer of the graph neural network (common Each layer outputs a layer readout vector that incorporates the global context of that layer through the HSI-Transformer. .

[0147] Shallow readout : Usually focuses on basic visual features such as cell nucleus morphology, color and texture.

[0148] Deep Read : Usually focuses on higher-order semantic features such as immune infiltration patterns and tumor microenvironment structure.

[0149] 2. Feature assembly and normalization: The readout vectors from all layers are concatenated along the feature dimension to form a very long-dimensional full-image representation vector. : ; Subsequently, layer normalization was applied to... Standardization is performed to eliminate differences in the magnitude of feature values ​​at different levels, ensuring the stability of classifier training.

[0150] Step 5.2: Nonlinear Mapping and Probability Output The aggregated full graph representation vector is input into the fully connected classification head (MLP Head) for dimensionality reduction and mapping.

[0151] 1. Feature Transformation: First, the network passes through a fully connected (FC) layer and a ReLU activation function, and then a Dropout strategy is applied to prevent overfitting. ; This step compresses high-dimensional features into the hidden layer space and extracts the most discriminative feature combination.

[0152] 2. Logits calculation and probability normalization: After being mapped to the target classification space (size is...) by a second fully connected network layer. (i.e., the number of disease categories), outputting unnormalized prediction scores (Logits): ; Finally, the Softmax function is used to convert Logits into a probability distribution: .

[0153] The category with the highest output probability is the multi-stain classification prediction label given by the system for the whole slice image (such as "high-grade serous carcinoma of the ovary" or "endometrioid carcinoma of the ovary").

[0154] Step 5.3: Automatic adaptation of incomplete data during the inference phase This is a key advantage of the invention in clinical applications. In actual classification (reasoning) processes, if the input patient sample lacks certain stained sections (e.g., only H&E and ER are provided, PR is missing): 1. No human intervention required: No need for doctors to manually mark "missing" values, and no need for any interpolation preprocessing.

[0155] 2. Automatic adaptation process: In the first stage, the graph construction algorithm will only generate nodes of existing slices and will not generate PR nodes.

[0156] In the second stage, the feature modulation module only modulates the features of existing nodes. Edge information also consists only of existing node information.

[0157] In the third phase, the Transformer only interacts between the existing H&E and ER prototypes.

[0158] In this stage, the aggregated readout vector It is still a complete vector (because the read vector is a global feature with a fixed dimension), and the prediction result can be obtained by directly inputting it into the classifier.

[0159] Thanks to the robust conditional prototype training in the fourth stage, the classifier has learned to make accurate judgments even with only partial modal inputs (i.e. seeing some features as 0 or specific patterns), thus achieving "plug-and-play" image classification of incomplete clinical data.

[0160] The expression for the pathological classification prediction result is as follows:

[0161]

[0162]

[0163]

[0164] in, For pathological classification prediction results; This is the predicted final disease classification result; Index for disease categories; For input multimodal heterogeneous pathological images; The first unnormalized pathological prediction score vector Predicted component values ​​for each category; The total number of categories; The first unnormalized pathological prediction score vector Predicted component values ​​for each category; The unnormalized pathological prediction score; This is the learnable weight matrix for the second fully connected classification network; The hidden layer classification feature vector is obtained after the graph-level feature representation undergoes dimensionality reduction and nonlinear transformation in the first fully connected network layer. This is the learnable bias vector for the second fully connected classification network; For random inactivation; To modify the activation function of the linear unit; This is the learnable weight matrix of the first fully connected classification network; For layer normalization; This represents graph-level features. This is the learnable bias vector for the first layer of the fully connected classification network; For splicing; The global context features across slices corresponding to each staining type are extracted from the Lth layer deep hierarchical feature extraction network output.

Claims

1. A pathological image classification method based on staining-aware intelligent feature modeling, characterized in that, include: Collect a dataset of multi-stained whole-section pathological images; The multi-stain whole-section pathological image dataset includes multi-stain whole-section image groups of different objects; The multi-stain whole-section image group data includes H&E staining images and several immunohistochemical staining images; Preprocess the data of each multi-stained whole-section image group to obtain several image blocks of equal size corresponding to each object; Based on the image patches of each object, a map is constructed to obtain a multimodal heterogeneous pathological map of each object; The pathological image classification model was trained using multimodal heterogeneous pathological images of each object as training data. The pathological image classification model includes: The edge attribute embedding module is used to embed edge attributes of multi-source decoupling multimodal heterogeneous pathological maps to obtain enhanced edge features; The L-layer feature modulation modules are all used to enhance node features based on metadata-based feature modulation mechanisms to obtain modulated node features; the input data of the first layer feature modulation module is a multimodal heterogeneous pathological image; the input data of the 2nd to Lth layer feature modulation modules is the final node features output by the previous deep hierarchical feature extraction network. The L-layer deep hierarchical feature extraction network is used to obtain the global context features across slices corresponding to each coloring type based on the enhanced edge features and the modulated node features output by the feature modulation module of the corresponding layer; the global context features across slices corresponding to each coloring type are broadcast back to the nodes with the corresponding coloring attribute labels, and combined with the gating coefficients to obtain the final node features. The MLP classification head is used to obtain a graph-level feature representation by splicing the global context features across slices corresponding to each staining type output by the L-layer deep hierarchical feature extraction network, and to obtain the pathological classification prediction result. The trained pathological image classification model is used to classify the pathological images to be tested.

2. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 1, characterized in that, The specific steps for obtaining several image blocks of equal size corresponding to each object are as follows: For the multi-stained whole-section image data of each object, the following operations were performed: For each stained image in the multi-stained whole-section image group of the current object, the following operations are performed: the stained image is converted to the HSV color space to obtain the HSV stained image; a binary mask of the HSV stained image is generated using the Otsu method, and the background area is removed based on the binary mask of the HSV stained image to obtain the effective area containing only biological tissue; at a preset magnification, the effective area is seamlessly cut into several non-overlapping image blocks of fixed size; By integrating the image blocks corresponding to each stained image in the multi-stained whole-slice image group data of the current object, several image blocks of the same size corresponding to the current object are obtained.

3. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 1, characterized in that, The obtained multimodal heterogeneous pathological images of each object are specifically as follows: For each object's several image blocks, perform the following operations respectively: Each image patch of the current object is input into the pre-trained UNI deep learning pathology basic model, and the output of the penultimate layer of the UNI deep learning pathology basic model is extracted as the initial visual feature vector corresponding to each image patch. Determine the coloring attribute labels for each image patch; Using image patches as nodes, connections are established based on the spatial location of each node to determine the set of physical spatial adjacency edges. Specifically: when the... The node and the first Nodes originating from the same multi-stained whole-slice image and whose Euclidean distance on the 2D grid is less than a preset threshold, or the nodes originating from the same multi-stained whole-slice image and whose Euclidean distance on the 2D grid is less than a preset threshold, or the ... nodes origin The node and the first The nodes come from different multi-stained whole-slice images but their corresponding coordinates are completely consistent. The node and the first The first node establishes a connection; otherwise, the second node... The node and the first No connection is established between any nodes; Using image patches as nodes, connections are established based on the initial visual feature vectors corresponding to each node to determine the semantic feature nearest neighbor set. Specifically, for each node, the nearest feature distance is selected. A connection is established between the nodes; the feature distance is the cosine similarity or Euclidean distance calculated based on the initial visual feature vectors of each node; Edges are determined using the set of adjacent edges in physical space and the set of nearest edges in semantic features, thus obtaining the graph structure; Define edge attributes and generate position encoding vectors for each node based on random walks. Introduce the position encoding vectors, initial visual feature vectors, and staining attribute labels of each node into the graph structure to obtain the multimodal heterogeneous pathological map of the current object.

4. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 3, characterized in that, The expression for the edge attribute is: in, For the first The node and the first The edge attributes of the connecting edges of each node; For coloring associated attributes, when the first The coloring attribute labels of each node With the The coloring attribute labels of each node If the values ​​are the same, a value of 0 indicates a homogeneous connection; otherwise, a value of 1 indicates a heterogeneous connection. For physical space adjacency attributes, when the first The node and the first If the connecting edge of a node belongs to the set of adjacent edges in the physical space, the value is 1; otherwise, the value is 0. For semantic features, the nearest neighbor attribute, when the... The node and the first If the connecting edges of a node belong to the semantic feature nearest neighbor edge set, the value is 1; otherwise, the value is 0. The expression for the position encoding vector of each node is: in, For the first The position encoding vector of each node; For the first Random walk of nodes The probability of self-returning after a step; For random walk The probability distribution matrix after the step.

5. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 1, characterized in that, The edge attribute embedding module includes: The attribute separation unit is used to separate the edge attributes of multimodal heterogeneous pathology maps into corresponding staining-related attributes. Staining interaction components Corresponding physical space adjacency attribute Region adjacency components and corresponding semantic features nearest neighbor attributes Feature nearest neighbor components ; The first embedding layer is used to capture the coloring interaction components. The latent semantics are used to obtain continuous low-dimensional vectors corresponding to the coloring interactions. ; The second embedding layer is used to encode region adjacency components. Connectivity information in physical space yields a continuous low-dimensional vector corresponding to that physical space. ; The third embedding layer is used to encode the feature nearest neighbor components. Connectivity information from feature similarity yields a continuous low-dimensional vector in the feature space. ; The edge feature fusion unit is used to convert the continuous low-dimensional vectors corresponding to the coloring interactions. Continuous low-dimensional vectors corresponding to physical space The continuous low-dimensional vectors corresponding to the feature space Concatenate along the feature channel dimension to generate enhanced edge features. .

6. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 1, characterized in that, Each of the aforementioned feature modulation modules includes: The node embedding layer is used to map the node coloring attributes of the input data to coloring embedding vectors corresponding to each coloring type, based on the coloring attribute labels. The linear projection layer is used to perform linear projection on the color embedding vectors corresponding to each color type to obtain the projection parameters corresponding to each color type. The parameter slicing layer is used to divide the projection parameters corresponding to each coloring type into a scaling factor vector and a translation factor vector on the last dimension respectively. The affine transformation layer is used to perform element-wise affine transformations on the current node features based on the coloring attribute labels, according to the scaling factor vector and translation factor vector of each coloring type, to obtain the modulated node features: in, For the first The modulated node features output by the layer feature modulation module; For the first The scaling factor vector of the coloring type corresponding to the coloring attribute label of the current node in the layer feature modulation module; For Hadamah accumulation; Features of the current node; For the first The translation factor vector of the coloring type corresponding to the coloring attribute label of the current node in the layer feature modulation module.

7. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 6, characterized in that, Each of the aforementioned deep hierarchical feature extraction networks includes: The graph attention aggregation unit is used to concatenate the position encoding vector of each node with the modulated node features output by the feature modulation module of the corresponding level along the channel dimension to obtain the enhanced node input matrix; based on the enhanced node input matrix and enhanced edge features, the attention weights of each node and its neighboring nodes are calculated; and neighborhood information is aggregated according to the attention weights of each node and its neighboring nodes to obtain the updated node features; the expression for the attention weights of each node and its neighboring nodes is as follows: in, For the first The node and the first Attention weights of each neighboring node; Use the Softmax activation function; For the modified linear unit activation function with leakage; This is a vector of learnable attention mechanism parameters; The feature projection matrix to be learned; To enhance the number of nodes in the input matrix Feature vectors of each node; To enhance the number of nodes in the input matrix Feature vectors of each neighboring node; For the first The node and the first The enhanced edge features between each neighbor node are taken from the overall enhanced edge features. ; For splicing; For transpose; The coloring-aware attention pooling and prototype extraction unit is used to calculate the importance attention score of each node using self-attention-based graph pooling; and retain the K nodes with the highest importance attention scores to obtain a sparse graph structure; the updated node features of each node in the sparse graph structure are multiplied by the normalized importance attention score of the node to obtain the weighted features of each node in the sparse graph structure; the nodes in the sparse graph structure are grouped according to the coloring attribute label to obtain node groups under each coloring attribute; for each node group under each coloring attribute, the weighted features of each node in the group are aggregated to obtain the coloring prototype vector of the overall state of each coloring attribute at the current level. in, For the first Deep Hierarchical Feature Extraction Network The coloring prototype vector of the overall state of each coloring attribute; For splicing; For the first The group of coloring attribute nodes corresponding to each coloring attribute The number of nodes; For node indexing; For coloring attribute node groups No. The weighted features of each node; For coloring attribute node groups The maximum value of the weighted feature of the middle node; The HSIT cross-coloring global interaction unit is used to arrange the coloring prototype vectors of the overall state of each coloring attribute in the current level into a coloring prototype sequence; the coloring prototype sequence is enhanced by position encoding using the coloring embedding vectors corresponding to each coloring type output by the feature modulation module of the corresponding level, resulting in a coloring prototype enhanced sequence; the Transformer encoder is used to extract features from the coloring prototype enhanced sequence to obtain the cross-slice global context features corresponding to each coloring type. The adaptive gating feedback mechanism unit broadcasts the global context features across slices corresponding to each staining type back to the nodes with the corresponding staining attribute labels in the sparse graph structure, and updates the node features by combining the gating coefficients to obtain the final node features: in, For the sparse graph structure, the first The final node characteristics of each node; To integrate the gating coefficients; For Hadamah accumulation; For the sparse graph structure, the first The updated node characteristics of each node; This is a feature projection function used to map and align the dimensions of global context features to a coordinate system. Same dimensions; For the sparse graph structure, the first Each node is determined by its assigned coloring attribute label. Received global context features across slices corresponding to the staining type; Sigmoid is the activation function; This is the learnable weight matrix of the fully connected layer inside the gated unit; For splicing.

8. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 1, characterized in that, The expression for the pathological classification prediction result is as follows: in, For pathological classification prediction results; This is the predicted final disease classification result; Index for disease categories; For input multimodal heterogeneous pathological images; The first in the unnormalized pathological prediction score vector Predicted component values ​​for each category; The total number of categories; The first unnormalized pathological prediction score vector Predicted component values ​​for each category; The unnormalized pathological prediction score; This is the learnable weight matrix for the second fully connected classification network; The hidden layer classification feature vector is obtained after the graph-level feature representation undergoes dimensionality reduction and nonlinear transformation in the first fully connected network layer. This is the learnable bias vector for the second fully connected classification network; For random inactivation; To modify the activation function of the linear unit; This is the learnable weight matrix of the first fully connected classification network; For layer normalization; This represents graph-level features. This is the learnable bias vector for the first layer of the fully connected classification network; For splicing; The global context features across slices corresponding to each staining type are extracted from the Lth layer deep hierarchical feature extraction network output.

9. The pathological image classification method based on staining-aware intelligent feature modeling according to claim 1, characterized in that, The loss function of the pathological image classification model is: in, The loss function for the pathological image classification model; Cross-entropy loss; This is an unnormalized pathological prediction score vector; The actual disease classification labels corresponding to the current multimodal heterogeneous pathology diagram; Weights for conditional contrastive loss; For conditional contrast loss; For the first A set of staining types present in a multimodal heterogeneous pathological map; for Projection features after projection head mapping and L2 normalization; To extract the first layer of the L-layer deep hierarchical feature extraction network output The deep feature representation is obtained by concatenating the global context features across slices corresponding to each staining type; The positive sample prototype vector retrieved from the structured 2D prototype mesh library, i.e., the coloring type index is... Furthermore, the disease category index is the true label of the current multimodal heterogeneous pathology map. The ideal feature center; Temperature coefficient; Index for coloring type; Index for disease categories; The total number of categories; The prototype vector retrieved from the structured 2D prototype mesh library, i.e., the coloring type index is... And the disease category index is Same-staining characteristic centers; For the feature projection function; For L2 normalization; each mesh element in the structured two-dimensional prototype mesh library Indicates the first The staining type in the first Ideal feature centers for each disease category; This is the color type index corresponding to the total set of color types.

Images