Pathological image classification method based on dual-domain collaborative graph attention network and related device
By employing a dual-domain collaborative graph attention network approach, which combines self-attention and spatial-frequency cross-attention mechanisms, the problem of balancing local key regions and global consistency patterns in existing pathological image classification models is solved, achieving higher classification accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUYI UNIV
- Filing Date
- 2026-01-20
- Publication Date
- 2026-05-29
AI Technical Summary
Existing deep learning models struggle to simultaneously ensure accurate identification of key local regions and grasp of global consistency patterns when processing whole-section images of breast cancer pathology, which affects the accuracy and robustness of HER2 status interpretation.
We employ a dual-domain collaborative graph attention network approach, which constructs a graph structure through a self-attention mechanism and combines it with a spatial-frequency domain cross-attention mechanism to achieve deep fusion of spatial and frequency domain features, thereby improving the accuracy and robustness of pathological image classification.
It improves the classification performance and generalization ability of complex pathological images, and can handle the high heterogeneity and complex structure of pathological tissues in a more balanced and effective manner, thereby improving classification accuracy and robustness.
Smart Images
Figure CN122115934A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of medical image processing technology, and in particular to a pathological image classification method and related apparatus based on a dual-domain collaborative graph attention network. Background Technology
[0002] Breast cancer is the most common malignant tumor among women worldwide, with persistently high incidence and mortality rates, posing a serious challenge to public health. In the clinical diagnosis and treatment of breast cancer, accurate molecular subtyping is the core basis for implementing personalized targeted therapy. Among these, the status of human epidermal growth factor receptor 2 (HER2) is one of the key molecular markers, directly determining whether patients can benefit from targeted drugs such as Herceptin. Therefore, accurate and efficient prediction of HER2 status is crucial for improving patient prognosis and survival rates.
[0003] Currently, the gold standard for clinical HER2 status interpretation relies on manual microscopic observation of whole-slide images (WSI) by physicians, combined with specialized staining and molecular detection techniques such as immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH). However, whole-slide images typically have gigapixel-level ultra-high resolution but lack pixel-level annotations, posing a challenge to automated analysis. While deep learning has made significant progress in WSI analysis in recent years, existing deep learning models struggle to simultaneously ensure accurate identification of key local regions and grasp of globally consistent patterns when dealing with highly heterogeneous pathological tissues, affecting the accuracy and robustness of the final classification. Summary of the Invention
[0004] This invention provides a pathological image classification method and related device based on a dual-domain collaborative graph attention network, which can more balancedly and effectively handle the high heterogeneity and complex structure of pathological tissues, improve classification performance and generalization ability, and enhance the accuracy and robustness of classifying complex pathological images.
[0005] In a first aspect, embodiments of the present invention provide a pathological image classification method based on a dual-domain collaborative graph attention network, comprising: Obtain whole-section images of pathological tissue; The pathological whole-section images are preprocessed to obtain instance feature matrices and instance coordinate matrices; Graph construction is performed on the instance feature matrix and the instance coordinate matrix to obtain the target graph structure; Based on the target graph structure, a self-attention mechanism is used to encode instances using graph-based self-attention, resulting in a graph feature matrix calculated by self-attention. The frequency domain transformation and enhancement are performed on the instance feature matrix to obtain frequency domain enhanced features; Spatial-frequency domain cross-attention fusion is performed on the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; The instance-level features are aggregated to obtain a global feature vector; The global feature vector is input into a classifier to predict the molecular marker status, thereby obtaining the classification prediction result corresponding to the pathological whole slice image.
[0006] In some embodiments, the preprocessing of the pathological whole-slice image to obtain the instance feature matrix and instance coordinate matrix includes: The pathological whole-section image is segmented into multiple image block instances; Extract the depth feature vector of each image patch instance to obtain the instance feature matrix and the instance coordinate matrix.
[0007] In some embodiments, the step of segmenting the whole pathological slide image into multiple image patch instances includes: The pathological whole-section image is converted from the RGB color space to the HSV color space to obtain the first whole-section image; The first full-slice image is subjected to median filtering for noise reduction to obtain the second full-slice image; The second full-slice image is subjected to threshold segmentation to obtain the tissue region and background; Morphological operations are performed on the tissue region and the background to obtain a foreground mask; The area covered by the foreground mask is divided into grids, and each grid cell covered by the foreground mask is taken as an instance. Multiple image block instances corresponding to the instances are obtained, and the center coordinates of the multiple image block instances in the pathological whole slice image are recorded.
[0008] In some embodiments, extracting the depth feature vector of each image patch instance to obtain the instance feature matrix and the instance coordinate matrix includes: Each of the image patch instances is converted into a dense feature vector; The deep feature vector of each image patch instance is extracted from the dense feature vector using a pre-trained ResNet50 feature extraction model, resulting in the instance feature matrix and the instance coordinate matrix.
[0009] In some embodiments, the step of constructing a graph from the instance feature matrix and the instance coordinate matrix to obtain a target graph structure includes: Clustering operations are performed on instances within the instance feature matrix based on the clustering algorithm, using the instance coordinate matrix as a reference to construct a graph structure. The instances are divided into clusters based on their cluster centers using clustering operations. The target graph structure is obtained by constructing intra-cluster edge relationships and weight relationships between instances in different clusters.
[0010] In some embodiments, performing frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features includes: The instance feature matrix is mapped from the spatial domain to the frequency domain using a fast Fourier transform to obtain the spectral representation of the instance feature matrix, wherein the spectral representation includes an amplitude spectrum and a phase spectrum; The amplitude spectrum is filtered and enhanced to obtain the target amplitude spectrum; The target amplitude spectrum and phase spectrum are recombined to obtain an enhanced spectral representation; The enhanced spectral representation is mapped back to the spatial domain using an inverse fast Fourier transform to obtain the frequency domain enhanced features.
[0011] In some embodiments, it also includes: A visual diagnostic report is generated based on the classification prediction results, wherein the classification prediction results include the classification results and confidence levels of molecular markers, and the classifier consists of a fully connected layer and a Softmax function.
[0012] Secondly, embodiments of the present invention also provide a pathological image classification device based on a dual-domain collaborative graph attention network, the device comprising: The acquisition module is used to acquire images of whole pathological slides; The preprocessing module is used to preprocess the pathological whole slide image to obtain the instance feature matrix and instance coordinate matrix; The graph construction module is used to construct a graph from the instance feature matrix and the instance coordinate matrix to obtain the target graph structure. The encoding module is used to perform graph-based self-attention encoding on instances according to the target graph structure using a self-attention mechanism, so as to obtain a graph feature matrix calculated by self-attention. The enhancement module is used to perform frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features; The fusion module is used to perform spatial-frequency domain cross-attention fusion on the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; The aggregation module is used to aggregate the instance-level features to obtain a global feature vector; The output module is used to input the global feature vector into the classifier to perform classification prediction of molecular marker states, and obtain the classification prediction result corresponding to the pathological whole slice image.
[0013] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the pathological image classification method based on a dual-domain collaborative graph attention network as described in the first aspect.
[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the pathological image classification method based on a dual-domain collaborative graph attention network as described in the first aspect.
[0015] The pathological image classification method and related apparatus based on a dual-domain collaborative graph attention network provided by embodiments of the present invention include: acquiring a whole-slice image of a pathological tissue; preprocessing the whole-slice image of the pathological tissue to obtain an instance feature matrix and an instance coordinate matrix; constructing a graph from the instance feature matrix and the instance coordinate matrix to obtain a target graph structure; performing graph-based self-attention encoding on instances according to the target graph structure using a self-attention mechanism to obtain a graph feature matrix calculated by self-attention; performing frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features; performing spatial-frequency domain cross-attention fusion on the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; performing feature aggregation on the instance-level features to obtain a global feature vector; and inputting the global feature vector into a classifier to perform classification prediction of molecular marker states to obtain the classification prediction result of the corresponding whole-slice image of the pathological tissue. This invention first utilizes a graph construction method based on feature similarity to generate a graph structure that better reflects the biological semantics of tissues. Then, through an innovative dual-path architecture and a spatial-frequency cross-attention mechanism, it achieves deep and dynamic fusion of spatial and frequency domain features. This allows the model to more fully utilize spatial morphological details and frequency global patterns, resulting in more discriminative feature representations. When facing challenges such as the high heterogeneity of pathological tissues, staining differences, and subtle inter-class differences, it exhibits stronger discrimination and generalization abilities, leading to more comprehensive and accurate pathological image classification. Based on this, the embodiments of this invention can more balancedly and effectively handle the high heterogeneity and complex structure of pathological tissues, improving classification performance and generalization ability, and enhancing the accuracy and robustness of classifying complex pathological images. Attached Figure Description
[0016] Figure 1A This is the main flowchart of a pathological image classification method based on a dual-domain collaborative graph attention network provided in an embodiment of the present invention; Figure 1B This is a schematic diagram of the overall framework of a space-frequency graph attention network model provided in one embodiment of the present invention; Figure 2This is a sub-flowchart of step S102 provided in one embodiment of the present invention; Figure 3 This is a sub-flowchart of step S210 provided in one embodiment of the present invention; Figure 4 This is a sub-flowchart of step S220 provided in one embodiment of the present invention; Figure 5 This is a sub-flowchart of step S103 provided in one embodiment of the present invention; Figure 6 This is a sub-flowchart of step S105 provided in one embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of a pathological image classification device based on a dual-domain collaborative graph attention network according to an embodiment of the present invention; Figure 8 This is a schematic diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0018] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the following drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0019] In this embodiment of the invention, the terms "furthermore," "exemplarily," or "optionally" are used as examples, illustrations, or descriptions and should not be construed as being more preferred or advantageous than other embodiments or designs. The use of the terms "furthermore," "exemplarily," or "optionally" is intended to present the relevant concepts in a specific manner.
[0020] To facilitate a more convenient description of the working principle of the embodiments of the present invention, the following introduction of relevant technical scenarios is given first.
[0021] Breast cancer is the most common malignant tumor among women worldwide, with persistently high incidence and mortality rates, posing a serious challenge to public health. In the clinical diagnosis and treatment of breast cancer, accurate molecular subtyping is the core basis for implementing personalized targeted therapy. Among these, the status of human epidermal growth factor receptor 2 (HER2) is one of the key molecular markers, directly determining whether patients can benefit from targeted drugs such as Herceptin. Therefore, accurate and efficient prediction of HER2 status is crucial for improving patient prognosis and survival rates.
[0022] Currently, the gold standard for clinical HER2 status interpretation relies on manual microscopic observation of whole-slide images (WSI) by physicians, combined with specialized staining and molecular detection techniques such as immunohistochemistry (IHC) and fluorescence in situ hybridization (FISH). However, whole-slide images typically have gigapixel-level ultra-high resolution but lack pixel-level annotations, posing a challenge to automated analysis. While deep learning has made significant progress in WSI analysis in recent years, existing deep learning models struggle to simultaneously ensure accurate identification of key local regions and grasp of globally consistent patterns when dealing with highly heterogeneous pathological tissues, affecting the accuracy and robustness of the final classification.
[0023] Based on this, the present invention provides a pathological image classification method and related apparatus based on a dual-domain collaborative graph attention network. The pathological image classification method based on the dual-domain collaborative graph attention network includes: acquiring whole-slice images of pathological tissue; preprocessing the whole-slice images to obtain instance feature matrices and instance coordinate matrices; constructing a target graph structure from the instance feature matrices and instance coordinate matrices; performing graph-based self-attention encoding on instances according to the target graph structure using a self-attention mechanism to obtain a graph feature matrix calculated by self-attention; performing frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features; performing spatial-frequency domain cross-attention fusion on the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; performing feature aggregation on the instance-level features to obtain a global feature vector; and inputting the global feature vector into a classifier to predict the classification of molecular marker states to obtain the classification prediction result for the corresponding whole-slice image of the pathological tissue. This invention first utilizes a graph construction method based on feature similarity to generate a graph structure that better reflects the biological semantics of tissues. Then, through an innovative dual-path architecture and a spatial-frequency cross-attention mechanism, it achieves deep and dynamic fusion of spatial and frequency domain features. This allows the model to more fully utilize spatial morphological details and frequency global patterns, resulting in more discriminative feature representations. When facing challenges such as the high heterogeneity of pathological tissues, staining differences, and subtle inter-class differences, it exhibits stronger discrimination and generalization abilities, leading to more comprehensive and accurate pathological image classification. Based on this, the embodiments of this invention can more balancedly and effectively handle the high heterogeneity and complex structure of pathological tissues, improving classification performance and generalization ability, and enhancing the accuracy and robustness of classifying complex pathological images.
[0024] The embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0025] like Figure 1A As shown, Figure 1A This is a flowchart of a pathological image classification method based on a dual-domain collaborative graph attention network provided in an embodiment of the present invention. The pathological image classification method based on the dual-domain collaborative graph attention network may include, but is not limited to, steps S101 to S108.
[0026] Step S101: Obtain whole pathological slide images; Step S102: Preprocess the whole pathological slide image to obtain the instance feature matrix and instance coordinate matrix; Step S103: Construct a graph from the instance feature matrix and instance coordinate matrix to obtain the target graph structure; Step S104: Based on the target graph structure, the instance is encoded using a self-attention mechanism to obtain the graph feature matrix after self-attention calculation. Step S105: Perform frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features; Step S106: Perform spatial-frequency domain cross-attention fusion on the graph feature matrix and frequency domain enhanced features to obtain instance-level features; Step S107: Perform feature aggregation on instance-level features to obtain a global feature vector; Step S108: Input the global feature vector into the classifier to perform classification prediction of molecular marker status, and obtain the classification prediction result of the corresponding pathological whole slice image.
[0027] It is understandable that, such as Figure 1B As shown, this invention employs an innovative spatial-frequency map attention network model, which deeply fuses spatial topological information and global features in the frequency domain through a dual-path architecture to achieve efficient and robust analysis of complex pathological images. The overall framework of this model can be divided into two main stages: a preprocessing stage and a model inference stage.
[0028] Understandably, the goal of the preprocessing stage is to convert a raw WSI image with a resolution of tens of billions of pixels into structured data that can be efficiently processed by the model, namely the depth features and spatial coordinates of a series of image patches (instances).
[0029] like Figure 2 As shown, step S102 includes the following steps: Step S210: Perform instance segmentation on the whole pathological slide image to obtain multiple image block instances; Step S220: Extract the depth feature vector of each image patch instance to obtain the instance feature matrix and instance coordinate matrix.
[0030] Understandably, instance segmentation of whole-slice pathological images aims to extract all fixed-size image patches (instances) representing valid tissue regions from a massive WSI. Since directly processing the entire WSI is not feasible, it is necessary to first separate the tissue regions (foreground) from the blank background.
[0031] like Figure 3 As shown, step S210 includes the following steps: Step S310: Convert the pathological whole slide image from RGB color space to HSV color space to obtain the first whole slide image; Step S320: Perform median filtering on the first full-slice image to reduce noise, and obtain the second full-slice image; Step S330: Threshold segmentation is performed on the second full-slice image to obtain the tissue region and background; Step S340: Perform morphological operations on the tissue region and background to obtain a foreground mask; Step S350: Divide the area covered by the foreground mask into grids. Each grid cell covered by the foreground mask is taken as an instance. Obtain multiple image block instances corresponding to the instances and record the center coordinates of the multiple image block instances in the pathological whole slice image.
[0032] Understandably, for image preprocessing and foreground separation: First, the WSI color space is converted from RGB to HSV color space because HSV is more robust to changes in color brightness and saturation, which is beneficial for subsequent segmentation. Next, medium-level filtering is applied to reduce image noise. Then, a threshold segmentation algorithm is used to automatically determine an optimal threshold, binarizing the image into tissue regions (foreground) and background. To obtain a cleaner foreground mask, morphological operations are also performed, such as closing operations to fill small holes and opening operations to remove isolated points. Finally, this invention yields a binary mask that accurately identifies all tissue regions.
[0033] Understandably, for the meshed instance: within the area covered by the foreground mask, this invention lays out a regular grid of fixed size (e.g., 512 pixels × 512 pixels). Each grid cell covered by the foreground is considered a valid instance. This invention saves the original image patch corresponding to the instance and records its center coordinates (x, y) within the entire WSI. In this way, one WSI is transformed into N instance images. and its coordinate set p.
[0034] like Figure 4 As shown, step S220 includes the following steps: Step S410: Convert each image patch instance into a dense feature vector; Step S420: Using the pre-trained ResNet50 feature extraction model, extract the depth feature vector of each image patch instance from the dense feature vector to obtain the instance feature matrix and instance coordinate matrix.
[0035] Understandably, to leverage the powerful representational capabilities of deep learning, this invention requires converting each image patch instance into a dense feature vector. In one specific embodiment, this invention employs a ResNet50 convolutional neural network pre-trained on a large natural image dataset (such as ImageNet) as a feature extractor. Each 512×512 instance image patch is input into ResNet50, its last fully connected classification layer is removed, and the output of the last pooling or convolutional layer is taken as the feature representation of that instance. Typically, this yields a 1024-dimensional deep feature vector. Ultimately, the entire WSI is represented as a feature matrix. and coordinate matrix .
[0036] At this point, the preprocessing stage is complete, and the output is... This serves as the input for the subsequent model inference stage.
[0037] During the model inference stage, the feature matrix output from the preprocessing is received. and coordinates This invention employs an innovative spatial-frequency graph attention model for feature learning and classification decisions. The spatial-frequency graph attention model comprises a dual-path system consisting of a graph structure analysis module and a frequency domain component analysis module, which are deeply integrated through a cross-attention mechanism.
[0038] In the model inference phase, the goal of graph construction and spatial feature learning is to construct a graph structure that reflects the semantic relationships between instances and to learn spatial domain features.
[0039] like Figure 5 As shown, step S103 includes the following steps: Step S510: Based on the clustering algorithm, perform clustering operations on the instances within the instance feature matrix, using the instance coordinate matrix as a reference to construct a graph structure; Step S520: The instance is divided into clusters based on the cluster center through clustering operations; Step S530: Construct the intra-cluster edge relationships and the weight relationships between instances in different clusters to obtain the target graph structure.
[0040] It is understandable that the graph construction based on feature similarity in this invention is a significant improvement over traditional graph construction based on spatial distance. Firstly, clustering algorithms such as K-Means can be used to construct the instance feature matrix. Clustering is performed, where instances are referenced by coordinate P to construct graph structure information, aiming to group instances with similar shapes and textures into the same cluster. The target number of clusters is set to [value missing]. After clustering, each instance is treated as a graph. A node in the graph. Edges of the graph. Based on clustering results: For two instance nodes belonging to the same cluster, a connection is established with high probability or directly; for instances from different clusters, no connection is established or a connection is established with low probability. Edge weights. It can be set to be proportional to the similarity between instance features (such as cosine similarity), which can be expressed as a formula. ,in Represents Euclidean distance. This is represented as the feature vector of instance i. The target graph structure constructed by this method has a topological structure dominated by feature similarity (semantics), which can better capture the regional associations related to biological functions.
[0041] It is understood that the pathological image graph construction method based on feature similarity adopted in this invention establishes connections between nodes (instances) in the graph structure based on the deep feature similarity of instances (such as through clustering algorithms), rather than relying solely on the proximity of spatial coordinates. Specifically, the implementation includes: clustering instance features extracted from whole-slice images (e.g., K-Means), grouping instances with similar features into the same cluster; constructing a graph based on this clustering result, where instance nodes belonging to the same cluster are connected with high probability or determinism, and edges can be assigned weights related to feature similarity. The graph structure generated by this method can more accurately reflect the semantic associations and biological characteristics of pathological tissues.
[0042] Understandably, for step S104, a self-attention mechanism is used to perform graph-based self-attention encoding on instances according to the target graph structure, in the constructed graph... Above, a graph attention network is applied for feature propagation and aggregation. For each target node, its new features are obtained by summing the attention-weighted features of its neighboring nodes. Specifically, the node features are first linearly projected into a query vector. Key vector Sum value vector Then, the attention scores between the target node and each source neighbor node are calculated. ,in This refers to the feature dimension. Finally, the aggregated features are... ,in This represents the set of neighbors of the target node. This process allows the model to dynamically focus on important neighboring nodes and output features rich in spatial-semantic relationships. .
[0043] During the model inference phase, frequency domain feature learning and enhancement are performed. The features are transformed and enhanced in the frequency domain to extract global texture patterns that are robust to coloring and noise.
[0044] like Figure 6 As shown, step S105 includes the following steps: Step S610: Map the instance feature matrix from the spatial domain to the frequency domain using a fast Fourier transform to obtain the spectral representation of the corresponding instance feature matrix, wherein the spectral representation includes the amplitude spectrum and the phase spectrum. Step S620: Perform spectral filtering and enhancement on the amplitude spectrum to obtain the target amplitude spectrum; Step S630: The target amplitude spectrum and phase spectrum are recombined to obtain an enhanced spectrum representation; Step S640: The enhanced spectral representation is mapped back to the spatial domain using the inverse fast Fourier transform to obtain the frequency domain enhanced features.
[0045] It is understandable that, for frequency domain transformation, the original instance feature matrix... (Or features that have undergone preliminary transformation) are mapped from the spatial domain to the frequency domain using a Fast Fourier Transform to obtain its spectral representation. The spectrum can be decomposed into two parts: amplitude spectrum and phase spectrum. The amplitude spectrum reflects the intensity (energy distribution) of various frequency components in the image and is related to the roughness and fineness of the texture; the phase spectrum contains spatial location information of the structure in the image.
[0046] Understandably, this is the core of the frequency path for spectral filtering and enhancement. A learnable spectral filter W_filter (whose parameters are optimized during training) is designed and applied to the amplitude spectrum for element-wise modulation. ,in This represents the Hadamard product (element-by-element multiplication). This indicates amplitude taking. This operation adaptively enhances frequency components relevant to pathological diagnosis (such as frequencies corresponding to specific cell arrangement patterns) while suppressing irrelevant or interfering frequency components introduced by staining differences or noise. Phase spectrum It will remain unchanged.
[0047] It is understandable that, for inverse transform and feature reconstruction, the enhanced amplitude spectrum will be... Compared with the original phase spectrum The components are recombined to form the enhanced spectrum. Then, it is mapped back to the spatial domain using an inverse fast Fourier transform to obtain the enhanced feature representation in the frequency domain. The entire process can be formally represented as: ,in Indicates the inverse Fourier transform. It is the imaginary unit.
[0048] It is understandable that in step S106, for spatial-frequency cross-attention fusion, the features output by the spatial path can be... As a query vector, the enhanced features output by the frequency path Both the key vector and value vector are input into a cross-attention module. This module calculates... and Attention weights are assigned between these features, allowing spatial features to dynamically retrieve and extract relevant global texture information from frequency features to complement themselves. The output is the fused instance-level features. It is a refined feature that integrates spatial topological context and frequency domain global mode. This process enables deep interaction and collaboration of information between the two paths.
[0049] It is understood that the spatial-frequency graph attention network model of the present invention includes a spatial-frequency cross-attention module and a parallel dual-path processing flow.
[0050] The spatial-frequency cross-attention module is the core of the model architecture, aiming to achieve deep and dynamic fusion of spatial and frequency domain features. Specifically, it involves: using features extracted from the spatial path as query vectors; mapping the original or intermediate features to the frequency domain via Fast Fourier Transform (FFT), adaptively modulating the amplitude spectrum using a learnable spectral filter to enhance discriminative components, and then reconstructing the data using an inverse transform as key and value vectors; finally, through the spatial-frequency cross-attention mechanism, enabling spatial features to dynamically retrieve and fuse global texture and pattern information from the enhanced frequency domain features. The spatial-frequency cross-attention module achieves collaborative optimization of spatial topological context and frequency domain global representation.
[0051] The parallel dual-path processing flow specifically includes a spatial path and a frequency path. The spatial path performs the aforementioned feature similarity-based graph construction and graph attention encoding; the frequency path performs frequency domain transformation, learnable filtering, and reconstruction on the features. By introducing a spatial-frequency cross-attention mechanism, which does not simply concatenate spatial and frequency domain features, but allows features learned by the spatial path to dynamically query and retrieve relevant information from the enhanced features of the frequency domain path, achieving deep fusion and complementarity of the two types of features at the representation level, thereby making fuller use of spatial topological relationships and frequency domain global texture information. It should be noted that the spatial path (especially the graph attention mechanism) focuses on capturing the spatial-semantic relationships and morphological details of local regions; the frequency path enhances the robustness of global patterns to staining and texture changes by performing spectral analysis and filtering on the features. The deep synergy between the two enables the model to handle the high heterogeneity and complex structure of pathological tissues more balancedly and effectively, improving classification performance and generalization ability.
[0052] It is understandable that in step S107, the fused instance-level features are... They are aggregated into a global feature vector representing the entire WSI.
[0053] Understandably, in step S108, this invention employs an aggregation method based on an attention mechanism. This mechanism learns a weight distribution, automatically assigns an importance score to each instance feature, and then performs a weighted summation: ,in It is the first Attention weights for each instance. Finally, the WSI-level global feature vector is... Input a classifier consisting of a fully connected layer and a softmax function. The classifier outputs the probability distribution of the WSI belonging to each target category (e.g., HER2 positive, HER2 negative), thus completing the automated classification and prediction of molecular marker status.
[0054] Based on this, the spatial-frequency graph attention network model of the present invention achieves end-to-end classification prediction of pathological whole slice images.
[0055] Understandably, this invention not only automates the analysis process from inputting raw whole-slice images to outputting molecular marker classification results, but also generates structured diagnostic reports based on the classification prediction results output by the spatial-frequency map attention network model. Furthermore, by utilizing the attention weights within the model (such as image attention weights and instance aggregation attention weights), a visual heatmap is generated and overlaid onto the original WSI, highlighting the key pathological regions upon which the model's decisions are based, providing pathologists with intuitive and interpretable auxiliary diagnostic evidence.
[0056] In addition, such as Figure 7 As shown, one embodiment of the present invention also discloses a pathological image classification device based on a dual-domain collaborative graph attention network, the device comprising: The acquisition module 110 is used to acquire whole pathological slide images; Preprocessing module 120 is used to preprocess the whole pathological slide image to obtain the instance feature matrix and instance coordinate matrix; Graph construction module 130 is used to construct a graph from the instance feature matrix and the instance coordinate matrix to obtain the target graph structure; The encoding module 140 is used to perform graph-based self-attention encoding on instances according to the target graph structure using a self-attention mechanism, so as to obtain a graph feature matrix calculated by self-attention. Enhancement module 150 is used to perform frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features; The fusion module 160 is used to perform spatial-frequency domain cross-attention fusion of the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; The aggregation module 170 is used to aggregate instance-level features to obtain a global feature vector. The output module 180 is used to input the global feature vector into the classifier to perform classification prediction of molecular marker status and obtain the classification prediction result of the corresponding pathological whole slice image.
[0057] The device can be deployed on hospital servers, high-performance workstations, or cloud computing platforms, and can be seamlessly integrated with existing image archiving and communication systems to form a complete intelligent pathology-assisted diagnostic workflow, enabling an automated analysis process from input of raw whole-slice images to output of molecular marker classification results.
[0058] The pathological image classification device based on a dual-domain cooperative graph attention network in this embodiment of the invention is used to execute the pathological image classification method based on a dual-domain cooperative graph attention network in the above embodiment. Its specific processing procedure is the same as that of the pathological image classification method based on a dual-domain cooperative graph attention network in the above embodiment, and will not be described in detail here.
[0059] In addition, such as Figure 8 As shown, one embodiment of the present invention also discloses an electronic device, including: at least one processor 210; at least one memory 220 for storing at least one program; when the at least one program is executed by the at least one processor 210, it implements the pathological image classification method based on dual-domain collaborative graph attention network as in any of the preceding embodiments.
[0060] The electronic device includes, but is not limited to, hardware components such as processor 210 and memory 220, and is configured to load and run the program to execute the solution of the present invention, such as a medical image analysis server, a diagnostic workstation, or an intelligent pathology scanning platform.
[0061] In addition, one embodiment of the present invention discloses a computer-readable storage medium storing computer-executable instructions for performing a pathological image classification method based on a dual-domain collaborative graph attention network as described in any of the preceding embodiments.
[0062] The system architecture and application scenarios described in the embodiments of this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.
[0063] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0064] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.
[0065] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process or execution thread, and components may be located on a single computer or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, or a network, such as the Internet interacting with other systems via signals).
Claims
1. A pathological image classification method based on a dual-domain collaborative graph attention network, comprising: Obtain whole-section images of pathological tissue; The pathological whole-section image is preprocessed to obtain the instance feature matrix and instance coordinate matrix; Graph construction is performed on the instance feature matrix and the instance coordinate matrix to obtain the target graph structure; Based on the target graph structure, a self-attention mechanism is used to encode instances using graph-based self-attention, resulting in a graph feature matrix calculated by self-attention. The frequency domain transformation and enhancement are performed on the instance feature matrix to obtain frequency domain enhanced features; Spatial-frequency domain cross-attention fusion is performed on the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; The instance-level features are aggregated to obtain a global feature vector; The global feature vector is input into a classifier to predict the molecular marker status, thereby obtaining the classification prediction result corresponding to the pathological whole slice image.
2. The method according to claim 1, characterized in that, The preprocessing of the pathological whole-slice image to obtain the instance feature matrix and instance coordinate matrix includes: The pathological whole-section image is segmented into multiple image block instances; Extract the depth feature vector of each image patch instance to obtain the instance feature matrix and the instance coordinate matrix.
3. The method according to claim 2, characterized in that, The process of segmenting the whole pathological slide image into multiple image patch instances includes: The pathological whole-section image is converted from the RGB color space to the HSV color space to obtain the first whole-section image; The first full-slice image is subjected to median filtering for noise reduction to obtain the second full-slice image; The second full-slice image is subjected to threshold segmentation to obtain the tissue region and background; Morphological operations are performed on the tissue region and the background to obtain a foreground mask; The area covered by the foreground mask is divided into grids, and each grid cell covered by the foreground mask is taken as an instance. Multiple image block instances corresponding to the instances are obtained, and the center coordinates of the multiple image block instances in the pathological whole slice image are recorded.
4. The method according to claim 2, characterized in that, The step of extracting the depth feature vector of each image patch instance to obtain the instance feature matrix and the instance coordinate matrix includes: Each of the image patch instances is converted into a dense feature vector; The deep feature vector of each image patch instance is extracted from the dense feature vector using a pre-trained ResNet50 feature extraction model, resulting in the instance feature matrix and the instance coordinate matrix.
5. The method according to claim 1, characterized in that, The step of constructing a graph from the instance feature matrix and the instance coordinate matrix to obtain the target graph structure includes: Clustering operations are performed on instances within the instance feature matrix based on the clustering algorithm, using the instance coordinate matrix as a reference to construct a graph structure. The instances are divided into clusters based on their cluster centers using clustering operations. The target graph structure is obtained by constructing intra-cluster edge relationships and weight relationships between instances in different clusters.
6. The method according to claim 1, characterized in that, The step of performing frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features includes: The instance feature matrix is mapped from the spatial domain to the frequency domain using a fast Fourier transform to obtain the spectral representation of the instance feature matrix, wherein the spectral representation includes an amplitude spectrum and a phase spectrum; The amplitude spectrum is subjected to spectral filtering and enhancement to obtain the target amplitude spectrum; The target amplitude spectrum and phase spectrum are recombined to obtain an enhanced spectral representation; The enhanced spectral representation is mapped back to the spatial domain using an inverse fast Fourier transform to obtain the frequency domain enhanced features.
7. The method according to claim 1, characterized in that, Also includes: A visual diagnostic report is generated based on the classification prediction results, wherein the classification prediction results include the classification results and confidence levels of molecular markers, and the classifier consists of a fully connected layer and a Softmax function.
8. A pathological image classification device based on a dual-domain cooperative graph attention network, characterized in that, The device includes: The acquisition module is used to acquire images of whole pathological slides; The preprocessing module is used to preprocess the pathological whole slide image to obtain the instance feature matrix and instance coordinate matrix; The graph construction module is used to construct a graph from the instance feature matrix and the instance coordinate matrix to obtain the target graph structure. The encoding module is used to perform graph-based self-attention encoding on instances according to the target graph structure using a self-attention mechanism, so as to obtain a graph feature matrix calculated by self-attention. The enhancement module is used to perform frequency domain transformation and enhancement on the instance feature matrix to obtain frequency domain enhanced features; The fusion module is used to perform spatial-frequency domain cross-attention fusion on the graph feature matrix and the frequency domain enhanced features to obtain instance-level features; The aggregation module is used to aggregate the instance-level features to obtain a global feature vector; The output module is used to input the global feature vector into the classifier to perform classification prediction of molecular marker states, and obtain the classification prediction result corresponding to the pathological whole slice image.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the pathological image classification method based on a dual-domain collaborative graph attention network as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing computer-executable instructions for performing the pathological image classification method based on a dual-domain collaborative graph attention network as described in any one of claims 1 to 7.