Pathological image classification method and system based on graph neural feature fusion, terminal and storage medium

By employing a graph neural feature fusion method, a pyramid-shaped multi-scale feature fusion model, and a dynamic graph construction mechanism, the problem of insufficient multi-scale pathological feature fusion in existing pathological image classification methods is solved, achieving efficient and accurate pathological image classification.

CN121095670APending Publication Date: 2025-12-09SHENZHEN TECH UNIV
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202511263391.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-05
Publication Date
2025-12-09

AI Technical Summary

Technical Problem

Existing pathological image classification methods struggle to achieve efficient fusion and accurate classification of multi-scale pathological features, resulting in high computational complexity and large hardware resource requirements, failing to meet the needs of low-cost and efficient clinical applications.

Method used

A graph neural network-based feature fusion method is adopted. By constructing an initial graph neural network model through a pyramid-shaped multi-scale feature fusion model and a dynamic graph construction mechanism, combined with a self-attention mechanism and a channel reduction mechanism, an efficient fusion and accurate classification of multi-scale pathological features are achieved.

Benefits of technology

It achieves efficient, accurate, and low-cost classification of pathological images, reduces computational complexity and hardware resource requirements, and improves the classification performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121095670A_ABST
    Figure CN121095670A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, and discloses a pathological image classification method and system based on graph neural feature fusion, a terminal and a storage medium, and the method comprises the steps: obtaining multi-scale digital image data, carrying out the extraction of the multi-scale digital image data, obtaining a plurality of image features, fusing the plurality of image features through a pyramid-shaped multi-scale feature fusion model to obtain a multi-resolution feature; obtaining a slide level label, constructing an initial graph neural network model based on a dynamic graph construction mechanism, a self-attention mechanism, a channel reduction mechanism and multi-resolution features, and training the graph neural network model according to the slide level label to obtain a target graph neural network model; and obtaining to-be-detected pathological image features, inputting the to-be-detected pathological image features into the target image neural network model for classification, and outputting a classification result. According to the method, features of the to-be-detected pathological image are classified according to the target image neural network model, and efficient and accurate classification is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, terminal, and computer-readable storage medium for classifying pathological images based on graph neural feature fusion. Background Technology

[0002] Current single-scale CNN architectures (such as ResNet50) struggle to address the inherent scale contradictions in pathological images: identifying individual tumor cells (approximately 10 μm in size) under high magnification (40×) requires local feature focusing, while analyzing tissue structures (over 10 mm in size) under low magnification (4×) presents challenges. Therefore, single-scale networks cannot comprehensively capture tissue structures and cellular details at different magnifications. Existing multi-scale feature fusion methods often employ simple stitching or stepwise downsampling strategies, leading to fragmentation of crucial biological information. Secondly, computational performance bottlenecks hinder clinical implementation. The sheer size of a single WSI (Whole Slide Image) places stringent demands on hardware resources. For example, processing a single WSI requires 32GB of video memory (e.g., ViT-L architecture), far exceeding the limits of low-end devices. Performance degradation is particularly pronounced when the algorithm is deployed on low-end devices (e.g., graphics cards with 8GB of video memory). Even with high-end GPUs, the inference time per case still exceeds 80 seconds (for breast cancer lymph node metastasis screening), which cannot meet the timeliness requirement of intraoperative frozen pathology results within 30 minutes.

[0003] However, existing pathological image classification methods have the following shortcomings: traditional single-scale convolutional neural networks are difficult to effectively extract multi-scale tissue and cell features from pathological sections, leading to the loss of key biological information; existing multi-scale feature fusion methods mostly adopt simple feature splicing or stepwise downsampling strategies, which cannot fully utilize the complementary information of images at different magnifications, thus limiting the classification performance of the model; in addition, although models based on self-attention mechanisms can capture long-distance dependencies, they have high computational complexity and large hardware resource requirements, making them difficult to apply in reality. As a result, existing pathological image classification methods cannot achieve efficient fusion and accurate classification of multi-scale pathological features, making it difficult to classify pathological images efficiently, accurately, and at low cost, which has become an urgent problem to be solved.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a pathological image classification method, system, terminal, and computer-readable storage medium based on graph neural feature fusion. This invention aims to solve the problem that existing pathological image classification methods cannot achieve efficient fusion and accurate classification of multi-scale pathological features, making it difficult to classify pathological images efficiently, accurately, and at low cost.

[0006] To achieve the above objectives, the present invention provides a pathological image classification method based on graph neural feature fusion, the pathological image classification method based on graph neural feature fusion comprising the following steps:

[0007] Multi-scale digital image data is acquired, and multiple image features are extracted from the multi-scale digital image data according to a preset magnification ratio. The multiple image features are then fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0008] Obtain slide-level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, and the multi-resolution features, and train the graph neural network model according to the slide-level labels to obtain the target graph neural network model;

[0009] The features of the pathological image to be detected are obtained, and the features are input into the target graph neural network model for classification, and the classification result of the features of the pathological image to be detected is output.

[0010] Optionally, in the pathological image classification method based on graph neural feature fusion, the preset magnification includes low magnification, medium magnification, and high magnification.

[0011] The image features include low-scale image features, medium-scale image features, and high-scale image features;

[0012] The process of acquiring multi-scale digital image data involves extracting multiple image features from the multi-scale digital image data according to a preset magnification ratio, and fusing these multiple image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features. Specifically, this includes:

[0013] Acquire multi-scale digital image data, and extract low-scale image features from the multi-scale digital image data based on low magnification.

[0014] Based on the medium magnification, the multi-scale digital image data is extracted to obtain the medium-scale image features;

[0015] High-scale image features are obtained by extracting the multi-scale digital image data based on high magnification.

[0016] The low-scale image features, the medium-scale image features, and the high-scale image features are fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0017] Optionally, the pathological image classification method based on graph neural network feature fusion, wherein the fusion of low-scale image features, mid-scale image features, and high-scale image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features specifically includes:

[0018] The attention weights of the pyramid-shaped multi-scale feature fusion model are dynamically calculated for the low magnification, the medium magnification, and the high magnification to obtain complementary information;

[0019] The low-scale image features, the medium-scale image features, and the high-scale image features are fused based on the complementary information to obtain multi-resolution features.

[0020] Optionally, the pathological image classification method based on graph neural network feature fusion, wherein fusing the low-scale image features, the mid-scale image features, and the high-scale image features according to the complementary information to obtain multi-resolution features specifically includes:

[0021] The low-scale image features, the medium-scale image features, and the high-scale image features are transformed to obtain multiple one-dimensional sequences. All the one-dimensional sequences are sorted to obtain a two-dimensional grid.

[0022] The two-dimensional grid is input into the pyramid-shaped multi-scale feature fusion model to determine the hierarchical relationship;

[0023] Based on the pyramid-shaped multi-scale feature fusion model, the low-scale image features, the medium-scale image features, and the high-scale image features are fused according to the complementary information and the hierarchical relationship to obtain multi-resolution features.

[0024] Optionally, the pathological image classification method based on graph neural network feature fusion, wherein obtaining slide-level labels, constructing an initial graph neural network model based on a dynamic graph construction mechanism, a self-attention mechanism, a channel reduction mechanism, and the multi-resolution features, and training the graph neural network model according to the slide-level labels to obtain a target graph neural network model, specifically includes:

[0025] Based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and the multi-resolution features, the graph structure, node information, and one-dimensional information are determined, and an initial graph neural network model is constructed based on the graph structure, node information, and one-dimensional information.

[0026] Obtain slide level labels, and perform weakly supervised training on the parameters of the graph neural network model based on the slide level labels and the backpropagation algorithm to obtain the target graph neural network model.

[0027] Optionally, the pathological image classification method based on graph neural network feature fusion, wherein the construction of the initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and the multi-resolution features specifically includes:

[0028] Based on the dynamic graph construction mechanism, multiple sub-features of the multi-resolution features are connected according to the spatial positional relationship of the multi-resolution features to obtain a graph structure;

[0029] Based on the self-attention mechanism, the triplet structure of the graph structure is calculated according to a preset joint attention scoring strategy to obtain node information;

[0030] Based on the channel reduction mechanism, the node information is compressed to obtain one-dimensional information;

[0031] Based on the dynamic nonlinear function, an initial graph neural network model is constructed according to the graph structure, the node information, and the one-dimensional information.

[0032] Optionally, in the aforementioned pathological image classification method based on graph neural feature fusion, the compression of the node information to obtain one-dimensional information based on the channel reduction mechanism specifically includes:

[0033] Obtain the query request and feature tags. Based on the channel reduction mechanism, compress the node information according to the query request and feature tags to obtain a one-dimensional scalar.

[0034] Based on the spatial dimension, the one-dimensional scalar is averaged and pooled to obtain one-dimensional information.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a pathological image classification system based on graph neural feature fusion, wherein the pathological image classification system based on graph neural feature fusion is as follows:

[0036] The feature fusion module is used to acquire multi-scale digital image data, extract multiple image features from the multi-scale digital image data according to a preset magnification, and fuse the multiple image features through a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0037] The model training module is used to obtain slide-level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism and the multi-resolution features, and train the graph neural network model according to the slide-level labels to obtain the target graph neural network model.

[0038] The model prediction module is used to acquire the features of the pathological image to be detected, input the features of the pathological image to be detected into the target graph neural network model for classification, and output the classification result of the features of the pathological image to be detected.

[0039] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a pathological image classification program based on graph neural feature fusion, and the pathological image classification program based on graph neural feature fusion, when executed by a processor, implements the steps of the pathological image classification method based on graph neural feature fusion as described above.

[0040] In this invention, multi-scale digital image data is acquired, and multiple image features are extracted from the multi-scale digital image data according to a preset magnification. These multiple image features are then fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features. Slide-level labels are acquired, and an initial graph neural network model is constructed based on a dynamic graph construction mechanism, a self-attention mechanism, a channel reduction mechanism, and the multi-resolution features. The graph neural network model is trained using the slide-level labels to obtain a target graph neural network model. Features of a pathological image to be detected are acquired, and these features are input into the target graph neural network model for classification. The classification result of the pathological image features to be detected is then output. This invention achieves efficient and accurate classification by classifying the features of the pathological image to be detected using the target graph neural network model. Attached Figure Description

[0041] Figure 1 This is a flowchart of a preferred embodiment of the pathological image classification method based on graph neural feature fusion of the present invention;

[0042] Figure 2 This is a schematic diagram of the MSFF module of a preferred embodiment of the pathological image classification method based on graph neural feature fusion of the present invention;

[0043] Figure 3 This is a structural diagram of CRA channel reduction in a preferred embodiment of the pathological image classification method based on graph neural feature fusion of the present invention;

[0044] Figure 4 This is a structural diagram of a preferred embodiment of the pathological image classification system based on graph neural feature fusion of the present invention;

[0045] Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the device of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0047] Existing pathological image classification methods suffer from the following shortcomings: traditional single-scale convolutional neural networks struggle to effectively extract multi-scale tissue and cellular features from pathological sections, leading to the loss of crucial biological information; existing multi-scale feature fusion methods often employ simple feature stitching or stepwise downsampling strategies, failing to fully utilize the complementary information of images at different magnifications, thus limiting the model's classification performance; furthermore, while models based on self-attention mechanisms can capture long-distance dependencies, their high computational complexity and large hardware resource requirements make them impractical for real-world applications. Consequently, existing pathological image classification methods cannot achieve efficient fusion and accurate classification of multi-scale pathological features, hindering efficient, accurate, and low-cost classification of pathological images. Therefore, a pathological image classification method based on graph neural network feature fusion is needed to achieve efficient, accurate, and low-cost classification of pathological images through efficient fusion and accurate classification of multi-scale pathological features.

[0048] The preferred embodiment of the pathological image classification method based on graph neural feature fusion described in this invention, such as... Figure 1 and Figure 2 As shown, the pathological image classification method based on graph neural feature fusion includes the following steps:

[0049] Step S10: Acquire multi-scale digital image data, extract multiple image features from the multi-scale digital image data according to a preset magnification, and fuse the multiple image features through a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0050] Step S10 includes:

[0051] Step S11: Acquire multi-scale digital image data, and extract low-scale image features from the multi-scale digital image data based on low magnification.

[0052] Step S12: Extract the multi-scale digital image data based on the medium magnification to obtain the medium-scale image features;

[0053] Step S13: Extract high-scale image features from the multi-scale digital image data based on the high magnification.

[0054] Step S14: The low-scale image features, the medium-scale image features, and the high-scale image features are fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0055] Specifically, multi-scale digital image data is acquired. Low-scale image features are extracted from the multi-scale digital image data at low magnification (scales are typically divided into top, middle, and bottom layers; the top layer refers to low magnification, such as 5x or 10x, providing an overview view of the entire slice with low resolution but a wide field of view, facilitating the location of regions of interest). Medium-scale image features are extracted from the multi-scale digital image data at medium magnification (the middle layer refers to medium magnification, such as 20x, balancing detail and field of view, used for analyzing cell populations or tissue structures). High-scale image features are extracted from the multi-scale digital image data at high magnification (the bottom layer refers to high magnification, such as 40x, providing high-resolution details, facilitating the observation of individual cells or subcellular structures (such as nuclear morphology)). The low-scale, middle-scale, and high-scale image features are then fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features (feature fusion model MSFF (Multi-Scale Feature)). Fusion (multi-scale feature fusion) dynamically adjusts the use of intermediate scales as a base and uses upsampling and downsampling methods to fuse multi-scale features to obtain multi-resolution features. The preset magnification includes low magnification, medium magnification and high magnification; the image features include low-scale image features, medium-scale image features and high-scale image features.

[0056] In this embodiment, for example, the WSI image from the dataset contains three layers of image data. The second layer (middle layer) is selected as the baseline. This layer shows the structure of the breast lymphocyte population. By upsampling the first layer (top layer), the breast lymphocyte population structure of the middle layer is combined with the overall tissue distribution of the slice, which is mainly used to identify the approximate location of the lesion tissue. The third layer (bottom layer) is downsampling, which combines the breast lymphocyte population structure of the middle layer with fine local details under high magnification, which can be used to detect single cell abnormalities or mitotic figures. Feature fusion is performed at different scales through upsampling and downsampling to further enhance the model's ability to extract features from multi-scale pathological images.

[0057] In this embodiment, a single WSI image consists of diseased tissue (such as tumor areas) and non-diseased tissue (such as normal stroma and blood vessels). To improve feature extraction capabilities and reduce computational overhead, the diseased tissue in the WSI image first needs to be automatically segmented to identify diseased tissue regions. For example, in lung tissue sections, irregular acinar structures and deeply stained atypical cell nuclei are automatically segmented; in colonic gland tissue sections, areas with crowded glandular ducts and reduced goblet cells are automatically segmented. The process involves converting pathological images to the HSV color space (HSV is a more effective color model for pathological image analysis than RGB, where H (Hue) physically represents color type (0°-360°), reflecting the characteristics of lesion tissue as concentrated hue in the lesion area (e.g., deep blue of cancerous nuclei ≈ 240°); S (Saturation) physically represents color purity (0%-100%), reflecting the characteristics of lesion tissue as deep staining and high saturation (>60%) of lesion cell nuclei; and V (Value) physically represents brightness (0%-100%), reflecting the characteristics of lesion tissue as low brightness in necrotic areas and high brightness in normal areas). A statistical histogram H(s) is then generated using the saturation channel (S). Subsequently, high-frequency, high-saturation intervals (i.e., lesion areas) are identified in the histogram, and a dynamic threshold is calculated using a peak function for thresholding. Areas above the threshold are given attention, while areas below the threshold are discarded, thus extracting the lesion tissue region. WSI's ultra-high resolution (up to billions of pixels) and massive size (single files often exceeding 1GB) make it impossible for it to be directly loaded by hardware or processed by deep learning models. Slicing decomposes images into smaller blocks hierarchically (e.g., sliding the top-level pathological image into several 256×256 pixel pathological slices), thereby overcoming the memory limitations of GPUs (Graphics Processing Units), enabling batch parallel computing, and reducing computational complexity to a level that the hardware can handle, facilitating subsequent image processing and model input.

[0058] Step S14 includes:

[0059] Step S141: Dynamically calculate the low magnification, medium magnification, and high magnification using the attention weights of the pyramid-shaped multi-scale feature fusion model to obtain complementary information;

[0060] Step S142: Based on the complementary information, fuse the low-scale image features, the medium-scale image features, and the high-scale image features to obtain multi-resolution features.

[0061] Specifically, the low magnification, medium magnification, and high magnification are dynamically calculated using the attention weights of the pyramid-shaped multi-scale feature fusion model to obtain complementary information (involving the use of a cross-attention module, with medium-scale (10x) features as the query and coarse / fine-scale features as the context, and the complementary information is dynamically fused using attention weights). Based on the complementary information, the low-scale image features, medium-scale image features, and high-scale image features are fused to obtain multi-resolution features.

[0062] Step S142 includes:

[0063] Step S1421: Transform the low-scale image features, the medium-scale image features, and the high-scale image features to obtain multiple one-dimensional sequences, and sort all the one-dimensional sequences to obtain a two-dimensional grid;

[0064] Step S1422: Input the two-dimensional mesh into the pyramid-shaped multi-scale feature fusion model to determine the hierarchical relationship;

[0065] Step S1423: Based on the pyramid-shaped multi-scale feature fusion model, the low-scale image features, the medium-scale image features, and the high-scale image features are fused according to the complementary information and the hierarchical relationship to obtain multi-resolution features.

[0066] Specifically, the low-scale image features, the mid-scale image features, and the high-scale image features are transformed to obtain multiple one-dimensional sequences (the process of converting the original two-dimensional / three-dimensional pathological image data into linear sequences, resulting in multiple one-dimensional sequences). All one-dimensional sequences are then sorted to obtain a two-dimensional grid. This two-dimensional grid is input into the pyramid-shaped multi-scale feature fusion model to determine the hierarchical relationships (rearranged into a two-dimensional grid, using Pyramidal Positional Encoding (PPEG)). The generator injects multi-scale spatial information (low-scale image features, mid-scale image features, and high-scale image features), and then combines it with the transformer of the pyramid-shaped multi-scale feature fusion model to capture long-range dependencies and preserve the hierarchical relationship from local to global. Based on the pyramid-shaped multi-scale feature fusion model, the low-scale image features, mid-scale image features, and high-scale image features are fused according to the complementary information and the hierarchical relationship to obtain multi-resolution features (multi-scale feature aggregation, stacking of features at each scale, initial fusion through 3D convolution, and parallel application of depth convolution with different kernel sizes (7×7, 5×5, 3×3, 1×1) to extract multi-resolution features). The multi-scale features (low-scale image features, mid-scale image features, and high-scale image features) are stored as h5 (Hierarchical Data Format version 5) files, which facilitates the computer loading training data into the video memory during training and improves training efficiency.

[0067] Step S20: Obtain slide level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism and the multi-resolution features, and train the graph neural network model according to the slide level labels to obtain the target graph neural network model.

[0068] Step S20 includes:

[0069] Step S21: Determine the graph structure, node information, and one-dimensional information based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and the multi-resolution features; and construct an initial graph neural network model based on the graph structure, node information, and one-dimensional information.

[0070] Step S22: Obtain slide level labels, and perform weakly supervised training on the parameters of the graph neural network model based on the slide level labels and the backpropagation algorithm to obtain the target graph neural network model.

[0071] Specifically, such as Figure 3As shown, based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and the multi-resolution features, the graph structure, node information, and one-dimensional information are determined. (Traditional self-attention (e.g., Transformer) has high computational complexity (O(N2C)) when processing high-resolution features. CRA (Channel-wise Recalibration Attention) achieves efficient global modeling through channel dimension reduction. The key idea is to reduce the channel dimension of query and key to one dimension (each header), thereby significantly reducing computational complexity. An initial graph neural network model is constructed based on the graph structure, node information, and one-dimensional information. Slide-level labels are obtained. Based on the slide-level labels and the backpropagation algorithm (training the initial graph neural network model based on the slide-level labels and updating the model parameters through backpropagation effectively solves the training bottleneck problem caused by the large scale of the pathological slide image dataset and further improves the model's classification performance for pathological slides), the parameters of the graph neural network model are weakly supervised to obtain the target graph neural network model.

[0072] In this embodiment, two pathological whole slide image (WSI) datasets were used for training, validation, and testing: Camelyon16 (a breast cancer dataset, mainly used for lymph node metastasis detection and algorithm evaluation) and The Cancer GenomeAtlas-LUNG (TCGA-LUNG, specifically referring to data collected from lung cancer patients, including multiple subtypes). Common classification model metrics such as accuracy, area under the curve (AUC), and F1-score were used to test the classification performance. The computation time, computational cost, and number of parameters of the model were tested using metrics such as time, GFLOPs (billions of floating-point operations), and Params (number of parameters). The coefficients of the target graph neural network model for the breast cancer dataset (Camelyon16) using the ResNet50 (Residual Network 50) neural network architecture with weights are as follows: 0.8698, 0.8391, and 0.8091; the coefficients for computation time, computational cost, and parameter count are as follows: 189.03, 1,278,542, and 17.34; the coefficients for the three classification models using UNI weights are as follows: 0.8938, 0.9143, and 0.8578; the coefficients for computation time, computational cost, and parameter count are as follows: 251.28, 1,856,158, and 21.98; and the average coefficients are as follows: 0.8818, 0.8767, 0.8335, 220.16, 1,567,350, and 19.66. The target graph neural network model, using ResNet50 weights for the TCGA-LUNG dataset, achieved the following coefficients for three classification metrics: 0.8635, 0.8867, and 0.8485; for computation time, computational cost, and parameter count: 103.31, 1266949, and 8.78; and for UNI weights: 0.8946, 0.9175, and 0.8546. The coefficients for the three quantitative indicators are 86.07, 1,377,447, and 7.98, respectively; the average coefficients are 0.8791, 0.9021, 0.8516, 94.69, 1,322,198, and 8.38, respectively. The classification performance of the target graph neural network model is superior to existing neural network models such as MobileNet and EfficientNet, and also superior to ViT (Visual Transformer) and Swin (Shift Window Transformer) which combine attention mechanisms. The experimental results demonstrate that this invention improves the model's classification performance for pathological images to a certain extent.

[0073] Step S21 includes:

[0074] Step S211: Based on the dynamic graph construction mechanism, connect multiple sub-features of the multi-resolution features according to the spatial positional relationship of the multi-resolution features to obtain a graph structure;

[0075] Step S212: Based on the self-attention mechanism, the triplet structure of the graph structure is calculated according to the preset joint attention scoring strategy to obtain node information;

[0076] Step S213: Based on the channel reduction mechanism, the node information is compressed to obtain one-dimensional information;

[0077] Step S214: Based on the dynamic nonlinear function, construct an initial graph neural network model according to the graph structure, the node information, and the one-dimensional information.

[0078] Specifically, based on the dynamic graph construction mechanism, multiple sub-features of the multi-resolution features are connected according to the spatial positional relationship of the multi-resolution features to obtain a graph structure (setting hyperparameter k, selecting the k neighbors with the highest similarity to each node i, and constructing edge embeddings based on this, the edge embeddings simultaneously contain the feature information (i.e., sub-features) of the source node and the target node, thus obtaining the graph structure). Based on the self-attention mechanism, the triplet structure of the graph structure (representing each graph structure (patch) as a head-tail embedding pair (h_i, t_i), where h_i represents the head embedding and t_i represents the tail embedding, dynamically constructing directed edge embeddings r_{i, j}, where r_{i, j} represents the relational feature from node i to node j, forming a (h, r, t) triplet structure) is calculated according to a preset joint attention scoring strategy to obtain node information (through joint attention scoring). The contribution of neighbors is calculated, and node features are updated using a dual-path approach of addition and element-wise multiplication. Based on the channel reduction mechanism, the node information is compressed to obtain one-dimensional information (channel reduction is introduced in the self-attention mechanism to compress the channel dimension of node information (query and key) to one-dimensional information). Based on the dynamic nonlinear function, an initial graph neural network model is constructed according to the graph structure, the node information, and the one-dimensional information (traditional neural network models rely on layer normalization (LayerNorm) for stable training, but it requires the calculation of mean and variance, which is inefficient. The DT (DynamicTanhmodule) module (formula: γ·tanh(αx)+β) replaces the normalization layer with dynamic nonlinearity, where α and γ represent learnable scaling parameters, and β represents learnable bias parameters).

[0079] In this embodiment, the channel reduction mechanism transforms the node information features of each pixel from a vector to a scalar (a single numerical value). This is specifically achieved through a linear projection layer, supplemented by spatial average pooling to reduce the number of tokens. The channel dimension is preserved to compensate for information loss (channel compression may lose the interaction information between channels, such as the collaborative features of the color and shape of lesion tissue), ensuring that even with scalarization, long-distance dependencies (such as associating scattered objects) can be modeled.

[0080] Step S30: Obtain the features of the pathological image to be detected, input the features of the pathological image to be detected into the target graph neural network model for classification, and output the classification result of the features of the pathological image to be detected.

[0081] Specifically, the features of the pathological image to be detected (the whole slide image of the pathological image to be detected) are obtained, and the features of the pathological image to be detected are input into the target graph neural network model for classification. The classification result of the features of the pathological image to be detected is output (the classification effect is tested by commonly used indicators of classification models such as Accuracy, AUC, and F1-score, and the computation time, computation amount and parameter amount of the model are tested by Time, GFLOPs and Params indicators).

[0082] In this embodiment, a weakly supervised training strategy is introduced. Instead of more refined annotation data, the pathological slide annotation data is divided into three categories based on granularity, with costs increasing sequentially: Slide-level labels, where the annotation content is the diagnostic classification of the entire WSI slide, in the form of text labels (1 label / slide), such as "Grade III gastric adenocarcinoma" or "chronic inflammation," are primarily used for weakly supervised learning; Region-level labels, where the annotation content is the boundary and type of the lesion region, in the form of polygon masks, are primarily used for instance segmentation model training; Pixel-level labels, where the annotation content is the category of a single cell / substructure, in the form of a category map for each pixel, such as labeling cancer cell nuclei (red) and lymphocytes (blue), are primarily used for segmentation model training. This yields a target graph neural network model. The features of the pathological image to be detected are input into the target graph neural network model for classification, and the classification result of the features of the pathological image to be detected is output.

[0083] Furthermore, such as Figure 4 As shown, based on the above-mentioned pathological image classification method based on graph neural feature fusion, the present invention also provides a pathological image classification system based on graph neural feature fusion, wherein the pathological image classification system based on graph neural feature fusion includes:

[0084] The feature fusion module 51 is used to acquire multi-scale digital image data, extract multiple image features from the multi-scale digital image data according to a preset magnification, and fuse the multiple image features through a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0085] The model training module 52 is used to obtain slide-level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism and the multi-resolution features, and train the graph neural network model according to the slide-level labels to obtain the target graph neural network model.

[0086] The model prediction module 53 is used to acquire the features of the pathological image to be detected, input the features of the pathological image to be detected into the target graph neural network model for classification, and output the classification result of the features of the pathological image to be detected.

[0087] Furthermore, such as Figure 5 As shown, based on the above-mentioned pathological image classification method and system based on graph neural feature fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0088] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a pathological image classification program 40 based on graph neural feature fusion, which can be executed by the processor 10 to implement the pathological image classification method based on graph neural feature fusion in this application.

[0089] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the pathological image classification method based on graph neural feature fusion.

[0090] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The terminals communicate with each other via a system bus.

[0091] In one embodiment, when the processor 10 executes the pathological image classification program 40 based on graph neural feature fusion in the memory 20, the following steps are performed:

[0092] Multi-scale digital image data is acquired, and multiple image features are extracted from the multi-scale digital image data according to a preset magnification ratio. The multiple image features are then fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0093] Obtain slide-level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, and the multi-resolution features, and train the graph neural network model according to the slide-level labels to obtain the target graph neural network model;

[0094] The features of the pathological image to be detected are obtained, and the features are input into the target graph neural network model for classification, and the classification result of the features of the pathological image to be detected is output.

[0095] The preset magnification includes low magnification, medium magnification, and high magnification.

[0096] The image features include low-scale image features, medium-scale image features, and high-scale image features;

[0097] The process of acquiring multi-scale digital image data involves extracting multiple image features from the multi-scale digital image data according to a preset magnification ratio, and fusing these multiple image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features. Specifically, this includes:

[0098] Acquire multi-scale digital image data, and extract low-scale image features from the multi-scale digital image data based on low magnification.

[0099] Based on the medium magnification, the multi-scale digital image data is extracted to obtain the medium-scale image features;

[0100] High-scale image features are obtained by extracting the multi-scale digital image data based on high magnification.

[0101] The low-scale image features, the medium-scale image features, and the high-scale image features are fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

[0102] Specifically, the process of fusing the low-scale image features, the mid-scale image features, and the high-scale image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features includes:

[0103] The attention weights of the pyramid-shaped multi-scale feature fusion model are dynamically calculated for the low magnification, the medium magnification, and the high magnification to obtain complementary information;

[0104] The low-scale image features, the medium-scale image features, and the high-scale image features are fused based on the complementary information to obtain multi-resolution features.

[0105] The step of fusing the low-scale image features, the mid-scale image features, and the high-scale image features based on the complementary information to obtain multi-resolution features specifically includes:

[0106] The low-scale image features, the medium-scale image features, and the high-scale image features are transformed to obtain multiple one-dimensional sequences. All the one-dimensional sequences are sorted to obtain a two-dimensional grid.

[0107] The two-dimensional grid is input into the pyramid-shaped multi-scale feature fusion model to determine the hierarchical relationship;

[0108] Based on the pyramid-shaped multi-scale feature fusion model, the low-scale image features, the medium-scale image features, and the high-scale image features are fused according to the complementary information and the hierarchical relationship to obtain multi-resolution features.

[0109] The step of obtaining slide-level labels, which involves constructing an initial graph neural network model based on a dynamic graph construction mechanism, a self-attention mechanism, a channel reduction mechanism, and the multi-resolution features, and training the graph neural network model according to the slide-level labels to obtain a target graph neural network model, specifically includes:

[0110] Based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and the multi-resolution features, the graph structure, node information, and one-dimensional information are determined, and an initial graph neural network model is constructed based on the graph structure, node information, and one-dimensional information.

[0111] Obtain slide level labels, and perform weakly supervised training on the parameters of the graph neural network model based on the slide level labels and the backpropagation algorithm to obtain the target graph neural network model.

[0112] The construction of the initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and multi-resolution features specifically includes:

[0113] Based on the dynamic graph construction mechanism, multiple sub-features of the multi-resolution features are connected according to the spatial positional relationship of the multi-resolution features to obtain a graph structure;

[0114] Based on the self-attention mechanism, the triplet structure of the graph structure is calculated according to a preset joint attention scoring strategy to obtain node information;

[0115] Based on the channel reduction mechanism, the node information is compressed to obtain one-dimensional information;

[0116] Based on the dynamic nonlinear function, an initial graph neural network model is constructed according to the graph structure, the node information, and the one-dimensional information.

[0117] Specifically, the compression of the node information based on the channel reduction mechanism to obtain one-dimensional information includes:

[0118] Obtain the query request and feature tags. Based on the channel reduction mechanism, compress the node information according to the query request and feature tags to obtain a one-dimensional scalar.

[0119] Based on the spatial dimension, the one-dimensional scalar is averaged and pooled to obtain one-dimensional information.

[0120] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a pathological image classification program based on graph neural feature fusion, and the pathological image classification program based on graph neural feature fusion, when executed by a processor, implements the steps of the pathological image classification method based on graph neural feature fusion as described above.

[0121] In summary, this invention provides a pathological image classification method, system, terminal, and storage medium based on graph neural network feature fusion. The method includes: acquiring multi-scale digital image data; extracting multiple image features from the multi-scale digital image data according to a preset magnification; fusing the multiple image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features; acquiring slide-level labels; constructing an initial graph neural network model based on a dynamic graph construction mechanism, a self-attention mechanism, a channel reduction mechanism, and the multi-resolution features; training the graph neural network model based on the slide-level labels to obtain a target graph neural network model; acquiring the features of the pathological image to be detected; inputting the features of the pathological image to be detected into the target graph neural network model for classification; and outputting the classification result of the features of the pathological image to be detected. This invention classifies the features of the pathological image to be detected based on the target graph neural network model, achieving efficient and accurate classification.

[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal system that includes that element.

[0123] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0124] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A pathological image classification method based on graph neural feature fusion, characterized in that, The pathological image classification method based on graph neural feature fusion includes: Multi-scale digital image data is acquired, and multiple image features are extracted from the multi-scale digital image data according to a preset magnification ratio. The multiple image features are then fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features. Obtain slide-level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, and the multi-resolution features, and train the graph neural network model according to the slide-level labels to obtain the target graph neural network model; The features of the pathological image to be detected are obtained, and the features are input into the target graph neural network model for classification, and the classification result of the features of the pathological image to be detected is output.

2. The pathological image classification method based on graph neural feature fusion according to claim 1, characterized in that, The preset magnification includes low magnification, medium magnification, and high magnification; The image features include low-scale image features, medium-scale image features, and high-scale image features; The process of acquiring multi-scale digital image data involves extracting multiple image features from the multi-scale digital image data according to a preset magnification ratio, and fusing these multiple image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features. Specifically, this includes: Acquire multi-scale digital image data, and extract low-scale image features from the multi-scale digital image data based on low magnification. Based on the medium magnification, the multi-scale digital image data is extracted to obtain the medium-scale image features; High-scale image features are obtained by extracting the multi-scale digital image data based on high magnification. The low-scale image features, the medium-scale image features, and the high-scale image features are fused using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features.

3. The pathological image classification method based on graph neural feature fusion according to claim 2, characterized in that, The process involves fusing low-scale image features, mid-scale image features, and high-scale image features using a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features, specifically including: The attention weights of the pyramid-shaped multi-scale feature fusion model are dynamically calculated for the low magnification, the medium magnification, and the high magnification to obtain complementary information; The low-scale image features, the medium-scale image features, and the high-scale image features are fused based on the complementary information to obtain multi-resolution features.

4. The pathological image classification method based on graph neural feature fusion according to claim 3, characterized in that, The step of fusing the low-scale image features, the mid-scale image features, and the high-scale image features based on the complementary information to obtain multi-resolution features specifically includes: The low-scale image features, the medium-scale image features, and the high-scale image features are transformed to obtain multiple one-dimensional sequences. All the one-dimensional sequences are sorted to obtain a two-dimensional grid. The two-dimensional grid is input into the pyramid-shaped multi-scale feature fusion model to determine the hierarchical relationship; Based on the pyramid-shaped multi-scale feature fusion model, the low-scale image features, the medium-scale image features, and the high-scale image features are fused according to the complementary information and the hierarchical relationship to obtain multi-resolution features.

5. The pathological image classification method based on graph neural feature fusion according to claim 1, characterized in that, The process of obtaining slide-level labels involves constructing an initial graph neural network model based on a dynamic graph construction mechanism, a self-attention mechanism, a channel reduction mechanism, and the multi-resolution features. The graph neural network model is then trained using the slide-level labels to obtain the target graph neural network model. Specifically, this includes: Based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and the multi-resolution features, the graph structure, node information, and one-dimensional information are determined, and an initial graph neural network model is constructed based on the graph structure, node information, and one-dimensional information. Obtain slide level labels, and perform weakly supervised training on the parameters of the graph neural network model based on the slide level labels and the backpropagation algorithm to obtain the target graph neural network model.

6. The pathological image classification method based on graph neural feature fusion according to claim 5, characterized in that, The initial graph neural network model constructed based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism, dynamic nonlinear function, and multi-resolution features specifically includes: Based on the dynamic graph construction mechanism, multiple sub-features of the multi-resolution features are connected according to the spatial positional relationship of the multi-resolution features to obtain a graph structure; Based on the self-attention mechanism, the triplet structure of the graph structure is calculated according to a preset joint attention scoring strategy to obtain node information; Based on the channel reduction mechanism, the node information is compressed to obtain one-dimensional information; Based on the dynamic nonlinear function, an initial graph neural network model is constructed according to the graph structure, the node information, and the one-dimensional information.

7. The pathological image classification method based on graph neural feature fusion according to claim 6, characterized in that, The compression of node information based on the channel reduction mechanism to obtain one-dimensional information specifically includes: Obtain the query request and feature tags. Based on the channel reduction mechanism, compress the node information according to the query request and feature tags to obtain a one-dimensional scalar. Based on the spatial dimension, the one-dimensional scalar is averaged and pooled to obtain one-dimensional information.

8. A pathological image classification system based on graph neural feature fusion, characterized in that, The pathological image classification system based on graph neural feature fusion includes: The feature fusion module is used to acquire multi-scale digital image data, extract multiple image features from the multi-scale digital image data according to a preset magnification, and fuse the multiple image features through a pyramid-shaped multi-scale feature fusion model to obtain multi-resolution features. The model training module is used to obtain slide-level labels, construct an initial graph neural network model based on the dynamic graph construction mechanism, self-attention mechanism, channel reduction mechanism and the multi-resolution features, and train the graph neural network model according to the slide-level labels to obtain the target graph neural network model. The model prediction module is used to acquire the features of the pathological image to be detected, input the features of the pathological image to be detected into the target graph neural network model for classification, and output the classification result of the features of the pathological image to be detected.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a pathological image classification program based on graph neural feature fusion stored in the memory and executable on the processor. When the pathological image classification program based on graph neural feature fusion is executed by the processor, it implements the steps of the pathological image classification method based on graph neural feature fusion as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a pathological image classification program based on graph neural feature fusion, which, when executed by a processor, implements the steps of the pathological image classification method based on graph neural feature fusion as described in any one of claims 1-7.

Citation Information

Cited By

  • A Multi-Level Attention-Based Classification Method and System for Breast Cancer Pathological Images

    CN122416447A

  • A three-dimensional target detection method, device, equipment and storage medium

    CN122435249A