A breast cancer histopathological image classification method, system and terminal

Abstract

The application discloses a breast cancer histopathological image classification method, system and terminal, and the method comprises the steps of constructing a classification model network, obtaining a full-view digital pathological section of a target tissue, inputting the full-view digital pathological section into a feature extraction module of the classification model network after cutting and splicing, extracting high-dimensional representation features, obtaining representation features, constructing a hypergraph according to the representation features, inputting the hypergraph into a feature fusion module for hypergraph coding, obtaining output features, performing hypergraph decoding on the output features, obtaining fusion features, inputting the fusion features into a hierarchical prediction module of the classification model network for node-level classification, obtaining a prediction classification result, integrating the prediction classification result by using a weighted majority voting mechanism, and obtaining a target classification result. The application combines large model fine-tuning and hierarchical hypergraph UNet structure, and effectively improves the accuracy of breast cancer histopathological image classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a breast cancer histopathological image classification method, system and terminal. BACKGROUND

[0002] With the development of digital pathology, a slide can be digitized into a high-resolution whole slide image (WSI) by a scanner, which lays a foundation for intelligent analysis of pathological images. Since a WSI image usually has hundreds of millions of pixels, conventional convolutional neural networks and other structures cannot directly process it. The mainstream method usually adopts a patch division strategy to divide the WSI into a plurality of small patches, and then performs image classification based on the small patches.

[0003] However, in the existing method, a residual network or a densely connected convolutional network is used to independently classify each patch of the WSI image, and then the prediction results are spliced or averaged to form a whole image output. This kind of method solves the problem of breast cancer histopathological image classification to some extent, but also has obvious limitations, such as ignoring the context association between patches and lacking structured modeling capability, which leads to inaccurate breast cancer histopathological image classification results.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a breast cancer histopathological image classification method, system, terminal and computer readable storage medium, which aims to solve the problem of ignoring the context association between patches and lacking structured modeling capability in the breast cancer histopathological image classification technology in the prior art, which leads to inaccurate breast cancer histopathological image classification results.

[0006] To achieve the above purpose, the present application provides a breast cancer histopathological image classification method, which comprises the following steps:

[0007] A classification model network is constructed, which comprises a feature extraction module, a feature fusion module and a hierarchical prediction module;

[0008] A whole slide digital pathology section of a target tissue is obtained, and after the whole slide digital pathology section is cropped and spliced, it is input to the feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain a representation feature;

[0009] An hypergraph is constructed according to the representation feature, the hypergraph is input to the feature fusion module for hypergraph coding to obtain an output feature, and the output feature is hypergraph decoded to obtain a fusion feature;

[0010] inputting the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and integrating the predicted classification result using a weighted majority voting mechanism to obtain a target classification result.

[0011] Optionally, the breast cancer histopathological image classification method, wherein after the full-field digital pathology slice is cropped and spliced, the high-dimensional representation feature is extracted from the feature extraction module of the classification model network to obtain a representation feature, and the method specifically comprises:

[0012] cropping the full-field digital pathology slice into a plurality of non-overlapping patches according to a fixed step size, and splicing a preset number of adjacent patches to obtain a plurality of regions;

[0013] wherein the patch represents an image block, and the region represents an image region;

[0014] using a large model as a feature extractor, freezing all backbone parameters of the feature extractor, and performing supervised optimization on a newly introduced task-specific linear head to obtain a fine-tuned pathological large model;

[0015] inputting a plurality of regions into the feature extraction module of the classification model network, and extracting features of the patches in each region through the fine-tuned pathological large model to obtain representation features of all the patches.

[0016] Optionally, the breast cancer histopathological image classification method, wherein the hypergraph is constructed according to the representation features, and the method specifically comprises:

[0017] taking the feature of each patch in a region as a node to obtain a node set, and taking the edge formed by the nodes as a hyperedge to obtain a hyperedge set;

[0018] constructing a hypergraph according to the node set and the hyperedge set, calculating the attention weight of the features of all nodes of each hyperedge, obtaining similar nodes according to the attention weight, and constructing a correlation matrix corresponding to each hyperedge using the similar nodes;

[0019] wherein the node set contains all patches in a region, and the feature of a node is the feature of a corresponding patch, and each hyperedge in the hyperedge set is composed of two or more nodes.

[0020] Optionally, the breast cancer histopathological image classification method, wherein the feature fusion module comprises a hypergraph encoding submodule and a hypergraph decoding submodule;

[0021] The hypergraph encoding submodule comprises a first hypergraph convolution unit, a hypergraph down-sampling unit and a Legendre KAN layer.

[0022] The hypergraph decoding submodule comprises a hypergraph up-sampling unit and a second hypergraph convolution unit.

[0023] Optionally, the breast cancer histopathological image classification method, wherein the hypergraph is input into the feature fusion module for hypergraph encoding to obtain output features, specifically comprising:

[0024] The hypergraph is input into the first hypergraph convolution unit, and node features of the hypergraph are aggregated into each hyperedge of the hypergraph through an association matrix of the hyperedge to obtain initial hyperedge representations.

[0025] The center hyperedge features are obtained through the association matrix, the initial hyperedge representations are updated using the center hyperedge features to obtain updated hyperedge features.

[0026] The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and the nodes are updated using the feedback information to obtain updated node features.

[0027] The updated node features are input into the hypergraph down-sampling unit of the hypergraph encoding submodule, a two-dimensional feature matrix is obtained according to the spatial position arrangement of the updated node features, the two-dimensional feature matrix is evenly divided into multiple blocks, each block is flattened and input into a linear layer for mapping to obtain compressed node features.

[0028] The node features are input into the Legendre KAN layer of the hypergraph encoding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest order using a Legendre polynomial to construct a target representation, and a set of learnable parameters are used to linearly combine the target representation to obtain output features.

[0029] Optionally, the breast cancer histopathological image classification method, wherein the output features are hypergraph decoded to obtain fusion features, specifically comprising:

[0030] The output features are input into the hypergraph up-sampling unit of the hypergraph decoding submodule, and the output features are expanded through the hypergraph up-sampling unit to obtain expanded node features.

[0031] The expanded node features are input into the second hypergraph convolution unit, and the expanded node features are decoded through the hypergraph convolution unit to obtain decoder features, the output features are connected and fused with the decoder features to obtain fusion features.

[0032] Optionally, the breast cancer histopathological image classification method, wherein the fusion feature is input into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and a weighted majority voting mechanism is used to integrate the predicted classification result to obtain a target classification result, and the method specifically comprises:

[0033] The fusion feature is input into the hierarchical prediction module of the classification model network, and each level of image of the fusion feature is classified by the hierarchical prediction module to obtain a plurality of predicted images.

[0034] According to a preset confidence, a weight is assigned to each predicted image, and according to the weight, a weighted majority voting mechanism is used to splice and restore the plurality of predicted images into a complete full-field digital pathology slice predicted image, and a target classification result is obtained according to the full-field digital pathology slice predicted image.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a breast cancer histopathological image classification system, wherein the breast cancer histopathological image classification system comprises:

[0036] A classification model construction module is configured to construct a classification model network, and the classification model network comprises a feature extraction module, a feature fusion module and a hierarchical prediction module.

[0037] A representation feature extraction module is configured to obtain a full-field digital pathology slice of a target tissue, and after the full-field digital pathology slice is cropped and spliced, the full-field digital pathology slice is input into the feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain a representation feature.

[0038] A hypergraph encoding and decoding module is configured to construct a hypergraph according to the representation feature, input the hypergraph into the feature fusion module for hypergraph encoding to obtain an output feature, and perform hypergraph decoding on the output feature to obtain a fusion feature.

[0039] A classification result output module is configured to input the fusion feature into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and a weighted majority voting mechanism is used to integrate the predicted classification result to obtain a target classification result.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a terminal, wherein the terminal comprises a memory, a processor and a breast cancer histopathological image classification program stored on the memory and executable on the processor, and the breast cancer histopathological image classification program is executed by the processor to realize the steps of the breast cancer histopathological image classification method as described above.

[0041] In addition, to achieve the above-mentioned object, the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a breast cancer histopathological image classification program, and the breast cancer histopathological image classification program realizes the steps of the breast cancer histopathological image classification method when executed by a processor.

[0042] In the present application, a classification model network is constructed, a full-view digital pathology section of a target tissue is obtained, the full-view digital pathology section is cropped and spliced, and then input to a feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain representation features; a hypergraph is constructed according to the representation features, the hypergraph is input to a feature fusion module for hypergraph coding to obtain output features, the output features are hypergraph decoded to obtain fusion features; the fusion features are input to a hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and the predicted classification result is integrated using a weighted majority voting mechanism to obtain a target classification result. The present application combines large model fine-tuning and hierarchical hypergraph structure, and effectively improves the accuracy of breast cancer histopathological image classification. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of a preferred embodiment of the breast cancer histopathological image classification method of the present application;

[0044] Figure 2 is a general architecture diagram of the classification model network in the breast cancer histopathological image classification method of the present application;

[0045] Figure 3 is a principle schematic diagram of the LegendreKAN layer in the breast cancer histopathological image classification method of the present application;

[0046] Figure 4 is a structure diagram of a preferred embodiment of the breast cancer histopathological image classification system of the present application;

[0047] Figure 5 is a structure diagram of a preferred embodiment of the terminal of the present application. DETAILED DESCRIPTION

[0048] The present application provides a breast cancer histopathological image classification method, system and terminal. In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application is further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0049] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0050] In addition, if the description of "first", "second" and the like is involved in the embodiments of the present application, the description of "first", "second" and the like is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second" can be explicitly or implicitly included at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the protection scope required by the present application.

[0051] The breast cancer histopathological image classification method described in the preferred embodiment of the present application, as shown in Figure 1 The breast cancer histopathological image classification method includes the following steps:

[0052] Step S10, a classification model network is constructed, which includes a feature extraction module, a feature fusion module and a hierarchical prediction module.

[0053] As shown in Figure 2 In this embodiment, a classification model network is constructed, which includes a feature extraction module, a feature fusion module and a hierarchical prediction module. In the feature extraction module, a mainstream pathological basic model with general representation ability is selected as a visual encoder for feature extraction. In the feature fusion module, a hierarchical dynamic hypergraph structure is used for modeling, with patch as node, an attention-driven dynamic hypergraph is constructed, and hypergraph convolution and UNet (a deep learning model for image segmentation) structure are used for multi-level context fusion modeling, and the context high-order dependency relationship between different patches is learned through downsampling. In the hierarchical prediction module, the classification stability, robustness and actual deployability of the classification model are improved through hierarchical prediction and weighted majority voting mechanism.

[0054] Further, before using the classification model network for breast cancer histopathological image classification, the classification model network is also trained.

[0055] Specifically, the BACH (from ICIAR 2018 challenge) dataset is used. The dataset aims to classify and localize clinically relevant histopathology in microscopy and whole-slide images, with a total of 30 WSI with annotations. In the feature extraction stage, each patch is divided into 512x512, and each region is composed of 8x8 patches, i.e., the size is 4096x4096. And each patch is scaled to 224x224 size to reduce the computational cost. For each region, a sliding window strategy with a step size of 1024 pixels is used for cropping. At the same time, the background area and sparse tissue are unified into a class, and an oversampling strategy is used to improve the proportion of the minority class in the training samples. The above model is trained using the PyTorch framework on a TITAN RTX 4090 GPU with 24GB of memory. During training, the initial learning rate is set to In addition, in order to optimize the performance of the model, the Adam optimizer is used to optimize the model, and the cross-entropy loss is used as the objective function when training the model.

[0056] The breast cancer histopathological image classification method proposed in the present application is fully tested on the BACH dataset commonly used for breast cancer pathological images, and significant performance improvement is achieved, verifying its effectiveness in fine-grained tissue region classification and having good practical application prospects.

[0057] Step S20, a whole field digital pathology section of the target tissue is obtained, and after the whole field digital pathology section is cropped and spliced, the high-dimensional feature extraction is performed on the feature extraction module of the classification model network to obtain the feature.

[0058] It can be understood that the present application realizes multi-classification of breast cancer histopathological images. Since the whole field digital pathology section (WSI) has super high resolution (usually more than tens of thousands of pixels), it is difficult to directly input the deep neural network for overall processing, therefore, the present application designs a two-stage hierarchical classification framework to balance the local discrimination ability and global context modeling ability.

[0059] Specifically, in the first stage, a whole field digital pathology section of the target tissue is obtained, the whole field digital pathology section is cropped into a plurality of non-overlapping patches according to a fixed step, and a preset number of adjacent patches are spliced to obtain a plurality of regions; wherein the patch represents an image block, and the region represents an image region.

[0060] In the present embodiment, the whole WSI is cropped into a plurality of non-overlapping image blocks, referred to as patches, according to a fixed step, and a preset number (e.g., 8) of adjacent patches are further spliced to obtain a plurality of regions. wherein, denotes the length of the whole WSI, denotes the width of the whole WSI) are stitched into a larger range of image regions, called regions. This process not only preserves the fine-grained features of each patch, but also provides a contextual context for subsequent modeling of the spatial dependency between patches.

[0061] Further, a large model is used as a feature extractor, all backbone parameters of the feature extractor are frozen, and a task-specific linear head newly introduced is supervisedly optimized to obtain a fine-tuned pathological large model.

[0062] The plurality of regions are input to the feature extraction module of the classification model network, and the patches in each region are feature-extracted by the fine-tuned pathological large model to obtain representation features of all the patches.

[0063] In this embodiment, a fine-tuned pathological large model is used to extract high-dimensional representation features for each patch to describe its tissue structure and cancer morphology. It can be understood that, since direct migration of a pre-trained model to a breast cancer pathological image may cause semantic drift, the large model fine-tuning strategy combining parameter freezing and linear adaptation is adopted in this embodiment to improve the model's perception ability of breast cancer image features.

[0064] It can be understood that the present application selects an existing large model as a basic feature extractor, freezes all its backbone parameters, and only supervises the optimization of a newly introduced task-specific linear head, thereby realizing the structural feature migration modeling of the breast cancer pathological image. Through limited supervised learning, the model can further adapt to the feature patterns of breast cancer tissues while maintaining the original generalization ability. This fine-tuning process does not require a large-scale medical image training set, and can realize specialized learning of pathological images, thereby better reflecting the morphological differences and tissue structure features of cancerous regions in the extracted features.

[0065] Step S30, constructing a hypergraph according to the representation features, inputting the hypergraph to the feature fusion module for hypergraph coding to obtain output features, and hypergraph decoding the output features to obtain fusion features.

[0066] It can be understood that although more representative patch-level breast cancer histopathological image features are extracted by fine-tuning the large model, the features still do not explicitly model the spatial context information interaction between patches, which may limit the overall model expression ability. In order to solve this problem, in the second stage of the above two-stage hierarchical classification framework, the present application regards each patch feature within a region as a node in a hypergraph, and constructs a dynamic hypergraph structure for modeling the high-order spatial correlation between patches. By introducing a hypergraph UNet architecture, context feature propagation and aggregation from encoding to decoding are realized, further improving the structural modeling ability of the model. In the encoding process, the node features output by each layer of hypergraph convolution will be sent to the LegendreKAN layer to enhance the non-linear ability of feature representation and improve the generalization performance of the model.

[0067] Specifically, in the second stage, the features of each patch within a region are taken as nodes to obtain a node set, and the edges formed by the nodes are taken as hyperedges to obtain a hyperedge set; a hypergraph is constructed according to the node set and the hyperedge set, and the attention weights of the features of all nodes of each hyperedge are calculated, and similar nodes are obtained according to the attention weights, and an association matrix corresponding to each hyperedge is constructed using the similar nodes; wherein the node set contains all patches in a region, and the feature of a node is the feature of the corresponding patch, and each hyperedge in the hyperedge set is composed of two or more nodes.

[0068] In this embodiment, the overlapping regions cropped from the WSI are denoted as , where the subscript represents the region, represents the total number of regions, and i represents the ordinal number of the region. In order to better construct the hypergraph, each will be divided into non-overlapping patches, denoted as , where the subscript represents the patch, represents the total number of patches, and j represents the ordinal number of the patch.

[0069] In order to dynamically capture the context relationship between image blocks, the present embodiment dynamically constructs a hypergraph based on the attention mechanism, where represents the node set, that is, contains all patches in a certain region, and the feature of each patch will be taken as the feature of the node; represents a hyperedge set, which defines the high-order connection structure between nodes, and each hyperedge is composed of at least two nodes. For each hyperedge, the attention weight of all node features is calculated, and the Top-K (i.e., selecting the top K elements from a set of data) nodes most similar to the current node and the node feature are selected to construct the dynamic adjacent hyperedge. This dynamic hypergraph structure based on attention-driven can flexibly express the non-local semantic dependence between patches.

[0070] Further, the hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit, and the hypergraph encoding submodule includes a first hypergraph convolution unit, a hypergraph downsampling unit and a LegendreKAN layer (a KAN nonlinear mapping layer based on Legendre polynomials); the hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit.

[0071] The hypergraph is input into the feature fusion module for hypergraph encoding to obtain output features, specifically including:

[0072] The hypergraph is input into the first hypergraph convolution unit, and the node features of the hypergraph are aggregated into each hyperedge of the hypergraph through the association matrix of the hyperedge to obtain initial hyperedge representation;

[0073] The center hyperedge feature is obtained through the association matrix, and the initial hyperedge representation is updated with the center hyperedge feature to obtain updated hyperedge features;

[0074] The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and the node features are updated according to the feedback information to obtain updated node features;

[0075] The updated node features are input into the hypergraph downsampling unit of the hypergraph encoding submodule, a two-dimensional feature matrix is obtained according to the spatial position arrangement of the updated node features, the two-dimensional feature matrix is evenly divided into multiple word blocks, and each word block is flattened and input into a linear layer for mapping to obtain compressed node features;

[0076] The node features are input into the LegendreKAN layer of the hypergraph encoding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest order using Legendre polynomials, a target representation is constructed, a set of learnable parameters is used to linearly combine the target representation, and output features are obtained.

[0077] ;

[0078] wherein, is the first characteristics of a patch of nodes, a non-linear mapping function from nodes to hyper-edges, a hyper-edge association matrix of a patch of nodes, an ordinal of a hyper-edge, a hyper-edge association matrix of a patch of nodes, an ordinal of a hyper-edge, a hyper-edge association matrix of a patch of nodes,

[0079] Further, to enhance the expressive power of the hyper-edge representation, a center hyper-edge feature is introduced, forming an updated hyper-edge feature:

[0080] ;

[0081] wherein, are learnable parameters.

[0082] Further, a node receives feedback information from its associated multiple hyper-edges, updates its own representation, and obtains an updated node feature:

[0083] ;

[0084] wherein, is an update function from hyper-edges to nodes, is an updated node feature of the node .

[0085] Further, after implementing hypergraph convolution to obtain an updated node feature, a node compression operation is used to gradually reduce the number of nodes and extract richer contextual features. Specifically, according to the spatial position of the node, its feature is arranged into a two-dimensional feature matrix of size , where C represents the dimension of the feature; then, the matrix is evenly divided into sub-blocks, each of which is flattened and mapped to a C-dimensional feature by a linear layer to form a compressed node feature . In this implementation, is a self-set parameter, and here the value of is set to 2, and the hypergraph structure after node compression will be reconstructed.

[0086] It can be understood that although the hypergraph convolution can effectively model the relationship between the node and its high-order structure neighborhood, its representation ability may still be limited to the linear combination form of the graph structure itself. Therefore, after the hypergraph is reconstructed, the application uses a LegendreKAN layer to improve the non-linear representation ability of the node feature.

[0087] As Figure 3As shown, specifically, first, the compressed node features are normalized, and then the Legendre polynomial expansion is used to expand the features of each channel to the highest order (In this embodiment, The value of is set to 4), the target representation is constructed as follows :

[0088] ;

[0089] wherein, represents the polynomial expansion order.

[0090] Then, a set of learnable parameters is used to linearly combine each node in the channel dimension to obtain the non-linear mapping output feature :

[0091] ;

[0092] wherein, is an activation function.

[0093] Further, the output feature is hypergraph decoded to obtain a fused feature, specifically including:

[0094] The output feature is input to the hypergraph upsampling unit of the hypergraph decoding submodule, and the hypergraph upsampling unit is used to expand the nodes of the output feature to obtain expanded node features;

[0095] The expanded node features are input to the second hypergraph convolution unit, and the hypergraph convolution unit is used to decode the expanded node features to obtain decoder features. The output feature and the decoder feature are connected and fused to obtain a fused feature.

[0096] In this embodiment, the hypergraph decoding submodule mainly consists of three expansion units, and the expansion unit includes a hypergraph upsampling unit and a hypergraph convolution unit, the purpose of which is to gradually increase the number of nodes and restore the features of the original resolution to achieve more accurate classification. Each expansion module consists of two parts: first, a node expansion operation is performed, and then a graph convolution layer is connected. The node expansion operation is similar to the node compression in the encoder, and a 2x2 upsampling convolution operation is used in the decoder to improve the spatial resolution of the feature map. Subsequently, in order to alleviate the over-smoothing problem in the graph neural network, a skip connection mechanism is adopted to connect and fuse the features from the corresponding layer of the encoder with the decoder features after upsampling.

[0097] Step S40, input the fusion feature into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and integrate the predicted classification result using a weighted majority voting mechanism to obtain a target classification result.

[0098] Specifically, the fusion feature is input into the hierarchical prediction module of the classification model network, and each level image of the fusion feature is subjected to node-level classification by the hierarchical prediction module to obtain a plurality of predicted images.

[0099] In this embodiment, in order to obtain better classification performance, a hierarchical supervision strategy is adopted, that is, each level image on the decoder is subjected to node-level classification to obtain four predicted images, respectively denoted as , wherein Correspondingly, the true label of the s-th layer is denoted as , wherein respectively represent the height, length and category number of the true image, and the true label is generated by region division and aggregation of the original pixel-level label.

[0100] The hierarchical classification learning from coarse to fine can be realized by calculating the cross-entropy loss at multiple scales, and updating the parameters of the hierarchical prediction module according to the cross-entropy loss, and the loss function is as follows:

[0101] ;

[0102] , wherein represents the cross-entropy loss calculated according to the predicted image and the true label denoted as .

[0103] Further, according to a preset confidence, each of the predicted images is assigned a weight, and according to the weight, the plurality of predicted images are spliced and restored into a complete full-view digital pathology slice predicted image based on a weighted majority voting mechanism, and a target classification result is obtained according to the full-view digital pathology slice predicted image.

[0104] It can be understood that, due to the overlap between the regions obtained by cropping, the proposed framework will generate multiple different predicted results for the same patch in the inference stage. In order to integrate these possibly inconsistent patch predictions, the weighted majority voting mechanism is used in this embodiment to aggregate these different predicted results to form the final target classification result.

[0105] Suppose that an image block has predicted results, and according to the prediction confidence, the i-th predicted result is assigned a weight ​The final full-field digital pathology slide prediction map is given by:

[0106] ;

[0107] ;

[0108] wherein, denotes the probability of the i-th prediction belonging to the j-th class, is an indicator function, and the weight is calculated according to the normalized entropy of the prediction probability distribution, wherein, denotes the entropy of the i-th prediction result, denotes a norm function. The formula shows that the smaller the entropy value, the more certain the prediction, and the greater the weight, that is, the greater the confidence. Therefore, the prediction with high confidence will have a greater impact on the final classification result in the majority voting. Finally, all image block-level predictions are spliced to restore the complete WSI-level prediction map. Since it is assumed that all pixels within each image block belong to the same class, the resolution of the prediction map is consistent with the original WSI.

[0109] It can be seen that the present application uses fine-tuning of a pre-trained large model to fully exploit breast cancer features and improve the model's perception of different tissue forms. A multi-level dynamically constructed hypergraph UNet network is designed to explicitly model the context structure dependency relationship between patches in the WSI. Finally, a Legendre KAN nonlinear mapping layer (Legendre KAN) based on Legendre polynomials is introduced to enhance high-order feature expression capability and improve model generalization performance. During training, multi-scale supervision and soft label mechanism are used, and multiple predictions in the reasoning stage are integrated through weighted majority voting to improve model stability and consistency, realizing multi-classification of breast cancer histopathological images, having strong patch-level discrimination ability and global structure modeling ability, and being particularly suitable for breast cancer lesion area recognition and fine classification under complex tissue morphology.

[0110] Further, as shown in Figure 4 based on the above breast cancer histopathological image classification method, the present application also correspondingly provides a breast cancer histopathological image classification system, wherein the breast cancer histopathological image classification system comprises:

[0111] a classification model construction module 51 for constructing a classification model network, wherein the classification model network comprises a feature extraction module, a feature fusion module and a hierarchical prediction module;

[0112] ​​​​The representation feature extraction module 52 is configured to obtain a full-view digital pathology slice of the target tissue, and input the full-view digital pathology slice to the feature extraction module of the classification model network after cutting and splicing, to obtain a high-dimensional representation feature.

[0113] The hypergraph coding and decoding module 53 is configured to construct a hypergraph according to the representation feature, input the hypergraph to the feature fusion module to obtain an output feature, and decode the output feature to obtain a fusion feature.

[0114] The classification result output module 54 is configured to input the fusion feature to the hierarchical prediction module of the classification model network to obtain a predicted classification result, and integrate the predicted classification result using a weighted majority voting mechanism to obtain a target classification result.

[0115] Further, as shown in Figure 5 Based on the breast cancer tissue pathology image classification method and system, the application further provides a terminal, which comprises a processor 10, a memory 20 and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that all the shown components are not required, and more or fewer components can be alternatively implemented.

[0116] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 20 can include both an internal storage unit and an external storage device of the terminal. The memory 20 is configured to store application software and various data installed on the terminal, such as program codes of the terminal, etc. The memory 20 can also be configured to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a breast cancer tissue pathology image classification program 40, which can be executed by the processor 10 to implement the breast cancer tissue pathology image classification method.

[0117] The processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, configured to run program codes or process data stored in the memory 20, such as to execute the breast cancer tissue pathology image classification method, etc.

[0118] The display 30 can be, in some embodiments, an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) toucher, etc. The display 30 is used to display information at the terminal and to display a visualized user interface. The components of the terminal communicate with each other through a system bus.

[0119] In an embodiment, the following steps are implemented when the processor 10 executes the breast cancer histopathological image classification program 40 in the memory 20:

[0120] A classification model network is constructed, which includes a feature extraction module, a feature fusion module, and a hierarchical prediction module;

[0121] A full-field digital pathology section of a target tissue is obtained, and after the full-field digital pathology section is cropped and spliced, it is input to the feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain representation features;

[0122] An hypergraph is constructed according to the representation features, the hypergraph is input to the feature fusion module for hypergraph coding to obtain output features, the output features are hypergraph decoded to obtain fusion features;

[0123] The fusion features are input to the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and a weighted majority voting mechanism is used to integrate the predicted classification result to obtain a target classification result.

[0124] The full-field digital pathology section is cropped and spliced, and then input to the feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain representation features, specifically including:

[0125] The full-field digital pathology section is cropped into a plurality of non-overlapping patches according to a fixed step size, and a preset number of adjacent patches are spliced to obtain a plurality of regions;

[0126] The patch represents an image block, and the region represents an image area.

[0127] A large model is used as a feature extractor, all backbone parameters of the feature extractor are frozen, a task-specific linear head newly introduced is supervised optimized to obtain a fine-tuned pathological large model;

[0128] The plurality of regions are input to the feature extraction module of the classification model network, and the patches in each region are subjected to feature extraction by the fine-tuned pathological large model to obtain representation features of all the patches.

[0129] The hypergraph is constructed according to the representation features, and specifically includes:

[0130] The features of each patch in a region are taken as nodes to obtain a node set, and edges formed by the nodes are taken as hyperedges to obtain a hyperedge set;

[0131] The hypergraph is constructed according to the node set and the hyperedge set, and the attention weights of the features of all nodes of each hyperedge are calculated, and similar nodes are obtained according to the attention weights, and an association matrix corresponding to each hyperedge is constructed using the similar nodes;

[0132] The node set contains all patches in a region, and the features of the nodes are the features of the corresponding patches, and each hyperedge in the hyperedge set is composed of two or more nodes.

[0133] The feature fusion module includes a hypergraph encoding submodule and a hypergraph decoding submodule;

[0134] The hypergraph encoding submodule includes a first hypergraph convolution unit, a hypergraph down-sampling unit and a LegendreKAN layer;

[0135] The hypergraph decoding submodule includes a hypergraph up-sampling unit and a second hypergraph convolution unit.

[0136] The hypergraph is input to the feature fusion module for hypergraph encoding to obtain output features, and specifically includes:

[0137] The hypergraph is input to the first hypergraph convolution unit, and the node features of the hypergraph are aggregated to each hyperedge of the hypergraph through the association matrix of the hyperedge to obtain an initial hyperedge representation;

[0138] The central hyperedge feature is obtained through the association matrix, the initial hyperedge representation is updated using the central hyperedge feature to obtain an updated hyperedge feature;

[0139] The nodes of the hypergraph respectively receive feedback information from the updated hyperedge feature associated with themselves, and the node features are updated according to the feedback information to obtain updated node features;

[0140] inputting the updated node features into a hypergraph down-sampling unit of the hypergraph encoding submodule, obtaining a two-dimensional feature matrix according to spatial position arrangement of the updated node features, dividing the two-dimensional feature matrix into a plurality of word blocks, and inputting each word block into a linear layer after flattening to obtain compressed node features;

[0141] inputting the node features into a Legendre KAN layer of the hypergraph encoding submodule, normalizing the compressed node features, expanding the normalized results on each channel to the highest order using a Legendre polynomial to construct a target representation, and linearly combining the target representation using a set of learnable parameters to obtain output features.

[0142] wherein the hypergraph decoding of the output features obtains fused features, specifically comprising:

[0143] inputting the output features into a hypergraph up-sampling unit of the hypergraph decoding submodule, performing node expansion on the output features through the hypergraph up-sampling unit to obtain expanded node features;

[0144] inputting the expanded node features into the second hypergraph convolution unit, decoding the expanded node features through the hypergraph convolution unit to obtain decoder features, and connecting and fusing the output features and the decoder features to obtain fused features.

[0145] wherein the inputting of the fused features into the hierarchical prediction module of the classification model network performs node-level classification to obtain a predicted classification result, and the predicted classification result is integrated using a weighted majority voting mechanism to obtain a target classification result, specifically comprising:

[0146] inputting the fused features into the hierarchical prediction module of the classification model network, performing node-level classification on each level of image of the fused features through the hierarchical prediction module to obtain a plurality of predicted maps;

[0147] assigning weights to each predicted map according to a preset confidence, splicing and restoring a plurality of predicted maps into a complete full-field digital pathology slide predicted map based on a weighted majority voting mechanism according to the weights, and obtaining a target classification result according to the full-field digital pathology slide predicted map.

[0148] The application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a breast cancer histopathological image classification program, and the breast cancer histopathological image classification program, when executed by a processor, implements the steps of the breast cancer histopathological image classification method.

[0149] In summary, the present application provides a breast cancer histopathological image classification method, system and terminal, the method comprising: constructing a classification model network, obtaining a full field digital pathology section of a target tissue, after the full field digital pathology section is cropped and spliced, inputting the full field digital pathology section to a feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain a representation feature; constructing a hypergraph according to the representation feature, inputting the hypergraph to a feature fusion module for hypergraph coding to obtain an output feature, and performing hypergraph decoding on the output feature to obtain a fusion feature; inputting the fusion feature to a hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and integrating the predicted classification result using a weighted majority voting mechanism to obtain a target classification result. The present application combines large model fine-tuning and hierarchical hypergraph UNet structure, effectively improving the accuracy of breast cancer histopathological image classification.

[0150] It should be noted that in this document, the terms "comprising", "containing", or any other similar term are intended to encompass non-exclusive inclusion, such that a process, method, article or terminal including a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or terminal. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or terminal including the element.

[0151] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, it can include the processes of the above-mentioned embodiments. Wherein, any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0152] It is to be understood that the application is not limited to the examples described above, which can be modified or adapted in several ways by those skilled in the art without departing from the scope of the present application, as defined by the appended claims.

Claims

1. A breast cancer histopathological image classification method, characterized in that, The breast cancer tissue pathological image classification method comprises: A classification model network is constructed, which comprises a feature extraction module, a feature fusion module and a hierarchical prediction module; A full-view digital pathological section of a target tissue is obtained, and after the full-view digital pathological section is cropped and spliced, the full-view digital pathological section is input into the feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain representation features; An hypergraph is constructed according to the representation features, the hypergraph is input into the feature fusion module for hypergraph coding to obtain output features, and the output features are hypergraph decoded to obtain fusion features; The fusion features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain a prediction classification result, and the prediction classification result is integrated using a weighted majority voting mechanism to obtain a target classification result; The hypergraph is input into the first hypergraph convolution unit, and the node features of the hypergraph are aggregated into each hyperedge of the hypergraph through the association matrix of the hyperedge to obtain initial hyperedge representations; The center hyperedge features are obtained through the association matrix, the center hyperedge features are used for hyperedge feature updating of the initial hyperedge representations to obtain updated hyperedge features; The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and the node features are updated according to the feedback information to obtain updated node features; The updated node features are input into the hypergraph downsampling unit of the hypergraph coding submodule, a two-dimensional feature matrix is obtained according to the spatial position arrangement of the updated node features, the two-dimensional feature matrix is evenly divided into a plurality of word blocks, each word block is flattened and input into a linear layer for mapping to obtain compressed node features; The node features are input into the LegendreKAN layer of the hypergraph coding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest order using the Legendre polynomial, a target representation is constructed, a set of learnable parameters are used for linear combination of the target representation to obtain output features; The output features are input into the hypergraph upsampling unit of the hypergraph decoding submodule, the output features are node-expanded through the hypergraph upsampling unit to obtain expanded node features; The expanded node features are input into the second hypergraph convolution unit, the expanded node features are decoded through the second hypergraph convolution unit to obtain decoder features, and the output features and the decoder features are connected and fused to obtain fusion features; The fusion features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain a prediction classification result, and the prediction classification result is integrated using a weighted majority voting mechanism to obtain a target classification result, specifically comprising: ​ ​ inputting the fusion features into the hierarchical prediction module of the classification model network, performing node-level classification on each level image of the fusion features through the hierarchical prediction module, and obtaining a plurality of prediction maps; assigning weights to each of the prediction maps according to a preset confidence, and based on the weights, splicing and restoring the plurality of prediction maps into a complete full-field digital pathology slice prediction map based on a weighted majority voting mechanism, and obtaining a target classification result according to the full-field digital pathology slice prediction map.

2. The breast cancer histopathological image classification method of claim 1, wherein, After the full-field digital pathology slice is cropped and spliced, the cropped and spliced full-field digital pathology slice is input into the feature extraction module of the classification model network for high-dimensional representation feature extraction, and representation features are obtained. Specifically, the method comprises: cropping the full-field digital pathology slice into a plurality of non-overlapping patches according to a fixed step size, and splicing a preset number of adjacent patches to obtain a plurality of regions; wherein the patch represents an image block, and the region represents an image region; using a large model as a feature extractor, freezing all backbone parameters of the feature extractor, and performing supervised optimization on a newly introduced task-specific linear head to obtain a fine-tuned pathological large model; inputting the plurality of regions into the feature extraction module of the classification model network, and performing feature extraction on the patches in each region through the fine-tuned pathological large model to obtain representation features of all the patches.

3. The breast cancer histopathological image classification method of claim 2, wherein, The method of constructing a hypergraph according to the representation features comprises: taking the features of each patch in a region as nodes to obtain a node set, and taking the edges formed by the nodes as hyperedges to obtain a hyperedge set; constructing a hypergraph according to the node set and the hyperedge set, calculating the attention weights of the features of all nodes of each hyperedge, obtaining similar nodes according to the attention weights, and constructing a correlation matrix corresponding to each hyperedge using the similar nodes; wherein the node set contains all patches in a region, and the features of the nodes are the features of the corresponding patches, and each hyperedge in the hyperedge set is composed of two or more nodes.

4. The breast cancer histopathological image classification method of claim 3, wherein, The feature fusion module comprises a hypergraph encoding submodule and a hypergraph decoding submodule; The hypergraph encoding submodule comprises a first hypergraph convolution unit, a hypergraph downsampling unit, and a LegendreKAN layer; The hypergraph decoding submodule comprises a hypergraph upsampling unit and a second hypergraph convolution unit.

5. A breast cancer histopathological image classification system, characterized by, The breast cancer histopathological image classification system is applied to the breast cancer histopathological image classification method of any one of claims 1-4, and the breast cancer histopathological image classification system comprises: a classification model construction module for constructing a classification model network, wherein the classification model network comprises a feature extraction module, a feature fusion module, and a hierarchical prediction module; a representation feature extraction module for obtaining a full-field digital pathology slice of a target tissue, inputting the cropped and spliced full-field digital pathology slice into the feature extraction module of the classification model network for high-dimensional representation feature extraction, and obtaining representation features; The supergraph encoding and coding module is configured to construct a supergraph according to the characteristic features, input the supergraph into the feature fusion module for supergraph encoding, obtain an output feature, and perform supergraph decoding on the output feature to obtain a fusion feature. The classification result output module is configured to input the fusion feature into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, integrate the predicted classification result using a weighted majority voting mechanism, and obtain a target classification result.

6. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a breast cancer histopathological image classification program stored on the memory and executable on the processor, and the breast cancer histopathological image classification program, when executed by the processor, implements the steps of the breast cancer histopathological image classification method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a breast cancer histopathological image classification program, and the breast cancer histopathological image classification program, when executed by a processor, implements the steps of the breast cancer histopathological image classification method according to any one of claims 1-4.

Images