Breast cancer histopathological image classification method, system and terminal

Abstract

The invention discloses a breast cancer tissue pathological image classification method, system and terminal, and the method comprises the steps: constructing a classification model network, obtaining a full-view digital pathological section of a target tissue, carrying out the cutting and splicing of the full-view digital pathological section, inputting the cut and spliced full-view digital pathological section into a feature extraction module of the classification model network, and carrying out the high-dimensional representation feature extraction, characterization characteristics are obtained; constructing a hypergraph according to the characterization features, inputting the hypergraph into a feature fusion module for hypergraph coding to obtain output features, and performing hypergraph decoding on the output features to obtain fusion features; and inputting the fusion features into a hierarchical prediction module of the classification model network for node-level classification to obtain prediction classification results, and integrating the prediction classification results by using a weighted majority voting mechanism to obtain a target classification result. According to the method, large model fine tuning and a hierarchical hypergraph UNet structure are combined, so that the accuracy of breast cancer histopathological image classification is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method, system and terminal for classifying breast cancer tissue pathology images. Background Art

[0002] With the development of digital pathology, slides can be digitized into high-resolution, full-field digital pathology slides (WSIs) using scanners, laying the foundation for intelligent analysis of pathology images. Because WSI images typically contain hundreds of millions of pixels, conventional convolutional neural networks and other architectures cannot directly process them. Mainstream methods typically use a patch partitioning strategy, dividing the WSI into several small tiles and using these as the basis for image classification.

[0003] However, existing methods utilize architectures such as residual networks and densely connected convolutional networks to independently classify each patch in a WSI image, then concatenate or average the predictions to form the overall image output. While these methods address the problem of breast cancer histopathology image classification to some extent, they also have significant limitations, such as neglecting contextual relationships between patches and lacking structured modeling capabilities. These limitations lead to inaccurate breast cancer histopathology image classification results.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a breast cancer tissue pathology image classification method, system, terminal and computer-readable storage medium, aiming to solve the problem that the existing breast cancer tissue pathology image classification technology ignores the contextual association between patches and lacks structured modeling capabilities, resulting in inaccurate breast cancer tissue pathology image classification results.

[0006] To achieve the above object, the present invention provides a method for classifying breast cancer histopathology images, which comprises the following steps: Constructing a classification model network, the classification model network including: a feature extraction module, a feature fusion module and a hierarchical prediction module; Obtaining full-field digital pathology sections of target tissue, cropping and splicing the full-field digital pathology sections, and inputting the sections into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features; Constructing a hypergraph according to the representation features, inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and performing hypergraph decoding on the output features to obtain fused features; The fusion features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and the predicted classification results are integrated using a weighted majority voting mechanism to obtain a target classification result.

[0007] Optionally, the breast cancer tissue pathology image classification method, wherein the cropping and splicing of the full-field digital pathology sections are input into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features, specifically includes: The full-field digital pathology slice is cut into multiple non-overlapping patches according to a fixed step size, and a preset number of adjacent patches are spliced ​​to obtain multiple regions; Wherein, the patch represents an image block, and the region represents an image region; Using the large model as a feature extractor, freezing all backbone parameters of the feature extractor, and performing supervised optimization on the newly introduced task-specific linear head to obtain a fine-tuned large pathology model; The multiple regions are input into the feature extraction module of the classification model network, and the patches in each region are subjected to feature extraction through the fine-tuned pathology model to obtain the characterization features of all the patches.

[0008] Optionally, in the breast cancer tissue pathology image classification method, the step of constructing a hypergraph based on the characterization features specifically includes: The features of each patch in the region are taken as nodes to obtain a node set, and the edges formed by the nodes are taken as hyperedges to obtain a hyperedge set; Constructing a hypergraph based on the node set and the hyperedge set, and calculating attention weights of features of all nodes of each hyperedge, obtaining similar nodes based on the attention weights, and constructing an association matrix corresponding to each hyperedge using the similar nodes; The node set includes all patches in a region, and the characteristics of the nodes are the characteristics of the corresponding patches. Each hyperedge in the hyperedge set is composed of two or more nodes.

[0009] Optionally, in the breast cancer tissue pathology image classification method, the feature fusion module includes a hypergraph encoding submodule and a hypergraph decoding submodule; The hypergraph encoding submodule includes a first hypergraph convolution unit, a hypergraph downsampling unit and a LegendreKAN layer; The hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit.

[0010] Optionally, the breast cancer tissue pathology image classification method, wherein the step of inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, specifically includes: Inputting the hypergraph into the first hypergraph convolution unit, aggregating node features of the hypergraph to each hyperedge of the hypergraph through the hyperedge association matrix, and obtaining an initial hyperedge representation; Obtaining a central hyperedge feature through the correlation matrix, and performing a hyperedge feature update on the initial hyperedge representation using the central hyperedge feature to obtain an updated hyperedge feature; The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and update the node features according to the feedback information to obtain updated node features; Inputting the updated node features into the hypergraph downsampling unit of the hypergraph encoding submodule, arranging the updated node features according to their spatial positions to obtain a two-dimensional feature matrix, dividing the two-dimensional feature matrix into a plurality of blocks, flattening each block and inputting it into a linear layer for mapping to obtain compressed node features; The node features are input into the LegendreKAN layer of the hypergraph encoding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest degree using Legendre polynomials to construct a target representation, and the target representation is linearly combined using a set of learnable parameters to obtain output features.

[0011] Optionally, the breast cancer tissue pathology image classification method, wherein the step of performing hypergraph decoding on the output features to obtain fusion features, specifically includes: Inputting the output features into the hypergraph upsampling unit of the hypergraph decoding submodule, performing node expansion on the output features by the hypergraph upsampling unit to obtain expanded node features; The expanded node features are input into the second hypergraph convolution unit, the expanded node features are decoded by the hypergraph convolution unit to obtain decoder features, and the output features are connected and fused with the decoder features to obtain fused features.

[0012] Optionally, the breast cancer tissue pathology image classification method, wherein the fusion features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and the predicted classification results are integrated using a weighted majority voting mechanism to obtain a target classification result, specifically includes: Inputting the fused features into the hierarchical prediction module of the classification model network, performing node-level classification on each level image of the fused features through the hierarchical prediction module to obtain multiple prediction graphs; A weight is assigned to each prediction image according to a preset confidence level. Based on the weight and a weighted majority voting mechanism, multiple prediction images are spliced ​​and restored into a complete full-field digital pathology slice prediction image, and a target classification result is obtained based on the full-field digital pathology slice prediction image.

[0013] In addition, to achieve the above-mentioned purpose, the present invention further provides a breast cancer tissue pathology image classification system, wherein the breast cancer tissue pathology image classification system comprises: A classification model building module is used to build a classification model network, which includes a feature extraction module, a feature fusion module and a hierarchical prediction module; a characterization feature extraction module, configured to obtain full-field digital pathology sections of target tissue, crop and splice the full-field digital pathology sections, and then input the sections into the feature extraction module of the classification model network to extract high-dimensional characterization features to obtain characterization features; A hypergraph encoding and decoding module, configured to construct a hypergraph based on the representation features, input the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and perform hypergraph decoding on the output features to obtain fused features; The classification result output module is used to input the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and integrate the predicted classification results using a weighted majority voting mechanism to obtain a target classification result.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a breast cancer tissue pathology image classification program stored in the memory and executable on the processor, wherein the breast cancer tissue pathology image classification program, when executed by the processor, implements the steps of the breast cancer tissue pathology image classification method described above.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a breast cancer tissue pathology image classification program, and when the breast cancer tissue pathology image classification program is executed by a processor, the steps of the breast cancer tissue pathology image classification method described above are implemented.

[0016] In the present invention, a classification model network is constructed to obtain full-field digital pathology slides of the target tissue. These full-field digital pathology slides are cropped and spliced, and then input into the feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain representation features. A hypergraph is constructed based on the representation features, and the hypergraph is input into the feature fusion module for hypergraph encoding to obtain output features. The output features are then hypergraph decoded to obtain fused features. The fused features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain predicted classification results. The predicted classification results are then integrated using a weighted majority voting mechanism to obtain the target classification result. The present invention combines large-scale model fine-tuning with a hierarchical hypergraph structure to effectively improve the accuracy of breast cancer tissue pathology image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flowchart of a preferred embodiment of the method for classifying breast cancer tissue pathology images of the present invention; Figure 2 This is the overall architecture diagram of the classification model network in the breast cancer tissue pathology image classification method of the present invention; Figure 3 Schematic diagram of the principle of the LegendreKAN layer in the breast cancer tissue pathology image classification method of the present invention; Figure 4 1 is a structural diagram of a preferred embodiment of the breast cancer tissue pathology image classification system of the present invention; Figure 5 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0018] This application provides a method, system, and terminal for classifying breast cancer histopathology images. To clarify the purpose, technical solutions, and effects of this application, the application is further described below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate this application and are not intended to limit it.

[0019] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0020] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features specified as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0021] The breast cancer tissue pathology image classification method described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the breast cancer tissue pathology image classification method includes the following steps: Step S10: constructing a classification model network, wherein the classification model network includes a feature extraction module, a feature fusion module and a hierarchical prediction module.

[0022] like Figure 2 As shown, in this embodiment, a classification model network is constructed, which includes a feature extraction module, a feature fusion module, and a hierarchical prediction module. In the feature extraction module, a mainstream pathology basic model with general representation capabilities is selected as a visual encoder for feature extraction. In the feature fusion module, a hierarchical dynamic hypergraph structure is used for modeling, and an attention-driven dynamic hypergraph is constructed with patches as nodes. Hypergraph convolution and UNet (a deep learning model for image segmentation) structure are used for multi-level context fusion modeling, and the contextual high-order dependencies between different patches are learned through downsampling. In the hierarchical prediction module, the classification stability, robustness, and practical deployability of the classification model are improved through hierarchical prediction and weighted majority voting mechanism.

[0023] Furthermore, before using the classification model network to classify breast cancer tissue pathology images, the method further includes: training the classification model network.

[0024] Specifically, the BACH (from the ICIAR 2018 Challenge) dataset was used. This dataset aims to classify and localize clinically relevant tissue pathology in microscopic examinations and whole slide images, with a total of 30 annotated WSIs. In the feature extraction stage, each patch is divided into 512×512, and each region consists of 8×8 patches, that is, the size is 4096×4096. Each patch is scaled to 224×224 to reduce the computational cost. For each region, a sliding window strategy with a step size of 1024 pixels is used for cropping. At the same time, the background area and sparse tissue are unified into one category, and an oversampling strategy is adopted to increase the proportion of the less-proportioned category in the training samples. The above model was trained using the PyTorch framework on a TITANRTX 4090 GPU with 24GB of memory. During the training process, the initial learning rate was set to In addition, in order to optimize the performance of the model, the Adam optimizer is used to optimize the model, and the cross entropy loss is used as the objective function when training the model. The breast cancer tissue pathology image classification method proposed in this application was fully experimented on the BACH dataset commonly used for breast cancer pathology images, achieving significant performance improvement, verifying its effectiveness in refined tissue region classification and having good practical application prospects.

[0025] Step S20: Acquire full-field digital pathology sections of the target tissue, crop and splice the full-field digital pathology sections, and input them into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features.

[0026] As can be understood, this application implements multi-classification of breast cancer histopathology images. Since full-field digital pathology slides (WSIs) have ultra-high resolution (typically exceeding tens of thousands of pixels), they are difficult to directly input into deep neural networks for overall processing. Therefore, this application designs a two-stage hierarchical classification framework to balance local discrimination capabilities with global context modeling capabilities.

[0027] Specifically, in the first stage, a full-field digital pathology slice of the target tissue is acquired, the full-field digital pathology slice is cropped into multiple non-overlapping patches according to a fixed step size, and a preset number of adjacent patches are spliced ​​to obtain multiple regions; wherein the patch represents an image block and the region represents an image area.

[0028] In this embodiment, the entire WSI is cropped into a number of non-overlapping image blocks with a fixed step size, called patches, and a preset number ( ,in, Indicates the length of the entire WSI, Representing the width of the entire WSI), adjacent patches are concatenated into a larger image region, called a region. This process not only preserves the fine-grained features of each patch but also provides context for subsequent modeling of spatial dependencies between patches.

[0029] Furthermore, the large model is used as a feature extractor, all backbone parameters of the feature extractor are frozen, and the newly introduced task-specific linear head is subjected to supervised optimization to obtain a fine-tuned pathology large model.

[0030] The multiple regions are input into the feature extraction module of the classification model network, and the patches in each region are subjected to feature extraction through the fine-tuned pathology model to obtain the characterization features of all the patches.

[0031] In this example, a fine-tuned large-scale pathology model is used to extract high-dimensional representational features for each patch, describing its tissue structure and cancerous morphology. As will be appreciated, due to the potential for semantic drift when directly transferring a pre-trained model to breast cancer pathology images, this example employs a large-scale model fine-tuning strategy combining parameter freezing and linear adaptation to improve the model's ability to perceive breast cancer image features.

[0032] It is understandable that this application uses an existing large model as the base feature extractor, freezes all its backbone parameters, and performs supervised optimization only on the newly introduced task-specific linear head, thereby achieving structural feature transfer modeling of breast cancer pathology images. Through limited supervised learning, the model maintains its original generalization capabilities while further adaptively adjusting the characteristic patterns of breast cancer tissue. This fine-tuning process achieves specialized learning of pathology images without relying on large-scale medical image training sets, thereby better reflecting the morphological differences and tissue structural characteristics of the cancerous area in the extracted features.

[0033] Step S30: construct a hypergraph based on the representation features, input the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and perform hypergraph decoding on the output features to obtain fused features.

[0034] It is understandable that although more representative patch-level breast cancer tissue pathology image features are extracted by fine-tuning the large model, these features still do not explicitly model the spatial contextual information interaction between patches, which may limit the expressive power of the overall model. To address this problem, in the second stage of the above-mentioned two-stage hierarchical classification framework, this application regards each patch feature within a region as a node in a hypergraph, and constructs a dynamic hypergraph structure for modeling high-order spatial associations between patches. By introducing the hypergraph UNet architecture, the propagation and aggregation of contextual features from encoding to decoding are realized, further improving the structural modeling capabilities of the model. During the encoding process, the node features output by each layer of hypergraph convolution will be fed into the LegendreKAN layer to enhance the nonlinear capability of feature representation and improve the generalization performance of the model.

[0035] Specifically, in the second stage, the features of each patch in the region are taken as nodes to obtain a node set, and the edges formed by the nodes are taken as hyperedges to obtain a hyperedge set; a hypergraph is constructed based on the node set and the hyperedge set, and the attention weights of the features of all nodes of each hyperedge are calculated, similar nodes are obtained based on the attention weights, and the association matrix corresponding to each hyperedge is constructed using the similar nodes; wherein, the node set contains all patches in a region, and the features of the nodes are the features of the corresponding patches, and each hyperedge in the hyperedge set is composed of two or more nodes.

[0036] In this embodiment, the data cut out from WSI The overlapping regions are denoted as , where the subscript Indicates region, Indicates the total number of regions, i indicates the ordinal number of the region. In order to better construct the hypergraph, each will be divided into non-overlapping patches, denoted as , where the subscript Indicates patch, represents the total number of patches, and j represents the ordinal number of the patch.

[0037] In order to dynamically capture the contextual relationship between image blocks, this embodiment dynamically constructs a hypergraph based on the attention mechanism. ,in, Represents a node set, which means all patches in a region, and the characteristics of each patch will be used as the feature of this node; Represents a set of hyperedges, which defines the high-order connectivity structure between nodes. Each hyperedge consists of at least two nodes. For each hyperedge, attention weights are calculated for all node features. The top-K (the top K elements in a set of data) nodes with the most similar features to the current node are selected to construct their dynamic adjacency hyperedges. This attention-driven dynamic hypergraph structure can flexibly express non-local semantic dependencies between patches.

[0038] Furthermore, the hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit, the hypergraph encoding submodule includes a first hypergraph convolution unit, a hypergraph downsampling unit and a LegendreKAN layer (a KAN nonlinear mapping layer based on Legendre polynomials); the hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit.

[0039] Inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features specifically includes: Inputting the hypergraph into the first hypergraph convolution unit, aggregating node features of the hypergraph to each hyperedge of the hypergraph through the hyperedge association matrix, and obtaining an initial hyperedge representation; Obtaining a central hyperedge feature through the correlation matrix, and performing a hyperedge feature update on the initial hyperedge representation using the central hyperedge feature to obtain an updated hyperedge feature; The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and update the node features according to the feedback information to obtain updated node features; Inputting the updated node features into the hypergraph downsampling unit of the hypergraph encoding submodule, arranging the updated node features according to their spatial positions to obtain a two-dimensional feature matrix, dividing the two-dimensional feature matrix into a plurality of blocks, flattening each block and inputting it into a linear layer for mapping to obtain compressed node features; The node features are input into the LegendreKAN layer of the hypergraph encoding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest degree using Legendre polynomials to construct a target representation, and the target representation is linearly combined using a set of learnable parameters to obtain output features.

[0040] ; in, For the The characteristics of each node (patch), represents the nonlinear mapping function from nodes to hyperedges, Indicates the The hyperedge incidence matrix of the nodes (patch), is the ordinal number of the hyperedge, Indicates the Super edge.

[0041] Furthermore, in order to enhance the expressive power of hyperedge representation, the central hyperedge feature Introduce and form the updated hyperedge feature: ; in, are learnable parameters.

[0042] Furthermore, the node receives feedback information from its associated multiple hyperedges, updates its own representation, and obtains the updated node features: ; in, is the update function from hyperedge to node, For nodes Updated node features.

[0043] Furthermore, after implementing hypergraph convolution and obtaining updated node features, a node compression operation is used to gradually reduce the number of nodes and extract richer context features. Specifically, the features of the nodes are arranged into a The two-dimensional characteristic matrix of , where C represents the dimension of the feature; then, the matrix is ​​evenly divided into Sub-blocks, each sub-block is flattened and mapped to C-dimensional features through a linear layer to form compressed node features In this implementation, For the parameters you set yourself, here you will If the value of is set to 2, the hypergraph structure will be rebuilt after the nodes are compressed.

[0044] It is understandable that although hypergraph convolution can effectively model the relationship between nodes and their high-order structural neighborhoods, its representation ability may still be limited by the linear combination form of the graph structure itself. Therefore, this application uses a LegendreKAN layer after hypergraph reconstruction to improve the nonlinear representation ability of node features.

[0045] like Figure 3 Specifically, the compressed node features are first normalized, and then the features of each channel are expanded to the highest degree using Legendre polynomials. (In this embodiment, The value of is set to 4), construct the target representation of the following form : ; in, Represents the number of polynomial expansions.

[0046] Then use a set of learnable parameters Perform linear combination on each node in the channel dimension to obtain nonlinear mapping output features : ; in, is the activation function.

[0047] Furthermore, the output features are hypergraph decoded to obtain fusion features, which specifically includes: Inputting the output features into the hypergraph upsampling unit of the hypergraph decoding submodule, performing node expansion on the output features by the hypergraph upsampling unit to obtain expanded node features; The expanded node features are input into the second hypergraph convolution unit, the expanded node features are decoded by the hypergraph convolution unit to obtain decoder features, and the output features are connected and fused with the decoder features to obtain fused features.

[0048] In this embodiment, the hypergraph decoding submodule is mainly composed of three expansion units, including a hypergraph upsampling unit and a hypergraph convolution unit. The purpose of the expansion unit is to gradually increase the number of nodes and restore the features of the original resolution to achieve more accurate classification. Each expansion module consists of two parts: first, a node expansion operation is performed, and then a graph convolution layer is connected. The node expansion operation is similar to the node compression in the encoder. A 2×2 upsampling convolution operation is used in the decoder to improve the spatial resolution of the feature map. Subsequently, in order to alleviate the over-smoothing problem in the graph neural network, this embodiment adopts a jump connection mechanism to connect and fuse the features from the corresponding layer of the encoder with the upsampled decoder features.

[0049] Step S40: Input the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and use a weighted majority voting mechanism to integrate the predicted classification results to obtain a target classification result.

[0050] Specifically, the fused features are input into the hierarchical prediction module of the classification model network, and each level image of the fused features is subjected to node-level classification by the hierarchical prediction module to obtain a plurality of prediction graphs.

[0051] In this embodiment, in order to obtain better classification performance, a hierarchical supervision strategy is adopted, that is, node-level classification is performed on each level of the decoder, thereby obtaining four prediction graphs, which are respectively denoted as ,in Correspondingly, the true label of the sth layer is recorded as ,in They represent the height, length, and number of categories of the real image respectively. The real label is generated by dividing and summarizing the original pixel-level label.

[0052] By calculating the cross entropy loss at multiple scales and updating the parameters of the hierarchical prediction module according to the cross entropy loss, coarse-to-fine hierarchical classification learning can be achieved. The loss function is as follows: ; in, According to the prediction graph and the true label Calculated cross entropy loss.

[0053] Furthermore, a weight is assigned to each of the prediction images according to a preset confidence level. Based on the weight, multiple prediction images are spliced ​​and restored into a complete full-field digital pathology slice prediction image based on a weighted majority voting mechanism, and the target classification result is obtained based on the full-field digital pathology slice prediction image.

[0054] Understandably, due to the overlap between cropped regions, the proposed framework generates multiple different predictions for the same patch during inference. To consolidate these potentially inconsistent patch predictions, this embodiment uses a weighted majority voting mechanism to aggregate these different predictions into the final target classification result.

[0055] Assume that an image block has prediction results, according to the prediction confidence Assign weights to the prediction results The final full-field digital pathology slide prediction image It is given by: ; ; in, Indicates the Among the predictions The probability of the class, is the indicator function, weight is the normalized entropy of the predicted probability distribution Calculated, where Indicates the The entropy of the prediction results, represents the norm function. This formula indicates that smaller entropy values ​​indicate more certain predictions and correspondingly greater weights, indicating greater confidence. Therefore, predictions with higher confidence in majority voting have a greater impact on the final classification result. Finally, all block-level predictions are concatenated to form a complete WSI-level prediction map. Since all pixels within each block are assumed to belong to the same class, the resolution of the prediction map matches the original WSI.

[0056] As can be seen, this application utilizes fine-tuning of a pre-trained large model to fully exploit breast cancer characteristics and enhance the model's perception of different tissue morphologies. It also designs a multi-level, dynamically constructed hypergraph UNet network to explicitly model the contextual structural dependencies between patches in WSI. Finally, it introduces a KAN nonlinear mapping layer (LegendreKAN) based on Legendre polynomials to enhance the representation of high-order features and improve the model's generalization performance. Multi-scale supervision and soft labeling are employed during training, and weighted majority voting is used to integrate multiple predictions during the inference phase. This improves model stability and consistency, enabling multi-classification of breast cancer histopathology images. The proposed method possesses strong patch-level discrimination and global structural modeling capabilities, making it particularly suitable for regional identification and detailed classification of breast cancer lesions in complex tissue morphologies.

[0057] Further, if Figure 4 As shown, based on the above-mentioned breast cancer tissue pathology image classification method, the present invention also provides a breast cancer tissue pathology image classification system, wherein the breast cancer tissue pathology image classification system includes: The classification model construction module 51 is used to construct a classification model network, which includes a feature extraction module, a feature fusion module and a hierarchical prediction module; a characterization feature extraction module 52 for obtaining full-field digital pathology sections of target tissue, cropping and splicing the full-field digital pathology sections, and then inputting the sections into the feature extraction module of the classification model network for high-dimensional characterization feature extraction to obtain characterization features; a hypergraph encoding and decoding module 53 for constructing a hypergraph based on the representation features, inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and performing hypergraph decoding on the output features to obtain fused features; The classification result output module 54 is used to input the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and integrate the predicted classification results using a weighted majority voting mechanism to obtain a target classification result.

[0058] Further, if Figure 5As shown, based on the above-mentioned breast cancer tissue pathology image classification method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0059] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal. Furthermore, the memory 20 may include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software installed on the terminal and various types of data, such as program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is about to be output. In one embodiment, the memory 20 stores a breast cancer tissue pathology image classification program 40, which can be executed by the processor 10, thereby implementing the breast cancer tissue pathology image classification method of the present application.

[0060] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 20, such as executing the breast cancer tissue pathology image classification method.

[0061] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0062] In one embodiment, when the processor 10 executes the breast cancer tissue pathology image classification program 40 in the memory 20, the following steps are implemented: Constructing a classification model network, the classification model network including: a feature extraction module, a feature fusion module and a hierarchical prediction module; Obtaining full-field digital pathology sections of target tissue, cropping and splicing the full-field digital pathology sections, and inputting the sections into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features; Constructing a hypergraph according to the representation features, inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and performing hypergraph decoding on the output features to obtain fused features; The fusion features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and the predicted classification results are integrated using a weighted majority voting mechanism to obtain a target classification result.

[0063] The full-field digital pathology sections are cropped and spliced, and then input into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features, specifically including: The full-field digital pathology slice is cut into multiple non-overlapping patches according to a fixed step size, and a preset number of adjacent patches are spliced ​​to obtain multiple regions; Wherein, the patch represents an image block, and the region represents an image region; Using the large model as a feature extractor, freezing all backbone parameters of the feature extractor, and performing supervised optimization on the newly introduced task-specific linear head to obtain a fine-tuned large pathology model; The multiple regions are input into the feature extraction module of the classification model network, and the patches in each region are subjected to feature extraction through the fine-tuned pathology model to obtain the characterization features of all the patches.

[0064] The step of constructing a hypergraph based on the representation features specifically includes: The features of each patch in the region are taken as nodes to obtain a node set, and the edges formed by the nodes are taken as hyperedges to obtain a hyperedge set; Constructing a hypergraph based on the node set and the hyperedge set, and calculating attention weights of features of all nodes of each hyperedge, obtaining similar nodes based on the attention weights, and constructing an association matrix corresponding to each hyperedge using the similar nodes; The node set includes all patches in a region, and the characteristics of the nodes are the characteristics of the corresponding patches. Each hyperedge in the hyperedge set is composed of two or more nodes.

[0065] Wherein, the feature fusion module includes a hypergraph encoding submodule and a hypergraph decoding submodule; The hypergraph encoding submodule includes a first hypergraph convolution unit, a hypergraph downsampling unit and a LegendreKAN layer; The hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit.

[0066] The step of inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features specifically includes: Inputting the hypergraph into the first hypergraph convolution unit, aggregating node features of the hypergraph to each hyperedge of the hypergraph through the hyperedge association matrix, and obtaining an initial hyperedge representation; Obtaining a central hyperedge feature through the correlation matrix, and performing a hyperedge feature update on the initial hyperedge representation using the central hyperedge feature to obtain an updated hyperedge feature; The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and update the node features according to the feedback information to obtain updated node features; Inputting the updated node features into the hypergraph downsampling unit of the hypergraph encoding submodule, arranging the updated node features according to their spatial positions to obtain a two-dimensional feature matrix, dividing the two-dimensional feature matrix into a plurality of blocks, flattening each block and inputting it into a linear layer for mapping to obtain compressed node features; The node features are input into the LegendreKAN layer of the hypergraph encoding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest degree using Legendre polynomials to construct a target representation, and the target representation is linearly combined using a set of learnable parameters to obtain output features.

[0067] The step of performing hypergraph decoding on the output features to obtain fusion features specifically includes: Inputting the output features into the hypergraph upsampling unit of the hypergraph decoding submodule, performing node expansion on the output features by the hypergraph upsampling unit to obtain expanded node features; The expanded node features are input into the second hypergraph convolution unit, the expanded node features are decoded by the hypergraph convolution unit to obtain decoder features, and the output features are connected and fused with the decoder features to obtain fused features.

[0068] The step of inputting the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a prediction classification result, and integrating the prediction classification results using a weighted majority voting mechanism to obtain a target classification result specifically includes: Inputting the fused features into the hierarchical prediction module of the classification model network, performing node-level classification on each level image of the fused features through the hierarchical prediction module to obtain multiple prediction graphs; A weight is assigned to each prediction image according to a preset confidence level. Based on the weight and a weighted majority voting mechanism, multiple prediction images are spliced ​​and restored into a complete full-field digital pathology slice prediction image, and a target classification result is obtained based on the full-field digital pathology slice prediction image.

[0069] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a breast cancer tissue pathology image classification program, and when the breast cancer tissue pathology image classification program is executed by a processor, the steps of the breast cancer tissue pathology image classification method described above are implemented.

[0070] In summary, the present invention provides a method, system, and terminal for classifying breast cancer tissue pathology images. The method comprises: constructing a classification model network, obtaining full-field digital pathology slides of the target tissue, cropping and splicing the full-field digital pathology slides, and then inputting them into a feature extraction module of the classification model network for high-dimensional representation feature extraction to obtain representation features; constructing a hypergraph based on the representation features, inputting the hypergraph into a feature fusion module for hypergraph encoding to obtain output features, and performing hypergraph decoding on the output features to obtain fused features; inputting the fused features into a hierarchical prediction module of the classification model network for node-level classification to obtain predicted classification results, and integrating the predicted classification results using a weighted majority voting mechanism to obtain the target classification results. The present invention combines large-scale model fine-tuning with a hierarchical hypergraph UNet structure to effectively improve the accuracy of breast cancer tissue pathology image classification.

[0071] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0072] Of course, those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0073] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for classifying breast cancer tissue pathology images, characterized in that: The breast cancer tissue pathology image classification method includes: Constructing a classification model network, the classification model network including: a feature extraction module, a feature fusion module and a hierarchical prediction module; Obtaining full-field digital pathology sections of target tissue, cropping and splicing the full-field digital pathology sections, and inputting the sections into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features; Constructing a hypergraph according to the representation features, inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and performing hypergraph decoding on the output features to obtain fused features; The fusion features are input into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and the predicted classification results are integrated using a weighted majority voting mechanism to obtain a target classification result.

2. The breast cancer tissue pathology image classification method according to claim 1, characterized in that: After the full-field digital pathology sections are cropped and spliced, they are input into the feature extraction module of the classification model network to extract high-dimensional representation features to obtain representation features, specifically including: The full-field digital pathology slice is cut into multiple non-overlapping patches according to a fixed step size, and a preset number of adjacent patches are spliced ​​to obtain multiple regions; Wherein, the patch represents an image block, and the region represents an image region; Using the large model as a feature extractor, freezing all backbone parameters of the feature extractor, and performing supervised optimization on the newly introduced task-specific linear head to obtain a fine-tuned large pathology model; The multiple regions are input into the feature extraction module of the classification model network, and the patches in each region are subjected to feature extraction through the fine-tuned pathology model to obtain the characterization features of all the patches.

3. The breast cancer tissue pathology image classification method according to claim 2, characterized in that: The constructing of a hypergraph according to the representation features specifically includes: The features of each patch in the region are taken as nodes to obtain a node set, and the edges formed by the nodes are taken as hyperedges to obtain a hyperedge set; Constructing a hypergraph based on the node set and the hyperedge set, and calculating attention weights of features of all nodes of each hyperedge, obtaining similar nodes based on the attention weights, and constructing an association matrix corresponding to each hyperedge using the similar nodes; The node set includes all patches in a region, and the characteristics of the nodes are the characteristics of the corresponding patches. Each hyperedge in the hyperedge set is composed of two or more nodes.

4. The breast cancer tissue pathology image classification method according to claim 3, characterized in that: The feature fusion module includes a hypergraph encoding submodule and a hypergraph decoding submodule; The hypergraph encoding submodule includes a first hypergraph convolution unit, a hypergraph downsampling unit and a LegendreKAN layer; The hypergraph decoding submodule includes a hypergraph upsampling unit and a second hypergraph convolution unit.

5. The breast cancer tissue pathology image classification method according to claim 4, characterized in that: Inputting the hypergraph into the feature fusion module for hypergraph encoding to obtain output features specifically includes: Inputting the hypergraph into the first hypergraph convolution unit, aggregating node features of the hypergraph to each hyperedge of the hypergraph through the hyperedge association matrix, and obtaining an initial hyperedge representation; Obtaining a central hyperedge feature through the correlation matrix, and performing a hyperedge feature update on the initial hyperedge representation using the central hyperedge feature to obtain an updated hyperedge feature; The nodes of the hypergraph respectively receive feedback information from the updated hyperedge features associated with themselves, and update the node features according to the feedback information to obtain updated node features; Inputting the updated node features into the hypergraph downsampling unit of the hypergraph encoding submodule, arranging the updated node features according to their spatial positions to obtain a two-dimensional feature matrix, dividing the two-dimensional feature matrix into a plurality of blocks, flattening each block and inputting it into a linear layer for mapping to obtain compressed node features; The node features are input into the LegendreKAN layer of the hypergraph encoding submodule, the compressed node features are normalized, the normalized results on each channel are expanded to the highest degree using Legendre polynomials to construct a target representation, and the target representation is linearly combined using a set of learnable parameters to obtain output features.

6. The breast cancer tissue pathology image classification method according to claim 4, characterized in that: The hypergraph decoding of the output features to obtain fusion features specifically includes: Inputting the output features into the hypergraph upsampling unit of the hypergraph decoding submodule, performing node expansion on the output features by the hypergraph upsampling unit to obtain expanded node features; The expanded node features are input into the second hypergraph convolution unit, the expanded node features are decoded by the hypergraph convolution unit to obtain decoder features, and the output features are connected and fused with the decoder features to obtain fused features.

7. The breast cancer tissue pathology image classification method according to claim 1, characterized in that: The step of inputting the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a prediction classification result, and integrating the prediction classification results using a weighted majority voting mechanism to obtain a target classification result specifically includes: Inputting the fused features into the hierarchical prediction module of the classification model network, performing node-level classification on each level image of the fused features through the hierarchical prediction module to obtain multiple prediction graphs; A weight is assigned to each prediction image according to a preset confidence level. Based on the weight and a weighted majority voting mechanism, multiple prediction images are spliced ​​and restored into a complete full-field digital pathology slice prediction image, and a target classification result is obtained based on the full-field digital pathology slice prediction image.

8. A breast cancer tissue pathology image classification system, characterized in that: The breast cancer tissue pathology image classification system includes: A classification model building module is used to build a classification model network, which includes a feature extraction module, a feature fusion module and a hierarchical prediction module; a characterization feature extraction module, configured to obtain full-field digital pathology sections of target tissue, crop and splice the full-field digital pathology sections, and then input the sections into the feature extraction module of the classification model network to extract high-dimensional characterization features to obtain characterization features; A hypergraph encoding and decoding module, configured to construct a hypergraph based on the representation features, input the hypergraph into the feature fusion module for hypergraph encoding to obtain output features, and perform hypergraph decoding on the output features to obtain fused features; The classification result output module is used to input the fusion features into the hierarchical prediction module of the classification model network for node-level classification to obtain a predicted classification result, and integrate the predicted classification results using a weighted majority voting mechanism to obtain a target classification result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a breast cancer tissue pathology image classification program stored in the memory and executable on the processor. When the breast cancer tissue pathology image classification program is executed by the processor, the steps of the breast cancer tissue pathology image classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a breast cancer tissue pathology image classification program, which, when executed by a processor, implements the steps of the breast cancer tissue pathology image classification method according to any one of claims 1 to 7.

Images