A bone marrow pathological image analysis system based on multi-view clustering

Abstract

The application belongs to the technical field of medical image processing and deep learning, and discloses a bone marrow pathological image analysis system based on multi-view clustering. The system firstly acquires bone marrow whole section images and corresponding pathological reports, extracts multi-scale image features and text semantic features respectively; then constructs a pyramid hypergraph fusing space topology and semantic association, aggregates multi-scale instance information through a graph attention network to generate image-level global features; subsequently, adaptively learns a cross-modal consensus prototype, calculates semantic similarity to obtain soft assignment probability and solves optimal pseudo labels; finally, realizes feature alignment of double modes with the help of symmetric cross-entropy knowledge distillation and entropy regularization, and outputs classification results through clustering assignment.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and deep learning technology, and more specifically to a bone marrow pathology image analysis system based on multi-view clustering. Background Technology

[0002] Bone marrow sections play a central role in the diagnosis of hematological diseases, leukemia, and other conditions. With the development of digital pathology technology, traditional bone marrow slides can be scanned into ultra-high resolution whole-slice images. WSI These images can clearly present microscopic information such as the morphology, distribution, and tissue structure of bone marrow cells, providing crucial information for doctors to determine the degree of hematopoietic cell proliferation, identify abnormal cell morphology, and thus achieve disease diagnosis, classification, and prognostic assessment. However, high resolution also brings a huge amount of data, requiring a single bone marrow image to generate a large amount of data. WSI Reachable GB The diverse cell morphologies and complex tissue backgrounds present significant challenges to automated analysis.

[0003] Currently, automated analysis methods for whole bone marrow slide pathological images are mainly divided into two categories: those based on local regions (…). Patch-based The method and the method based on whole slices ( Slide-level The method.

[0004] The first type is the local area analysis method, the core process of which is to... WSI Cut into a large number of small image blocks of fixed size ( Patch ), for each Patch Feature extraction and classification are performed independently, and the results are then fused using strategies such as voting, thresholding, or clustering to ultimately provide slice-level diagnostic conclusions. While this method can adapt to limited GPU memory and utilize pre-trained models, it also has significant shortcomings: firstly, the segmentation process disrupts the original spatial continuity of the image, leading to microscopic... Patch The loss of contextual associations with macroscopic organizational structures makes it difficult for models to capture overall distribution information such as "trabeculae-mesenchyma-cell populations"; secondly, if one wants to... Patch High-level fine-grained annotation improves model performance, requiring pathologists to process massive amounts of data. Patch Manual annotation is extremely costly and practically impossible in clinical practice; thirdly, the images contain a large number of non-interest-bearing regions (such as fat cavities and hemorrhage areas), which are redundant. Patch It will consume a lot of computing resources, and if not effectively screened, it can easily lead to model overfitting and low inference efficiency.

[0005] The second type, the whole-image classification method, aims to overcome the fragmentation problem of local analysis. This method typically involves classifying the entire image. WSIAfter downsampling and preprocessing, the data can be directly input into a lightweight neural network for end-to-end prediction, or global features can be extracted using a convolutional neural network and then classified using pooling or attention mechanisms. Its advantage lies in preserving the overall structural information of the slices, and the process is relatively simple. However, this method also has inherent drawbacks: First, downsampling loses key microscopic details such as cell nuclear morphology and mitotic figures at high resolution, reducing the model's sensitivity to minor lesions; second, different diseases require different optimal observation scales for diagnosis (e.g., myelofibrosis requires attention to macroscopic tissue texture, while acute leukemia requires focus on microscopic cell morphology), making it difficult for a single downsampling strategy to meet multiple diagnostic needs; finally, with only weak supervision labels at the slice level, the model lacks supervision signals for local regions, making it difficult to learn cell-level discriminative features, easily leading to overfitting or unstable prediction results.

[0006] To address the aforementioned challenges, especially in weakly supervised scenarios, slice-level label learning and multi-instance learning are employed. MIL The framework was introduced WSI Analysis. Within this framework, the entire sheet WSI The image is treated as a "package," with the cut-out image patches considered as "instances." Model training requires only slide-level diagnostic labels, significantly reducing the reliance on fine-grained annotations and promising to preserve more complete contextual information. However, existing... MIL The methods still have the following key unresolved issues: First, most methods still rely on slide-level labels as supervision signals. In completely unsupervised scenarios, the accuracy of package-level classification is generally low, making it difficult to meet the reliability requirements of clinical diagnosis; Second, WSI Image information typically contains information at different magnifications, and the optical properties and information density vary significantly across scales. Direct fusion can easily lead to semantic gaps and feature conflicts; thirdly, in positive... WSI In this context, critical lesion areas often constitute a small percentage, while background areas dominate, leading to a severe imbalance between positive and negative instances. Traditional max-pooling-based methods... Max-Pooling )of MIL Methods tend to overlook the relationships between instances, while attention-based methods are susceptible to noise interference due to the lack of explicit distance or structural constraints, and have limited ability to identify key instances.

[0007] In conclusion, current technologies have failed to adequately address the issue of bone marrow damage. WSI The analysis focuses on key challenges such as the coordinated use of global context and local details, effective learning under weak supervision, and multi-scale information fusion. Summary of the Invention

[0008] In view of the above problems, the present invention provides a bone marrow pathology image analysis system based on multi-view clustering that overcomes or at least partially solves the above problems.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: including a data acquisition module, an image multi-scale feature extraction module, a text semantic feature extraction module, a hypergraph construction and feature aggregation module, a pseudo-label optimization module, a cross-modal feature alignment module, and a classification result generation module.

[0010] The data acquisition module is used to acquire whole bone marrow slide pathological images and corresponding original diagnostic reports as well as consensus semantic prototypes;

[0011] The image multi-scale feature extraction module is used to perform multi-scale segmentation on the whole bone marrow pathological image to obtain multi-scale image features; the text semantic feature extraction module is used to preprocess the original diagnostic report to obtain text semantic features.

[0012] The hypergraph construction and feature aggregation module is used to construct a pyramid semantic hypergraph, which integrates the multi-scale information and spatial topology of the image's multi-scale features to generate image-level global features.

[0013] The pseudo-label optimization module is used to generate a set of consensus prototypes based on the interaction information between the image-level global features and the text semantic features, calculate the semantic similarity between the image-level global features and the text semantic features and the consensus prototypes, and obtain the first soft allocation probability and the second soft allocation probability, so as to maximize the overall similarity between the image-level global features and the text semantic features and the consensus prototypes as the optimization objective to solve for the optimal pseudo-label.

[0014] The cross-modal feature alignment module is used to optimize the first soft assignment probability and the second soft assignment probability through the knowledge distillation loss function using the optimal pseudo-label as a supervision signal, so as to realize the mutual supervision and alignment of the image-level global features and the text semantic features.

[0015] The classification result generation module is used to perform clustering and allocation based on the aligned image-level global features and the text semantic features, and output the classification result.

[0016] Preferably, the image multi-scale feature extraction module includes:

[0017] A multi-resolution cutting unit is used to perform multi-resolution cutting on the whole bone marrow digital pathological image while maintaining spatial alignment between scales.

[0018] Background filtering unit, used to filter blank background areas based on grayscale statistics;

[0019] The feature extraction unit is used to extract feature representations of image patches at various scales to form multi-scale image features.

[0020] Preferably, the text semantic feature extraction module includes:

[0021] A text preprocessing unit is used to perform text cleaning and standardization on the original diagnostic report;

[0022] A key entity identification unit is used to identify and extract key medical entities from the original diagnostic report;

[0023] Entity standardization unit, used to map key medical entities to unified concept identifiers through a medical terminology standardization system;

[0024] The semantic encoding unit is used to semantically encode standardized key medical entities using a pre-trained language model to obtain textual semantic features.

[0025] Preferably, the hypergraph construction and feature aggregation module includes a spatial topology construction unit, which is used for:

[0026] Based on the multi-scale image features, spatial adjacency relationships and semantic association relationships between image features at the same scale are constructed to form topologies within multiple scales;

[0027] And to construct hierarchical inclusion relationships of image features based on coordinate mapping and semantic complementarity relationships based on feature similarity at different scales, so as to form cross-scale topology;

[0028] The intra-scale topology and cross-scale topology are fused to form a spatial topology structure.

[0029] Furthermore, the hypergraph construction and feature aggregation module also includes a feature fusion unit, which is used for:

[0030] For each constructed scale-in-topology, its corresponding scale image features are used as the first node features. The first graph attention network is used to fuse the first node features and the adjacency matrix of the scale-in-topology to obtain the node aggregation features of that scale.

[0031] The node aggregation features obtained at each scale are spliced ​​together to form a second node feature. The second node feature is then fused with the adjacency matrix of the cross-scale topology through a graph attention network to obtain the enhanced features of all image patches.

[0032] Enhancement features of all the image patches are performed Kmeans Semantic clustering is performed, and attention weights are calculated based on the affiliation of each image patch with each semantic category.

[0033] The enhanced features of all image patches are weighted and summed according to their corresponding attention weights to obtain the image-level global features.

[0034] Preferably, the formula for calculating the optimal pseudo-label by the pseudo-label optimization module is as follows:

[0035]

[0036]

[0037] in, This is a consensus prototype matrix; Image-level global features; Textual semantic features; and These are the optimal pseudo-labels for image-level global features and text semantic features, respectively.

[0038] Furthermore, the knowledge distillation loss function in the cross-modal feature alignment module is a symmetric cross-entropy loss function:

[0039]

[0040] in, and These are the first soft allocation probability and the second soft allocation probability, respectively.

[0041] Furthermore, the cross-modal feature alignment module also includes a regularization unit, which is used to introduce an entropy regularization term:

[0042]

[0043] Total loss function: .

[0044] Furthermore, the classification result generation module is used for:

[0045] Obtain the optimized first soft assignment probability and the second soft assignment probability, and calculate their average probability;

[0046] The category with the highest average probability is selected as the classification result output.

[0047] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0048] 1. Based on multi-scale image and text dual-modal multi-scale self-supervised learning, the semantics of the image are anchored by expert priors in the text such as "metastasis type" and "lesion location" (e.g., "axillary lymph node micrometastasis"), thus getting rid of the dependence on slide-level labels;

[0049] 2. To address the multi-scale characteristics of whole bone marrow pathological images, a hypergraph structure with hierarchical edges and semantically complementary edges is designed. This structure retains the logic of macroscopic localization and microscopic verification, and repairs the multi-scale semantic gap through hypergraph topology, thus solving the problem of traditional fusion methods that focus on the overall picture but lose details or get bogged down in details and lose the overall picture.

[0050] 3. By leveraging a multi-view learning mechanism based on consensus-based contrastive clustering, the positive pathological features of semantic coefficients in images are enhanced by coordinating consistent complementary information between images and text, thereby obtaining cluster-friendly representations and outputting classification results. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is an overall architecture diagram of a bone marrow pathology image analysis system based on multi-view clustering in an embodiment of the present invention;

[0053] Figure 2 This is a detailed structural diagram and feature mapping relationship diagram of the text semantic feature extraction module and the image multi-scale feature extraction module in this embodiment of the invention;

[0054] Figure 3 This is a schematic diagram illustrating the working principle of the hypergraph construction and feature aggregation module in an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] like Figure 1 As shown in the figure, this invention discloses a bone marrow pathology image analysis system based on multi-view clustering, including a data acquisition module, an image multi-scale feature extraction module, a text semantic feature extraction module, a hypergraph construction and feature aggregation module, a pseudo-label optimization module, a cross-modal feature alignment module, and a classification result generation module.

[0057] The data acquisition module is used to acquire whole bone marrow slide pathological images, corresponding original diagnostic reports, and consensus semantic prototypes.

[0058] The image multi-scale feature extraction module is used to perform multi-scale segmentation of whole bone marrow pathological images to obtain multi-scale image features; the text semantic feature extraction module is used to preprocess the original diagnostic report to obtain text semantic features.

[0059] The hypergraph construction and feature aggregation module is used to construct a pyramid semantic hypergraph, which integrates multi-scale information and spatial topology of image features to generate image-level global features.

[0060] The pseudo-label optimization module is used to generate a set of consensus prototypes based on the interaction information between the image-level global features and the text semantic features, calculate the semantic similarity between the image-level global features and the text semantic features and the consensus prototypes, and obtain the first soft allocation probability and the second soft allocation probability. The optimization objective is to maximize the overall similarity between the image-level global features and the text semantic features and the consensus prototypes to solve for the optimal pseudo-label.

[0061] The cross-modal feature alignment module is used to optimize the first soft assignment probability and the second soft assignment probability through the knowledge distillation loss function using the optimal pseudo-label as the supervision signal, so as to achieve mutual supervision and alignment between image-level global features and text semantic features.

[0062] The classification result generation module is used to perform clustering assignment based on aligned image-level global features and text semantic features, and output the classification results.

[0063] like Figure 2 As shown, in one embodiment, the image multi-scale feature extraction module includes:

[0064] The multi-resolution cutting unit is used to perform multi-resolution cutting of whole bone marrow digital pathology images while maintaining spatial alignment between scales.

[0065] Background filtering unit, used to filter blank background areas based on grayscale statistics;

[0066] The feature extraction unit is used to extract feature representations of image patches at various scales to form multi-scale image features.

[0067] Specifically, the image multi-scale feature extraction module is used for whole bone marrow slide pathological images ( WSI Perform instance segmentation and preliminary feature extraction:

[0068] Multi-resolution cutting unit for WSI Instance splitting: utilizing OpenSlide Library WSI According to different resolutions ( X5 / X20 Cut to ensure spatial alignment (e.g., 1) X5 patch Corresponding to 4×4 X20 son patchTo ensure that other resolutions are consistent with X5 segmentation patch Spatial coverage consistency.

[0069] Background filtering unit is too much background filtering: By Otsu Threshold calculation patch Grayscale variance, filtering out blank backgrounds with variance below a threshold. patch Reduce noise interference.

[0070] The feature extraction unit is used for feature pre-extraction: a pre-trained model is used, such as... Dino Extract each patch Image features yield a multi-scale representation set. {X1,X2} , used to characterize multi-scale image I.

[0071] In one embodiment, the text semantic feature extraction module includes:

[0072] The text preprocessing unit is used to clean and standardize the original diagnostic report.

[0073] Key entity identification unit, used to identify and extract key medical entities in the original diagnostic report;

[0074] Entity standardization unit, used to map key medical entities to unified concept identifiers through a medical terminology standardization system;

[0075] The semantic encoding unit is used to semantically encode standardized key medical entities using a pre-trained language model to obtain textual semantic features.

[0076] Specifically, the text semantic feature extraction module extracts text representations T of relevant pathological descriptive semantic information from the original diagnostic report:

[0077] Text preprocessing unit: Standardizes the format of the original diagnostic report (removes special symbols, unifies capitalization, and breaks long sentences), filters out non-diagnostic noise such as test information, and retains core diagnostic descriptions such as tumor location and lesion type.

[0078] Key entity recognition unit: Sampled pre-trained language model in the medical field BioBERT / BioClinicalBERT Model combined with named entity recognition ( NER The task involves extracting core entities such as cancer location, lesion type, and metastasis status; the extremely long report is supplemented with... Path-BigBird Sliding windows ensure the connection between entities across paragraphs.

[0079] Standardized physical units: through a unified medical language system ( UMLS Mapping entities to standardized concept identifiers ( CUI This addresses synonyms and terminological heterogeneity, achieving domain-specific alignment of pathological semantics.

[0080] Semantic coding unit: based on BERT The pre-trained model generates fixed-dimensional text semantic feature vectors.

[0081] against Figure 2 Key annotation explanation:

[0082] Entity tokens 1-3: These correspond to the three core entities extracted by the text semantic feature extraction module: “cancer location”, “lesion type”, and “metastatic status”. They are the key inputs to the semantic coding unit.

[0083] Common subspace: a 1536-dimensional feature space used to represent text. T (Linear layer mapping) and image representation I Alignment ensures that bimodal information can be directly fused;

[0084] Projection layer: Located between the text encoder and the common subspace, it represents the 768-dimensional text. T Mapped to 1536 dimensions, corresponding to image representation I Dimensions are consistent.

[0085] like Figure 3 As shown, in one embodiment, the hypergraph construction and feature aggregation module includes a spatial topology construction unit, which is used for:

[0086] Based on multi-scale image features, spatial adjacency relationships and semantic association relationships between image features at the same scale are constructed to form topologies within multiple scales;

[0087] And to construct hierarchical inclusion relationships of image features based on coordinate mapping and semantic complementarity relationships based on feature similarity at different scales, so as to form cross-scale topology;

[0088] The spatial topology is formed by merging intra-scale topology and cross-scale topology.

[0089] Specifically, the spatial topology building unit constructs an adjacency hypergraph between instances for multi-scale images, which can be divided into two pyramid layers:

[0090] Top layer (low resolution): X5 Scale-based semantic topology graph structure It reflects the global distribution, such as the extent of tumor infiltration.

[0091] Low layer (high resolution): X20 Scale-based semantic topology graph structure Focusing on cellular details such as the morphology of the nucleolus.

[0092] semantic topology graph between instances within the scale and Build:

[0093] Constructing the same subview collaboratively from both spatial topology and semantic attributes patch Relationship, the joint effect of two adjacent edges, both retain WSI The inherent spatial structure further strengthens the semantic association of a single view:

[0094] Spatial Adjacency Edges: Each edge is connected according to "8 - Connected Chessboard Distance". patch Physically adjacent in 8 directions patch Set as "spatial neighbor", create spatial adjacency edges, and retain WSI Spatial structure (such as glands, continuity of pathological tissues).

[0095] Semantic adjacency edges: utilizing kmeans Clustering selection A consensus semantic instance, constructing other consensus semantic patches. patch Anchor charts and all patch Similar images, before filtering Top-N semantically similar patch Establish semantic edges and aggregate instances of the same pathological attribute.

[0096] Cross-scale instance topology graph Build:

[0097] Similar to intra-scale topological graph construction methods, this approach collaboratively models the hierarchical relationships of cross-scale instance hyperedges from two aspects: "structural association" of coordinate mapping and "semantic association" of feature similarity.

[0098] Hierarchy includes associations: WSI Global coordinate mapping to establish low resolution patch (Parent node) and high resolution patch The "containment-being-contained" spatial relationship of (child nodes) depicts the correspondence between the "macro-micro" spatial structures.

[0099] Unified multi-scale coordinate system: WSI Original maximum resolution X20 Based on the benchmark, the calculation scale X5 Corresponding scaling factor Then the scale X5 of patch Physical dimensions = X20 Base size × ;

[0100] Father-Son patch Spatial extent mapping: Calculate each scale separately X5 / X20 p atch exist X20 The coverage of the coordinate system and ,like The scope is completely contained within If it is inside, then it is determined to be For the parent node, For each child node, establish a hierarchical containment edge between them.

[0101] Semantic complementarity association: Feature similarity measurement associates low-resolution global semantics with high-resolution local features to achieve “global-local” semantic alignment.

[0102] Global semantic learning and local detail extraction: such as X5 low resolution patch The global semantic vector is obtained by global mean pooling. X20 high resolution patch Then directly use Dino Preprocessed features As a local feature vector .

[0103] Semantic relevance matching: Calculate the cosine similarity between the representation vectors of instances at different scales, and construct semantic complementary edges between cross-scale instances with high similarity.

[0104] In one embodiment, the hypergraph construction and feature aggregation module further includes a feature fusion unit, which is used for:

[0105] For each scale-in-topology constructed, its corresponding scale image features are used as the first node features. The first graph attention network is used to fuse the first node features and the adjacency matrix of the scale-in-topology to obtain the node aggregation features of that scale.

[0106] The node aggregation features obtained at each scale are spliced ​​together to form the second node features. The second node features are then fused with the adjacency matrix of the cross-scale topology through a graph attention network to obtain the enhanced features of all image patches.

[0107] Enhancement features for all image patches Kmeans Semantic clustering is performed, and attention weights are calculated based on the affiliation of each image patch with each semantic category.

[0108] The enhanced features of all image patches are weighted and summed according to their corresponding attention weights to obtain the image-level global features.

[0109] Specifically, the feature fusion unit utilizes a graph attention mechanism ( GAT This module aggregates multi-scale image features of the pyramid hypergraph and contains two... GNN Layer: First layer Correlation processing of image patches at the same scale (i.e., processing multiple images within one scale) patch (relationship); another layer Enable interaction between scales to ensure that image patches of different resolutions share contextual information.

[0110]

[0111] in and Represent X5 low resolution and X20 Patch features of high-resolution WSI images and , and Represent X5 and X20 WSI images construct intra-scale semantic topology graphs between instances based on their respective patches. Then it means based on X5 and X20 A cross-scale semantic topology graph jointly constructed from WSI images. "express Layers and Cascading operations of layers, that is, first through Learn separately X5 and X20 Based on graph structure and Graph representation vector and Then splice the representation enter Layers based on cross-scale graph structure Learning integrates X5 and X20 Graph representation vectors of multi-scale features .

[0112] Intra-hierarchical hypergraph convolution : at the current scale patch feature For input, use GAT Based on adjacency matrix Aggregate similar features of neighboring nodes while preserving information specific to a single scale (such as cell details in X20 and global distribution in X5):

[0113]

[0114] in GAT At the computing node When considering inter-edge attention, the dynamic weights of the hyperedges should also be taken into account. Similarity to node features:

[0115]

[0116] and These are the attention vector and the linear transformation matrix, respectively. and They represent the first The nodes correspond to the image features of the patch. It is worth noting that... This represents the dimension of the node features after the linear transformation, in order to avoid the problem of excessively large values ​​when the dot product of the linearly transformed features and the attention vector is performed.

[0117] Cross-level hypergraph convolution : Piecing together the aggregated graph representations from each scale level Enter after GAT And based on cross-scale hypergraphs Information fusion based on topological relationships:

[0118]

[0119] in It is an enhanced feature that aggregates all image patches.

[0120] Multi-scale node features based on pyramid hypergraph aggregation Generate through attention aggregation guided by consensus semantics bag Level characterization, highlighting key pathological information (such as tumor regions):

[0121] Consensus semantic attention weight: for all nodes conduct Kmeans Clustering to obtain consensus semantic prototypes For example, it represents pathological semantics such as normal, inflammation, and tumor. Calculate each node. Compared to the latest prototype Class semantic similarity: .in It is a node Multi-scale fusion representations were learned through pyramid hypergraph aggregation. For the first k A representation vector of a consensus prototype.

[0122] bag Level representation fusion: summing all node representations by weighting them according to global semantic attention. This yields image-level global features. For nodes semantic similarity of classes It is a node Multi-scale fusion representations are learned through pyramid hypergraph aggregation.

[0123] against Figure 3 Key annotation explanation:

[0124] Instance attention score: for each patch Cosine similarity to consensus prototype A higher value indicates a greater contribution of the patch to disease diagnosis (e.g., the higher the value of the tumor patch). Typically >0.8, background patch <0.2);

[0125] Positive / negative labeling: in bag The label on the right side of the level characterization generation layer represents the "positive pathology" after attention weight filtering. patch "(like AML primitive myeloid cells patch ) and "negative background" patch (such as normal interstitium) patch This intuitively demonstrates the model's ability to focus on key areas;

[0126] Pyramid arrow: An arrow pointing from the bottom (X20) to the top (X5) represents the logic of microscopic details supporting macroscopic judgments, which is consistent with the reading habits of pathologists who first look at cell morphology and then look at the overall distribution.

[0127] In one embodiment, the formula for calculating the optimal pseudo-label by the pseudo-label optimization module is as follows:

[0128]

[0129]

[0130] in, This is a consensus prototype matrix; Image-level global features; Text semantic features and These are the optimal pseudo-labels for image-level global features and text semantic features, respectively.

[0131] Specifically, image semantic representation Text semantic representation matrix Projecting to consensus prototype Maximize the similarity between the representation and the prototype, and solve for the optimal pseudo-label. :

[0132]

[0133]

[0134] in, This is a consensus prototype matrix; Image-level global features; Textual semantic features; and The optimal pseudo-labels are those for image-level global features and textual semantic features.

[0135] In one embodiment, the knowledge distillation loss function in the cross-modal feature alignment module is a symmetric cross-entropy loss function:

[0136]

[0137] in, and These are the first soft allocation probability and the second soft allocation probability, respectively.

[0138] Specifically, adaptive learning involves a set of consensus prototypes that simultaneously carry image and text semantics. The optimized prototype can match both WSI image features (such as tumor cell morphology) and textual semantic features (such as "axillary lymph node micrometastasis"), and is optimized through the following objective function:

[0139]

[0140] in I For image-level global features, T For text semantic features (mapped to 1536 dimensions via a linear layer), ensure C Simultaneously align image features with text semantic features.

[0141] Image representation Calculate its connection to each consensus prototype Soft assignment probability:

[0142]

[0143] in The predicted label vector (soft-assigned probability set) for image samples is obtained similarly to the predicted label vector (soft-assigned probability set) for text. Temperature parameters =0.1.

[0144] By leveraging the pairing relationship between images and text—that is, the whole bone marrow slice image and the corresponding diagnostic report of the same patient will necessarily point to the same disease category—the clustering results of the samples, dominated by their respective soft assignment probabilities, tend to be consistent. This is achieved through a distillation function.

[0145]

[0146] ; For mutual supervision, false labels.

[0147] It should be noted that the core objective of self-knowledge distillation is to ensure consistency between the two clustering results through bidirectional supervision. Essentially, it aims to ensure that the soft assignment probabilities of image features and text features not only conform to the disease semantics themselves, but also reach a consensus on the category judgment of the same sample.

[0148] In one embodiment, the cross-modal feature alignment module further includes an introduction regularization unit, which is used to introduce an entropy regularization term:

[0149]

[0150] Total loss function: .

[0151] Specifically, an entropy regularization term is introduced into the distillation loss function to avoid overfitting by pseudo-labels.

[0152]

[0153] The total loss function is obtained as follows: .

[0154] In one embodiment, the classification result generation module is used to:

[0155] Obtain the optimized first soft assignment probability and second soft assignment probability, and calculate the average probability of the two;

[0156] Select the category with the highest average probability as the final diagnosis result and output it.

[0157] Specifically, the classification result generation module selects the one with the highest average probability as the classification result based on the optimized first soft assignment probability and second soft assignment probability.

[0158] It should be noted that the optimized first and second soft assignment probabilities are obtained by using the aforementioned optimal pseudo-label as the core supervision signal, through the knowledge distillation loss function acting alone, or through the synergistic effect of the total loss function formed by introducing an entropy regularization term, and through multiple rounds of iterative optimization and gradient backpropagation adjustment. The core objective is to achieve mutual supervision and semantic alignment between the first soft assignment probability of the image modality and the second soft assignment probability of the text modality, so that the two tend to be consistent in the consensus semantic prototype space, providing accurate and reliable probabilistic basis for subsequent clustering assignment and classification result output.

[0159] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A bone marrow pathology image analysis system based on multi-view clustering, characterized in that, It includes a data acquisition module, an image multi-scale feature extraction module, a text semantic feature extraction module, a hypergraph construction and feature aggregation module, a pseudo-label optimization module, a cross-modal feature alignment module, and a classification result generation module. The data acquisition module is used to acquire whole bone marrow slide pathological images and corresponding original diagnostic reports as well as consensus semantic prototypes; The image multi-scale feature extraction module is used to perform multi-scale segmentation on the whole bone marrow pathological image to obtain multi-scale image features; the text semantic feature extraction module is used to preprocess the original diagnostic report to obtain text semantic features. The hypergraph construction and feature aggregation module is used to construct a pyramid semantic hypergraph, which integrates the multi-scale information and spatial topology of the image's multi-scale features to generate image-level global features. The pseudo-label optimization module is used to generate a set of consensus prototypes based on the interaction information between the image-level global features and the text semantic features, calculate the semantic similarity between the image-level global features and the text semantic features and the consensus prototypes, and obtain the first soft allocation probability and the second soft allocation probability, so as to maximize the overall similarity between the image-level global features and the text semantic features and the consensus prototypes as the optimization objective to solve for the optimal pseudo-label. The cross-modal feature alignment module is used to optimize the first soft assignment probability and the second soft assignment probability through the knowledge distillation loss function using the optimal pseudo-label as a supervision signal, so as to realize the mutual supervision and alignment of the image-level global features and the text semantic features. The classification result generation module is used to perform clustering and allocation based on the aligned image-level global features and the text semantic features, and output the classification result.

2. The bone marrow pathology image analysis system based on multi-view clustering according to claim 1, characterized in that, The image multi-scale feature extraction module includes: A multi-resolution cutting unit is used to perform multi-resolution cutting on the whole bone marrow digital pathological image while maintaining spatial alignment between scales. Background filtering unit, used to filter blank background areas based on grayscale statistics; The feature extraction unit is used to extract feature representations of image patches at various scales to form multi-scale image features.

3. The bone marrow pathology image analysis system based on multi-view clustering according to claim 1, characterized in that, The text semantic feature extraction module includes: A text preprocessing unit is used to perform text cleaning and standardization on the original diagnostic report; A key entity identification unit is used to identify and extract key medical entities from the original diagnostic report; Entity standardization unit, used to map key medical entities to unified concept identifiers through a medical terminology standardization system; The semantic encoding unit is used to semantically encode standardized key medical entities using a pre-trained language model to obtain textual semantic features.

4. The bone marrow pathology image analysis system based on multi-view clustering according to claim 1, characterized in that, The hypergraph construction and feature aggregation module includes a spatial topology construction unit, which is used for: Based on the multi-scale image features, spatial adjacency relationships and semantic association relationships between image features at the same scale are constructed to form topologies within multiple scales; And to construct hierarchical inclusion relationships of image features based on coordinate mapping and semantic complementarity relationships based on feature similarity at different scales, so as to form cross-scale topology; The intra-scale topology and cross-scale topology are fused to form a spatial topology structure.

5. A bone marrow pathology image analysis system based on multi-view clustering according to claim 4, characterized in that, The hypergraph construction and feature aggregation module further includes a feature fusion unit, which is used for: For each constructed scale-in-the-scale topology, its corresponding scale image features are used as the first node features. The first graph attention network is used to fuse the first node features and the adjacency matrix of the scale-in-the-scale topology to obtain the node aggregation features of that scale. The node aggregation features obtained at each scale are spliced ​​together to form a second node feature. The second node feature is then fused with the adjacency matrix of the cross-scale topology through a graph attention network to obtain the enhanced features of all image patches. Enhancement features of all the image patches are performed Kmeans Semantic clustering is performed, and attention weights are calculated based on the affiliation of each image patch with each semantic category. The enhanced features of all image patches are weighted and summed according to their corresponding attention weights to obtain the image-level global features.

6. A bone marrow pathology image analysis system based on multi-view clustering according to claim 5, characterized in that, The formula for calculating the optimal pseudo-label by the pseudo-label optimization module is as follows: in, Consensus prototype matrix; Image-level global features; Textual semantic features; and These are the optimal pseudo-labels for image-level global features and text semantic features, respectively.

7. A bone marrow pathology image analysis system based on multi-view clustering according to claim 6, characterized in that, The knowledge distillation loss function in the cross-modal feature alignment module is the symmetric cross-entropy loss function: in, and These are the first soft allocation probability and the second soft allocation probability, respectively.

8. A bone marrow pathology image analysis system based on multi-view clustering according to claim 7, characterized in that, The cross-modal feature alignment module further includes a regularization unit, which is used to introduce an entropy regularization term: Total loss function: .

9. A bone marrow pathology image analysis system based on multi-view clustering according to claim 8, characterized in that, The classification result generation module is used for: Obtain the optimized first soft assignment probability and the second soft assignment probability, and calculate their average probability; The category with the highest average probability is selected as the classification result output.

Images