A method and system for generating a whole slide pathology image summary

Abstract

The application discloses a kind of generation methods and systems of whole slice pathological image abstracts.The generation method includes: obtaining the candidate image block set of whole slice pathological image;Random masking, encoding and decoding reconstruction are carried out to candidate image block set, and reconstruction model is obtained;High-level slice semantic reconstruction is carried out to candidate image block set, and unified global semantic vector is fused and constructed;The classification label representation vector of reconstruction model is approximated to global semantic vector using knowledge distillation mechanism constraint;In the reasoning stage, the semantic similarity between the classification label representation of current reconstruction model and global semantic vector is calculated for each random masking reconstruction;The unmasked image block set corresponding to the highest semantic similarity and the corresponding spatial coordinates are taken as abstract image block set.The application can still maintain the expression ability to the main pathological semantic information of original WSI under the premise of only retaining a small amount of image blocks, and realizes efficient image block level compression.

Description

Technical Field

[0001] This invention belongs to the field of computer pathology technology, specifically relating to a method and system for generating whole-section pathological image summaries. Background Technology

[0002] With the development of digital pathology scanning and storage technologies, a large number of slides are being produced as whole-slide images (WSI). WSI features extremely high resolution and a field of view covering complete tissue sections, providing a rich source of information for tasks such as tumor subtyping, biomarker prediction, prognostic assessment, and lesion detection. However, WSI often contains hundreds of millions or even billions of pixels, and performing end-to-end modeling directly based on the entire image would result in extremely high computational and storage costs, making it difficult to meet efficiency and resource constraints in practical applications.

[0003] To reduce computational complexity, existing techniques generally employ a process of dividing the WSI (Width Instantaneous Image Sequence) into several small image patches, and then performing feature extraction and analysis at the patch level. For example, one type of multi-instance learning method treats the WSI as a bag composed of a large number of image patches, extracts features through an image patch encoding network, and then uses attention aggregation or pooling operations to obtain a WSI-level representation, which is then trained based on the bag-level labels. These methods alleviate the difficulty of pixel-level modeling to some extent and improve the model's performance in tasks such as classification.

[0004] However, given a fixed granularity of image patch segmentation, a typical WSI is often divided into thousands or even tens of thousands of image patches, many of which contain only background, normal tissue, or repetitive regions, resulting in significant redundancy. To control computational costs, existing methods typically employ random sampling, threshold-based prior cropping, or simple region selection strategies to select a subset of image patches for subsequent modeling. Such heuristic-based image patch selection methods fail to fully utilize the semantic differences between image patches and struggle to guarantee that the retained set of image patches can completely represent the key pathological information of the WSI.

[0005] On the other hand, in recent years, some methods have attempted to extract "important regions" from WSI-level models. For example, they estimate the importance of each image patch based on attention weights or saliency maps, and generate heatmaps or visualizations to help pathologists understand the model's decision-making process. However, these methods often rely on specific network structures or training strategies, and the resulting attention distributions and saliency maps are easily affected by training noise, model initialization, and other factors, resulting in insufficient robustness and stability. Furthermore, they typically only perform coarse-grained weight allocation in the high-level semantic space and cannot explicitly model redundancy from the semantics of low-level image patches.

[0006] Existing technologies have attempted to learn general visual representations at the image level through self-supervised learning or masking reconstruction for downstream task transfer. However, in WSI scenarios, a unified and systematic solution is still lacking for how to characterize the redundancy and representativeness of image patches at the patch level using reconstruction errors and effectively combine them with WSI-level task semantics. Existing methods typically only focus on local reconstruction or local contrast, lacking direct constraints on WSI-level multi-task discriminative performance, making it difficult to guarantee the discriminative ability of the selected image patch set on different downstream tasks (such as cancer subtyping, biomarker prediction, metastatic lesion identification, etc.).

[0007] Furthermore, in practical applications, pathological analysis models often involve multiple base models with different structures and training strategies, such as WSI-level classification networks trained for different tasks or datasets. Existing image patch selection or visualization methods typically rely solely on the output of a single base model, failing to adequately utilize the complementary semantic information inherent in different models. This makes it difficult to construct a unified and stable global semantic representation, thus limiting the generalization ability and interpretive value of the summarization results across tasks and models.

[0008] Therefore, there is an urgent need for a WSI summary generation technology that can simultaneously consider low-level image patch semantic reconstruction and high-level slice semantic alignment, so as to improve the performance and visualization analysis efficiency of pathological AI models while significantly reducing computational and storage overhead. Summary of the Invention

[0009] The main objective of this invention is to provide a method and system for generating whole-section pathological image summaries to overcome the shortcomings of the prior art.

[0010] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: An embodiment of the present invention provides a method for generating a whole-section pathological image summary, comprising: Obtain a candidate image patch set for whole-section pathological images, wherein the candidate image patch set is obtained based on representative image patches of whole-section pathological images; Random masking, encoding and decoding reconstruction are performed on the candidate image patch set. The reconstruction error is calculated and the image patch redundancy is evaluated to realize low-level image patch semantic reconstruction and obtain the reconstruction model. High-level slice semantic reconstruction is performed on the candidate image patch set. Classification label features are extracted from multiple base models and fused to construct a unified global semantic vector. A knowledge distillation mechanism is used to constrain the classification label representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic alignment. During the inference phase, multiple random masking and reconstruction operations are performed on the candidate image patch set. For each random masking and reconstruction, the semantic similarity between the classification label representation of the current reconstruction model and the global semantic vector is calculated. The set of unmasked image patches corresponding to the highest semantic similarity and the corresponding spatial coordinates are used as the summary image patch set.

[0011] In a preferred embodiment, the step of performing random masking, encoding, and decoding reconstruction on the candidate image patch set, calculating the reconstruction error and evaluating the image patch redundancy, realizing low-level image patch semantic reconstruction and obtaining a reconstruction model includes: A random masking strategy is adopted on the candidate image block set to divide the candidate image blocks in the candidate image block set into an unmasked image block subset and a masked image block subset; The feature representation of the unmasked image block in the subset of unmasked image blocks and its two-dimensional spatial location code are input into the encoder to obtain the context semantic representation; the context semantic representation, along with the masking mark used to represent the masked position and the consistent spatial location code, are input into the decoder to reconstruct the feature representation of the masked image block to obtain the reconstructed features of the masked image block. Based on the reconstruction error between the reconstruction features of the masked image patch and the features of its original image patch, the reconstruction error value of each candidate image patch is obtained, which is used as a measure of the information content or redundancy of the image patch.

[0012] In a preferred embodiment, the method for obtaining the candidate image patch set includes: A whole-section pathological image is acquired and divided into image blocks to generate an image block set including multiple image blocks; the image block division includes: dividing the whole-section pathological image into a regular grid or sampling with a sliding window to generate multiple image blocks; The image blocks in the image block set are input into a pre-trained image block feature encoding network to obtain an image block feature vector set; Cluster analysis is performed on the image patch feature vector set to obtain multiple image patch feature clusters. At least one representative image patch is selected from each image patch feature cluster to construct a candidate image patch set.

[0013] In a preferred embodiment, the step of inputting image patches from the image patch set into a pre-trained image patch feature encoding network to obtain an image patch feature vector set includes: The image patch set is input into a pre-trained image patch feature encoding network, which performs feature extraction and normalization on the image patches and outputs image patch feature vectors to obtain an image patch feature vector set. The two-dimensional coordinates of the image patch in the whole-slice pathological image are encoded and superimposed or spliced ​​with the image patch feature vector to explicitly introduce spatial location information.

[0014] In a preferred embodiment, the K-means clustering algorithm is used to divide all image patch features into K clusters, and within each cluster, a number of image patches are selected as representative image patches according to their distance from the cluster center in ascending order.

[0015] An embodiment of the present invention provides a system for generating whole-section pathological image summaries, comprising: The acquisition module is used to acquire a set of candidate image blocks for whole-section pathological images, wherein the set of candidate image blocks is obtained based on representative image blocks of whole-section pathological images; The low-level image patch semantic reconstruction module is used to perform random masking, encoding and decoding reconstruction on the candidate image patch set, calculate the reconstruction error and evaluate the image patch redundancy, realize low-level image patch semantic reconstruction and obtain the reconstruction model. The high-level slice semantic reconstruction module is used to perform high-level slice semantic reconstruction on the candidate image patch set, extract classification label features from multiple base models, and fuse them to construct a unified global semantic vector; a knowledge distillation mechanism is used to constrain the classification label representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic alignment; The summary generation module is used to perform multiple random masking and reconstruction on the candidate image patch set during the inference phase; for each random masking and reconstruction, the semantic similarity between the classification label representation of the current reconstruction model and the global semantic vector is calculated; and the set of unmasked image patches corresponding to the highest semantic similarity and the corresponding spatial coordinates are used as the summary image patch set.

[0016] Compared with existing technologies, the advantages of this invention are as follows: As a WSI summary generation paradigm, it combines image patch-level clustering sparse sampling and masking reconstruction, maintaining the ability to express the main pathological semantic information of the original WSI while retaining only a very small number of image patches, thus achieving efficient image patch-level compression; it explicitly characterizes image patch redundancy using image patch reconstruction error, which, compared with image patch selection methods based on heuristic rules or simple attention weights, more systematically measures the importance of image patches, and can balance the integrity and representativeness of pathological information while compressing the number of image patches; it introduces knowledge distillation of multiple WSI-level basic models, fusing the complementary global semantic information contained in multiple models into a unified global semantic vector, and performing CLS representation of the summary model... Alignment constraints are implemented to explicitly embed slice-level task semantics during image patch selection, improving the discriminative performance of the summary in various downstream tasks. During the training phase, image patch-level reconstruction loss and slice-level distillation loss are jointly optimized, while during the inference phase, multiple random masking and semantic similarity selection strategies are used to ensure that the final summary image patch set is semantically as close as possible to the global representation of the original WSI, exhibiting good task independence and transferability. The generated summary image patch set retains the original pixel content and accurate spatial coordinates, facilitating alignment with pathologists' reading habits and enhancing the visualization and interpretability of the model's decision-making process. At the same time, since only a small number of image patches need to be transmitted and stored, the computational, storage, and communication overhead of cross-institutional data transmission and model inference is significantly reduced. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a method for generating a whole-section pathological image summary according to one embodiment of this application. Figure 2 This is a comparative diagram of three different WSI analysis frameworks provided in one embodiment of this application; Figure 3 This is a schematic diagram of the network structure of the summarization method provided in one embodiment of this application; Figure 4 This is a flowchart illustrating a method for generating whole-slice pathological image summaries based on two-layer semantic reconstruction, as provided in one embodiment of this application. Detailed Implementation

[0019] The invention will be more fully understood through the following detailed description, which should be read in conjunction with the accompanying drawings. Detailed embodiments of the invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely exemplary of the invention, which may be embodied in various forms. Therefore, the specific functional details disclosed herein should not be construed as limiting, but rather as the basis for the claims and as intended to teach those skilled in the art to employ the representative basis of the invention in different ways in any suitable detailed embodiment.

[0020] Example 1 See Figure 1 This embodiment provides a method for generating whole-section pathological image summaries, including: S101, Obtain a candidate image patch set for the whole-slide pathological image; wherein the candidate image patch set is obtained based on representative image patches of the whole-slide pathological image, and the reconstruction model is obtained by performing random masking reconstruction training on the candidate image patch set, and is used to output the slice-level classification label representation; the whole-slide pathological image is also known as WholeSlideImage, or WSI for short; The method for obtaining the candidate image patch set includes: S201, acquire a whole-slice pathological image and divide it into image blocks to generate an image block set including multiple image blocks; dividing the whole-slice pathological image into image blocks includes: dividing the whole-slice pathological image into a regular grid or sampling with a sliding window to generate multiple image blocks; In practical applications, whole-slice pathological images (WSI) can be acquired, and preprocessing such as color normalization and background removal can be performed on the WSI; the resolution can be... WSI according to preset image patch size With step size By performing regular grid division or sliding window sampling, a set of image patches is obtained. ,in The total number of image patches is expressed by the formula: For each image patch Normalization or color standardization preprocessing is performed and then input into the image patch feature encoding network to obtain feature vectors. Among them, through Normalize the feature vector magnitude to facilitate subsequent clustering and similarity calculations. A feature encoding network for image patches.

[0021] In a preferred embodiment, the image patch size can be set to The pixel size is set to a step size that is the same as or a fraction of the image patch size, in order to balance coverage and sampling density.

[0022] S202, Input the image blocks in the image block set into the pre-trained image block feature encoding network to obtain the image block feature vector set; Step S202 may specifically include: inputting the image patch set into a pre-trained image patch feature encoding network, performing feature extraction and normalization on the image patches, outputting image patch feature vectors, and obtaining an image patch feature vector set; The two-dimensional coordinates of the image patch in the whole-slice pathological image are encoded and superimposed or spliced ​​with the image patch feature vector to explicitly introduce spatial location information.

[0023] In practical applications, the set of image patches (i.e., the set of image patches) can be input into a pre-trained image patch feature encoding network to obtain the corresponding set of image patch feature embeddings. ,in For image patch-level base models pre-trained on large-scale pathological images or image-text datasets, The feature dimension is defined as follows. In one embodiment, the image patch feature encoding network can employ a vision-language foundational model based on the ViT-Large structure, where each image patch is mapped to a 768-dimensional feature vector, and optionally... Normalization is performed to facilitate subsequent clustering and similarity calculations. Let represent a d-dimensional real vector space.

[0024] S203, perform cluster analysis on the image patch feature vector set to obtain multiple image patch feature clusters, select at least one representative image patch from each image patch feature cluster, and construct a candidate image patch set.

[0025] In step S203, the K-means clustering algorithm is used to divide all image patch features into K clusters, and within each cluster, a number of image patches are selected as representative image patches according to their distance from the cluster center from small to large, in order to ensure that the candidate image patch set covers a variety of organizational forms and semantic patterns.

[0026] In practical applications, image patch features can be embedded into a set. Clustering is performed on the feature space (set of image patch feature vectors) to divide semantically similar image patches into multiple clusters. For example, the K-means clustering algorithm can be used to cluster the feature space (set of image patch features). ) divided into Cluster Cluster centers are obtained by minimizing the following objective function. : Within each cluster, select the one that makes The smallest at least one image patch is used as the representative image patch of the cluster. Representative image patches from all clusters are combined to form a candidate image patch set, thereby compressing the number of image patches while covering global semantic diversity. In other words, the representative image patch sets of each cluster are merged to form the candidate image patch set. Its size is denoted as ,satisfy .

[0027] K can be set to 50, and the number of candidate image blocks M in the candidate image block set can be set to 500, so as to reduce the number of redundant image blocks and control the subsequent computational overhead while ensuring the semantic diversity coverage of WSI.

[0028] S102, Perform random masking, encoding and decoding reconstruction on the candidate image block set, calculate the reconstruction error and evaluate the image block redundancy, realize low-level image block semantic reconstruction and obtain the reconstruction model; Specifically, it includes: S301, a random masking strategy is adopted on the candidate image block set to divide the candidate image blocks in the candidate image block set into an unmasked image block subset and a masked image block subset; S302, The feature representation of the unmasked image block in the subset of unmasked image blocks and its two-dimensional spatial position code are input into the encoder to obtain the context semantic representation; The context semantic representation, along with the masking mark used to represent the masked position and the consistent spatial position code, are input into the decoder to reconstruct the feature representation of the masked image block to obtain the reconstructed features of the masked image block. S303. Based on the reconstruction error between the reconstruction features of the masked image patch and its original image patch features, the reconstruction error value of each candidate image patch is obtained. This value is used as a measure of the information content or redundancy of the image patch. Image patches with smaller reconstruction errors and higher semantic redundancy are suppressed, thereby achieving low-level semantic reconstruction and redundancy compression at the image patch level.

[0029] Then, the features of the unmasked image patch and its two-dimensional spatial coordinates in the original WSI are combined using positional encoding to obtain a sequence input encoder, which yields the contextual semantic representation of the unmasked image patch. This contextual semantic representation, along with the masking marker and the current positional encoding, is then input into the decoder to reconstruct the features of the masked image patch, resulting in reconstructed features. .

[0030] Using the masked image patches as the statistical objects, we define the image patch-level reconstruction loss: in, The feature distance metric function can be L2 distance, L1 distance, cosine distance, or a weighted combination thereof. The reconstruction error for each image patch. It can be used as a measure of the information content or redundancy of the image block. The smaller the reconstruction error, the easier it is to infer the image block from other image blocks and the higher the semantic redundancy.

[0031] Among them, the masking ratio The value range is, for example, 50% to 95%, preferably 87.5%, that is, only about 12.5% ​​of the candidate image patches are retained as visible image patches, so as to force the model to learn the contextual relationship between image patches under strong compression conditions, thereby more significantly distinguishing redundant image patches from key information image patches.

[0032] The specific process includes, for example, superimposing the feature vector of the unmasked image patch with its two-dimensional positional encoding to form an input sequence that is fed into the encoder, and then reconstructing the features of the masked image patch on the decoder side by combining the masking markers to obtain the reconstructed features. ; With the set of masked image patches For the statistical object, the reconstruction loss is defined as... and the reconstruction error of a single image patch As a measure of the redundancy of the image patch, image patches with high redundancy are suppressed, thereby completing the semantic reconstruction of the lower-level image patches.

[0033] S103, perform high-level slice semantic reconstruction on the candidate image patch set, extract classification label features from multiple supervised training base models, and fuse them to construct a unified global semantic vector; use a knowledge distillation mechanism to constrain the classification label representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic alignment; This can be understood as follows: for the same WSI image, extract its WSI-level representation or CLS label from multiple pre-trained WSI-level pathology baseline models, and fuse the resulting vectors according to a preset method (such as concatenation, weighted summation, and dimensionality reduction transformation) to construct a unified global semantic vector. This is used to characterize the overall semantic information of the WSI at the slice level.

[0034] In practical applications, the aforementioned basic models can include pathological models trained based on contrastive learning, masked autoencoders, and knowledge distillation, such as Gigapath, TITAN, PRISM, and CHIEF. The CLS vectors of the same WSI are extracted and concatenated, then projected onto a low-dimensional space using principal component analysis (PCA) or a fully connected layer to obtain a compact global semantic vector. .

[0035] The WSI level CLS output of the above encoder-decoder structure is denoted as . Constructing knowledge distillation loss: The high-level semantic representation used to constrain the summary model in the feature space approximates the global semantic vector obtained by fusing multiple fundamental models.

[0036] The joint loss and model training process includes: The total training loss is obtained by weighted summing of the image block-level reconstruction loss and the slice-level distillation loss: in, , where are weight coefficients used to adjust the importance of low-level reconstruction accuracy and high-level semantic alignment. During the training phase, steps S1 to S5 are repeated on the input WSI, updating the encoder, decoder, and related projection layer parameters through backpropagation and gradient descent, so that the model can maintain its ability to characterize the global semantics of WSI as much as possible while compressing the image patch set.

[0037] S104, During the inference phase, multiple random masking and reconstructions are performed on the candidate image patch set; for each random masking and reconstruction, the semantic similarity between the classification label representation of the current reconstruction model and the global semantic vector is calculated; the set of unmasked image patches corresponding to the highest semantic similarity and the corresponding spatial coordinates are taken as the summary image patch set.

[0038] This can be understood as follows: During the inference phase, image patch construction, image patch feature encoding, and candidate image patch set generation are first completed in the same manner as the training phase; then, with the pre-trained encoder-decoder structure and basic model fixed, multiple random masking and reconstruction operations are performed. Each time, a set of masking configurations is randomly generated on the candidate image patch set, resulting in the corresponding set of unmasked image patches. and the set of masked image patches The CLS representation under this configuration is obtained through encoder-decoder. ; Calculate its relationship with the global semantic vector Similarity, for example traverse the preset number of times After (e.g., 32 or 64 times), select the index with the highest similarity. The set of unmasked image patches under this masking configuration The corresponding spatial coordinates are used as the final set of WSI summary image patches. The number of these summary image patches is much smaller than the original number of image patches, but their semantic representation is as close as possible to the global semantic vector. This allows the system to maintain high discrimination performance across a variety of downstream tasks.

[0039] In practical applications, the set of abstract image patches can be used as a "package" and input into a multi-instance learning classifier (such as ABMIL, TransMIL, MambaMIL, etc.) to complete tasks such as cancer typing, biomarker prediction, or metastatic lesion typing, so as to further verify the task utility of the abstract results.

[0040] As an example, a method for generating a summary of a whole-section pathological image is provided, including the following steps: (1) Image patch sampling and feature extraction steps Obtain whole-slide pathological images (WSI), and perform regular grid division or sliding window sampling on the WSI to obtain multiple image patches. The image patches are input into a pre-trained image patch feature encoding network to obtain the corresponding set of image patch feature vectors. ,in For image patch feature coding networks, For feature dimensions; (2) Steps for constructing the candidate image patch set Based on the image patch feature vector set Perform clustering operations to obtain multiple image patch feature clusters, and select at least one representative image patch from each image patch feature cluster to construct a candidate image patch set; (3) Semantic reconstruction steps of low-level image patches A random masking operation is performed on the candidate image block set to divide the candidate image blocks into an unmasked image block set and a masked image block set. The feature vector of the unmasked image patch and its position encoding are input into the encoder together. The masking label is then used by the decoder to reconstruct the features of the masked image patch, resulting in the reconstructed features of the masked image patch. Based on this, the reconstruction error of each masked image patch is calculated to characterize the redundancy of the image patch, realize the semantic reconstruction of the low-level image patch and obtain the reconstruction model; (4) High-level slice semantic reconstruction steps WSI-level classification label (CLS) features are extracted from multiple base models trained with WSI-level supervision to construct a unified global semantic vector; The knowledge distillation mechanism is used to constrain the CLS representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic reconstruction. (5) Steps for generating the summary image patch set During the inference phase, multiple random masking and reconstruction operations are performed on the candidate image patch set; For each random masking reconstruction, calculate the semantic similarity between the current reconstruction model CLS representation and the global semantic vector; The image patch with the highest CLS similarity is selected from multiple random masking reconstruction results, and its corresponding unmasked image patch and its spatial coordinates constitute the summary image patch set of the WSI.

[0041] The process of image patch sampling for WSI in S1 includes: With resolution WSI by image patch size With step size Perform regular sliding window segmentation to obtain a set of image patches. ,in For each image patch Normalization or color standardization preprocessing is performed and then input into the image patch feature encoding network to obtain feature vectors. in A network for encoding image patch features. Through... Normalize the feature vector magnitude to facilitate subsequent clustering and similarity calculations.

[0042] The process of clustering image patch feature vectors in S2 uses the K-means clustering algorithm, specifically including: Image patch feature set Divided into Cluster Cluster centers are obtained by minimizing the following objective function. : In each cluster In the middle, select to make The smallest image patch is used as the representative image patch of the cluster. Representative image patches of all clusters are combined into a candidate image patch set, thereby compressing the number of image patches while covering global semantic diversity.

[0043] The semantic reconstruction process of low-level image blocks in S3 includes: Randomly generate masking markers for each image patch in the candidate image patch set. ,in This indicates that the image patch is masked. This indicates that the object was not concealed and meets the following conditions. To preset the masking ratio, The size of the candidate image patch set; The feature vectors of the unmasked image patches are superimposed with their two-dimensional positional codes to form an input sequence, which is then fed into the encoder. At the decoder, the masking markers are combined to reconstruct the features of the masked image patches, resulting in the reconstructed features. ; With the set of masked image patches For the statistical object, the reconstruction loss is defined as... and the reconstruction error of a single image patch As a measure of the redundancy of the image patch, image patches with high redundancy are suppressed, thereby completing the semantic reconstruction of the lower-level image patches.

[0044] The process of high-level slice semantic reconstruction and summary image patch set generation in S4 and S5 includes: Select A base model trained under WSI supervision is used as the teacher model to extract CLS features from the same WSI. And construct a unified global semantic vector Let the CLS representation of the reconstructed model be... Knowledge distillation loss High-level semantic representation of the constraint reconstruction model; Define the total loss as in To balance the coefficients, we optimize both low-level reconstruction and high-level semantic alignment by minimizing the total loss. During the inference phase, the candidate image patch set is executed. After a random masking reconstruction, we obtain CLS represents And calculate the similarity for each iteration. Select the index with the highest similarity The first The set of unmasked image patches and their spatial coordinates corresponding to the random masking are used as the final set of WSI summary image patches.

[0045] This embodiment provides a method for generating whole-slice pathological image summaries. Addressing the problems of existing whole-slice pathological image summaries (WSIs) such as extremely high resolution, high information redundancy, huge computational and storage costs due to direct modeling based on the entire image, and the difficulty in balancing summary compactness and downstream task discrimination performance with existing image patch selection strategies that rely heavily on heuristic rules, the aim is to maintain the original WSI's discrimination capability in various downstream tasks while retaining only a very small number of representative image patches, significantly reducing computational and storage costs, and improving the model's interpretability and visualization efficiency.

[0046] Specifically, compared to related technologies, image patch selection and aggregation techniques for WSI suffer from three main drawbacks: First, they lack a mechanism to systematically characterize image patch redundancy from the perspective of image patch reconstruction error, making it difficult to maintain the integrity of pathological information while compressing the image patch set. Second, image patch-level selection and WSI-level task semantics are often separated, lacking coordinated optimization, making it impossible to stably maintain the discrimination performance of multiple downstream tasks while keeping a small number of image patches. Third, the fusion and alignment of global semantic information from multiple models are insufficient, affecting the robustness and interpretability of downstream task results.

[0047] The technical solution provided in this embodiment can serve as a WSI summary generation paradigm. By combining image patch-level clustering sparse sampling and masking reconstruction, it can maintain the expressive power of the main pathological semantic information of the original WSI while retaining only a very small number of image patches, achieving efficient image patch-level compression. It explicitly characterizes image patch redundancy using image patch reconstruction errors, which, compared to image patch selection methods based on heuristic rules or simple attention weights, more systematically measures the importance of image patches, and can balance the integrity and representativeness of pathological information while compressing the number of image patches. Furthermore, it introduces knowledge distillation from multiple WSI-level base models, fusing the complementary global semantic information contained in multiple models into a unified global semantic vector, and aligning the CLS representation of the summary model. This approach explicitly embeds slice-level task semantics into the image patch selection process, improving the discriminative performance of the summary across various downstream tasks. During training, image patch-level reconstruction loss and slice-level distillation loss are jointly optimized. In the inference phase, multiple random masking and semantic similarity selection strategies are employed to ensure that the final summary image patch set semantically approximates the global representation of the original WSI, exhibiting good task independence and transferability. The generated summary image patch set retains the original pixel content and precise spatial coordinates, facilitating alignment with pathologists' reading habits and enhancing the visualization and interpretability of the model's decision-making process. Furthermore, since only a small number of image patches need to be transmitted and stored, the computational, storage, and communication overhead for cross-institutional data transmission and model inference is significantly reduced.

[0048] Example 2 This embodiment provides a system for generating whole-section pathological image summaries, including: The acquisition module is used to acquire a set of candidate image patches for whole-section pathological images; wherein the set of candidate image patches is obtained based on representative image patches of whole-section pathological images, and the reconstruction model is obtained by performing random masking reconstruction training on the set of candidate image patches, and is used to output slice-level classification label representations; The low-level image patch semantic reconstruction module is used to perform random masking, encoding and decoding reconstruction on the candidate image patch set, calculate the reconstruction error and evaluate the image patch redundancy, realize low-level image patch semantic reconstruction and obtain the reconstruction model. The high-level slice semantic reconstruction module is used to perform high-level slice semantic reconstruction on the candidate image patch set. It extracts classification label features from multiple supervised training base models and fuses them to construct a unified global semantic vector. A knowledge distillation mechanism is used to constrain the classification label representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic alignment. The summary generation module is used to perform multiple random masking and reconstruction on the candidate image patch set during the inference phase; for each random masking and reconstruction, the semantic similarity between the classification label representation of the current reconstruction model and the global semantic vector is calculated; and the set of unmasked image patches corresponding to the highest semantic similarity and the corresponding spatial coordinates are used as the summary image patch set.

[0049] In one embodiment, the method for obtaining the candidate image patch set includes: A whole-section pathological image is acquired and divided into image blocks to generate an image block set including multiple image blocks; the image block division includes: dividing the whole-section pathological image into a regular grid or sampling with a sliding window to generate multiple image blocks; The image blocks in the image block set are input into a pre-trained image block feature encoding network to obtain an image block feature vector set; Cluster analysis is performed on the image patch feature vector set to obtain multiple image patch feature clusters. At least one representative image patch is selected from each image patch feature cluster to construct a candidate image patch set.

[0050] In one embodiment, the low-level image patch semantic reconstruction module includes: A subset partitioning unit is used to divide the candidate image blocks in the candidate image block set into an unmasked image block subset and a masked image block subset by adopting a random masking strategy on the candidate image block set; The semantic representation acquisition unit is used to input the feature representation of the unmasked image block in the subset of unmasked image blocks and its two-dimensional spatial position encoding into the encoder to obtain the contextual semantic representation; The reconstructed feature acquisition unit is used to input the context semantic representation together with the masking mark used to represent the masked position and the consistent spatial location code into the decoder to reconstruct the feature representation of the masked image block and obtain the reconstructed features of the masked image block. The error value acquisition unit is used to obtain the reconstruction error value of each candidate image block based on the reconstruction error between the reconstruction features of the masked image block and its original image block features. This error value is used as a measure of the information content or redundancy of the image block. Image blocks with small reconstruction errors and high semantic redundancy are suppressed, thereby realizing low-level semantic reconstruction and redundancy compression at the image block level.

[0051] In one embodiment, inputting image patches from the image patch set into a pre-trained image patch feature encoding network to obtain an image patch feature vector set includes: The image patch set is input into a pre-trained image patch feature encoding network, which performs feature extraction and normalization on the image patches and outputs image patch feature vectors to obtain an image patch feature vector set. The two-dimensional coordinates of the image patch in the whole-slice pathological image are encoded and superimposed or spliced ​​with the image patch feature vector to explicitly introduce spatial location information.

[0052] In one embodiment, the clustering analysis of the image patch feature vector set includes: The K-means clustering algorithm is used to divide all image patch features into K clusters. Within each cluster, a number of image patches are selected as representative image patches according to their distance from the cluster center in ascending order. This ensures that the candidate image patch set covers a variety of organizational forms and semantic patterns.

[0053] As an example, a system for generating whole-slice pathological image summaries is provided, for generating whole-slice pathological image summaries based on two-layer semantic reconstruction, including: The acquisition module includes: an input and preprocessing submodule, used to acquire the WSI data to be processed and perform preprocessing such as color normalization, background region removal, and resolution unification; an image patch construction submodule, used to divide the WSI according to a preset image patch size and step size or to perform sliding sampling to obtain an image patch set and record the spatial coordinates of each image patch in the whole-slice pathological image; an image patch feature encoding submodule, used to input each image patch into a pre-trained image patch feature encoding network to obtain the corresponding image patch feature embedding set, and optionally to perform feature normalization processing; and a clustering and candidate generation submodule, used to perform clustering analysis based on the image patch feature embedding and select representative image patches from each cluster to construct a candidate image patch set, so as to ensure semantic diversity while compressing the number of image patches. The low-level semantic reconstruction module (i.e., the low-level image patch semantic reconstruction module) is used to reconstruct the features of the masked image patches by adopting a random masking strategy on the candidate image patch set through an encoder-decoder structure, calculate the image patch-level reconstruction loss and the reconstruction error of each image patch, and realize low-level semantic reconstruction and redundancy estimation at the image patch level. The high-level semantic construction and distillation module (i.e., the high-level slice semantic reconstruction module) is used to extract WSI-level CLS representations from multiple WSI-level base models, fuse them to obtain a unified global semantic vector, and apply knowledge distillation constraints to the CLS representations output by the reconstruction model as the teacher's goal, thereby completing the slice-level high-level semantic reconstruction. The abstract generation module is used to perform multiple random masking and reconstruction of the candidate image patch set during the inference stage, calculate the similarity between each CLS representation and the global semantic vector, select the unmasked image patch set corresponding to the one with the highest similarity as the final WSI abstract image patch set, and output the image content and spatial coordinates of each image patch. The above modules can be deployed as software functional modules on the same server or cluster, or as hardware modules in a distributed manner, and data interaction can be achieved through bus or network connection. Those skilled in the art can choose according to the specific application scenario, without any restrictions.

[0054] Furthermore, the image patch feature encoding module includes: A pre-trained visual feature extraction network is used to perform feature extraction and normalization on image patches and output image patch feature vectors; The location coding unit is used to encode the two-dimensional coordinates of the image patch in WSI and superimpose or concatenate them with the image patch feature vector to explicitly introduce spatial location information.

[0055] Furthermore, the global semantic construction and distillation module includes: The teacher feature aggregation unit is used to perform weighted averaging or concatenation transformation on the CLS features output by multiple teacher models to obtain a unified global semantic vector. ; Student CLS extraction unit, used to extract the CLS representation of the reconstructed model. ; Distillation loss calculation unit, used to calculate total loss. The loss is then fed back into the parameter update process of the reconstruction model to simultaneously optimize the reconstruction accuracy of low-level image patches and the semantic alignment of high-level images.

[0056] like Figure 2 As shown, Figure 2Three different WSI analysis frameworks are illustrated: (a) is a traditional image patch selection-based analysis framework, which selects image patches from WSI using methods such as random sampling, clustering, or information entropy, and then uses them for different tasks; (b) is an analysis framework based on a WSI-level base model, which supports multiple downstream tasks through a single global embedding, but it is difficult to maintain spatial details and interpretability; (c) is the WSI summarization framework proposed in this invention (understood as the framework of the WSISum model), which generates a set of compact summary image patches through the WSISum model of this invention to approximate the original WSI and support multiple downstream tasks, while maintaining discriminative performance and significantly reducing computational and storage overhead.

[0057] like Figure 3 As shown, Figure 3 This is a schematic diagram of the network structure of the WSI summarization method of this invention. The overall structure mainly consists of an image patch encoding and clustering module, a low-level image patch semantic reconstruction module, and a high-level slice semantic reconstruction module. The left side illustrates the process of uniformly sampling from WSI and extracting image patch embeddings through an image patch coding network, and constructing a candidate image patch set through K-means clustering. The middle part illustrates the random masking performed on the candidate image patch set. The features and position codes of the unmasked image patches are input into the encoder, and the decoder reconstructs the masked image patches. The low-level semantic reconstruction is driven by the image patch-level reconstruction loss. The right side illustrates how CLS vectors extracted from multiple WSI-level base models are concatenated and dimension-reduced to form a unified global semantic vector. The model CLS representation is then reconstructed using knowledge distillation loss constraints to align with this global semantic vector, thereby achieving slice-level high-level semantic reconstruction. like Figure 4 As shown, Figure 4 This is a flowchart of the overall process for generating whole-slice pathological images (WSI) summaries based on two-layer semantic reconstruction proposed in this invention. The process begins with acquiring and preprocessing the WSI. First, the original slices are divided into patches and features are extracted (image patch segmentation and feature extraction). Then, a candidate patch set is constructed through clustering (K-means clustering to construct a candidate image patch set). During the training phase, random masking, patch-level reconstruction, and slice-level semantic distillation based on a multi-foundation model are performed sequentially to achieve joint optimization of low-level fine-grained semantics and high-level global semantics (random masking of candidate image patches yields unmasked / masked image patches; encoder-decoder reconstructs the masked image patches and calculates the image patch-level reconstruction loss). Extracting CLS from multiple WSI base models and fusing them to construct a global semantic vector; calculating distillation loss. Total loss During the inference phase, through multiple random masking and semantic similarity selection mechanisms, a compact summary patch set that can represent the main pathological information of the original WSI is finally generated (the similarity between CLS and g is calculated by randomly masking candidate image patches multiple times; the set of unmasked image patches with the highest similarity is selected as the WSI summary).

[0058] Those skilled in the art will understand that the parameters mentioned in the above embodiments, such as image patch size, number of clusters, number of candidate image patches, masking ratio, type of basic model, and number of random masking times, can be adjusted according to specific datasets and task requirements, and do not constitute a substantial limitation of the present invention.

[0059] Although the invention has been described with reference to illustrative embodiments, those skilled in the art will understand that various other changes, omissions, and / or additions can be made without departing from the spirit and scope of the invention, and that elements of the described embodiments can be substituted with substantially equivalents. Furthermore, many modifications can be made without departing from the scope of the invention to adapt particular situations or materials to the teachings of the invention. Therefore, this document is not intended to limit the invention to the specific embodiments disclosed for carrying out the invention, but rather to include all embodiments falling within the scope of the appended claims.

Claims

1. A method for generating a whole-section pathological image summary, characterized in that, include: Obtain a candidate image patch set for whole-section pathological images, wherein the candidate image patch set is obtained based on representative image patches of whole-section pathological images; Random masking, encoding and decoding reconstruction are performed on the candidate image patch set. The reconstruction error is calculated and the image patch redundancy is evaluated to realize low-level image patch semantic reconstruction and obtain the reconstruction model. High-level slice semantic reconstruction is performed on the candidate image patch set. Classification label features are extracted from multiple base models and fused to construct a unified global semantic vector. A knowledge distillation mechanism is used to constrain the classification label representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic alignment. During the inference phase, multiple random masking and reconstruction operations are performed on the candidate image patch set; For each random masking reconstruction, the semantic similarity between the classification label representation of the current reconstruction model and the global semantic vector is calculated; the set of unmasked image patches corresponding to the highest semantic similarity and the corresponding spatial coordinates are used as the set of summary image patches.

2. The generation method according to claim 1, characterized in that, The process of performing random masking, encoding, and decoding reconstruction on the candidate image patch set, calculating the reconstruction error, evaluating image patch redundancy, achieving low-level image patch semantic reconstruction, and obtaining a reconstruction model includes: A random masking strategy is adopted on the candidate image block set to divide the candidate image blocks in the candidate image block set into an unmasked image block subset and a masked image block subset; The feature representation of the unmasked image block in the subset of unmasked image blocks and its two-dimensional spatial location code are input into the encoder to obtain the context semantic representation; the context semantic representation, along with the masking mark used to represent the masked position and the consistent spatial location code, are input into the decoder to reconstruct the feature representation of the masked image block to obtain the reconstructed features of the masked image block. Based on the reconstruction error between the reconstruction features of the masked image patch and the features of its original image patch, the reconstruction error value of each candidate image patch is obtained, which is used as a measure of the information content or redundancy of the image patch.

3. The generation method according to claim 2, characterized in that, The methods for obtaining the candidate image patch set include: A whole-section pathological image is acquired and divided into image blocks to generate an image block set including multiple image blocks; the image block division includes: dividing the whole-section pathological image into a regular grid or sampling with a sliding window to generate multiple image blocks; The image blocks in the image block set are input into a pre-trained image block feature encoding network to obtain an image block feature vector set; Cluster analysis is performed on the image patch feature vector set to obtain multiple image patch feature clusters. At least one representative image patch is selected from each image patch feature cluster to construct a candidate image patch set.

4. The generation method according to claim 3, characterized in that, The step of inputting image patches from the image patch set into a pre-trained image patch feature encoding network to obtain an image patch feature vector set includes: The image patch set is input into a pre-trained image patch feature encoding network, which performs feature extraction and normalization on the image patches and outputs image patch feature vectors to obtain an image patch feature vector set. The two-dimensional coordinates of the image patch in the whole-slice pathological image are encoded and superimposed or spliced ​​with the image patch feature vector to explicitly introduce spatial location information.

5. The generation method according to claim 3, characterized in that, The clustering analysis of the image patch feature vector set includes: The K-means clustering algorithm is used to divide all image patch features into K clusters, and within each cluster, a number of image patches are selected as representative image patches according to their distance from the cluster center in ascending order.

6. A system for generating whole-section pathological image summaries, characterized in that, include: The acquisition module is used to acquire a set of candidate image blocks for whole-section pathological images, wherein the set of candidate image blocks is obtained based on representative image blocks of whole-section pathological images; The low-level image patch semantic reconstruction module is used to perform random masking, encoding and decoding reconstruction on the candidate image patch set, calculate the reconstruction error and evaluate the image patch redundancy, realize low-level image patch semantic reconstruction and obtain the reconstruction model. The high-level slice semantic reconstruction module is used to perform high-level slice semantic reconstruction on the candidate image patch set, extract classification label features from multiple base models, and fuse them to construct a unified global semantic vector; a knowledge distillation mechanism is used to constrain the classification label representation vector of the reconstruction model to approximate the global semantic vector, thereby achieving high-level slice semantic alignment; The summary generation module is used to perform multiple random masking and reconstruction operations on the candidate image patch set during the inference phase. For each random masking reconstruction, the semantic similarity between the classification label representation of the current reconstruction model and the global semantic vector is calculated; the set of unmasked image patches corresponding to the highest semantic similarity and the corresponding spatial coordinates are used as the set of summary image patches.

7. The generation system according to claim 6, characterized in that, The low-level image patch semantic reconstruction module includes: A subset partitioning unit is used to divide the candidate image blocks in the candidate image block set into an unmasked image block subset and a masked image block subset by adopting a random masking strategy on the candidate image block set; The semantic representation acquisition unit is used to input the feature representation of the unmasked image block in the subset of unmasked image blocks and its two-dimensional spatial position encoding into the encoder to obtain the contextual semantic representation; The reconstructed feature acquisition unit is used to input the context semantic representation together with the masking mark used to represent the masked position and the consistent spatial location code into the decoder to reconstruct the feature representation of the masked image block and obtain the reconstructed features of the masked image block. The error value acquisition unit is used to obtain the reconstruction error value of each candidate image block based on the reconstruction error between the reconstruction features of the masked image block and its original image block features. This error value is used as a measure of the information content or redundancy of the image block. Image blocks with small reconstruction errors and high semantic redundancy are suppressed, thereby realizing low-level semantic reconstruction and redundancy compression at the image block level.

8. The generation system according to claim 6, characterized in that, The methods for obtaining the candidate image patch set include: A whole-section pathological image is acquired and divided into image blocks to generate an image block set including multiple image blocks; the image block division includes: dividing the whole-section pathological image into a regular grid or sampling with a sliding window to generate multiple image blocks; The image blocks in the image block set are input into a pre-trained image block feature encoding network to obtain an image block feature vector set; Cluster analysis is performed on the image patch feature vector set to obtain multiple image patch feature clusters. At least one representative image patch is selected from each image patch feature cluster to construct a candidate image patch set.

9. The generation system according to claim 8, characterized in that, The step of inputting image patches from the image patch set into a pre-trained image patch feature encoding network to obtain an image patch feature vector set includes: The image patch set is input into a pre-trained image patch feature encoding network, which performs feature extraction and normalization on the image patches and outputs image patch feature vectors to obtain an image patch feature vector set. The two-dimensional coordinates of the image patch in the whole-slice pathological image are encoded and superimposed or spliced ​​with the image patch feature vector to explicitly introduce spatial location information.

10. The generation system according to claim 8, characterized in that, The clustering analysis of the image patch feature vector set includes: The K-means clustering algorithm is used to divide all image patch features into K clusters, and within each cluster, a number of image patches are selected as representative image patches according to their distance from the cluster center in ascending order.

Images