Classification method and system of whole slide images based on learnable feature merging and topology awareness

Abstract

A classification method and system of whole slide images based on learnable feature merging and topology-awareness are disclosed. The classification method comprises: performing image block segmentation on a whole slide image to obtain a plurality of image blocks, and obtaining an initial feature sequence containing two-dimensional spatial coordinates and image block features; processing the initial feature sequence to obtain an assignment matrix for feature soft assignment, and weighting and aggregating the image block features based on the assignment matrix into a plurality of merged features to obtain a merged feature sequence; calculating a spatial distance matrix according to the two-dimensional spatial coordinates, and mapping the spatial distance matrix into a topology-aware spatial bias matrix using the assignment matrix; performing global attention calculation on the merged feature sequence by taking the topology-aware spatial bias matrix as a negative bias term, so that the attention interaction weight decays with the increase of the spatial distance, and predicting the classification result of the whole slide image. The classification method can reduce the computational overhead while maintaining the boundary information and spatial consistency.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and medical image processing technology, and more specifically, to a whole-slide image (WSI) classification method, classification system, and electronic device based on deep learning and multiple instance learning (MIL). Background Technology

[0002] Digital pathology has revolutionized disease diagnosis through whole-slice images (WSI). However, automated WSI analysis is severely hampered by its massive size (typically greater than 100,000 × 100,000 pixels) and extreme information imbalance. Specifically, typical pathology slides are filled with a large amount of redundant background tissue, while sparse diagnostic clues (such as tumor nests) often occupy less than 5% of the area, making uniform, intensive computational processing prohibitively computationally expensive.

[0003] Modern multi-instance learning architectures utilize advanced sequence modeling methods (such as Transformers) to capture long-range tissue dependencies, but their extremely high computational complexity constitutes a significant bottleneck. To alleviate this computational explosion, existing techniques primarily employ sequence compression and token (also known as feature vector) reduction strategies. For example, token pruning directly discards feature vectors based on importance, but in computational pathology where diagnostic signals are extremely sparse, this irreversible discarding carries the risk of permanently losing crucial microscopic evidence. Similarly, conventional token merging provides a relatively safe, lossless paradigm by aggregating redundant features rather than discarding them; however, general visual merging algorithms heavily rely on heuristic local feature similarity, easily leading to the erroneous fusion of spatially separated but textured tumor foci, thereby disrupting the multifocal distribution required for clinical staging or eroding the delicate tumor-stromal boundary.

[0004] Therefore, there is an urgent need in this field for an image classification technology that can effectively compress processing sequences and reduce computational complexity, while strictly maintaining the multifocal spatial topology and key diagnostic boundaries of pathological images. Summary of the Invention

[0005] To address the problems of excessive computational load in full-slice image processing and the tendency of existing feature compression methods to lead to spatial misalignment and tumor boundary degradation, this disclosure provides a classification method, classification system, and electronic device for full-slice images based on learnable feature merging and topology awareness.

[0006] According to one or more aspects of embodiments of this disclosure, a classification method for a full-slice image based on learnable feature merging and topology awareness is provided. The classification method includes: segmenting the full-slice image into multiple image patches and obtaining an initial feature sequence, the initial feature sequence including two-dimensional spatial coordinates representing the positional relationship between the image patch and the full-slice image and image patch features representing the visual morphology of the image patch; obtaining an allocation matrix for soft feature allocation based on the initial feature sequence, and weighting and aggregating the image patch features into multiple merged features according to the allocation matrix to obtain a merged feature sequence composed of the multiple merged features, the length of the merged feature sequence being less than the length of the initial feature sequence; calculating a spatial distance matrix between the multiple image patches based on the two-dimensional spatial coordinates of each image patch, and mapping the spatial distance matrix to a topology-aware spatial bias matrix reflecting the spatial distance relationship between the multiple merged features using the allocation matrix; and performing global attention calculation on the merged feature sequence using the topology-aware spatial bias matrix as a negative bias term, and predicting the classification result of the full-slice image based on the features output by the global attention calculation.

[0007] In some embodiments, the step of obtaining the initial feature sequence includes: determining the two-dimensional spatial coordinates based on the segmentation position of each image patch in the full slice image; and extracting coded features from each image patch using a pre-trained visual encoder, and calculating the image patch features using a sliding window attention mechanism, such that the image patch features are fused with the local morphological context of adjacent image patches. The calculation of the sliding window attention mechanism includes: applying two-dimensional rotational position encoding to the query features obtained by linear projection of the coded features, calculating attention weights using a Sigmoid activation function, and weighting and aggregating the value features obtained by linear projection of the coded features based on the attention weights to obtain the image patch features.

[0008] In some embodiments, the step of obtaining an allocation matrix for soft feature allocation based on the initial feature sequence and weighted aggregating the image patch features into multiple merged features according to the allocation matrix includes: projecting the image patch features of each image patch in the initial feature sequence into corresponding embedded features through an embedding layer to obtain an embedding feature sequence composed of multiple embedded features; setting multiple learnable query prototypes and obtaining the allocation matrix by calculating the normalized inner product similarity between the embedded feature sequence and the multiple query prototypes; and weighted aggregating the image patch features based on the allocation matrix to obtain the multiple merged features, wherein the multiple query prototypes and the embedding layer including trainable parameters are jointly optimized through a training process so that the multiple query prototypes correspond to one of multiple categories in the whole slice image, and the normalized inner product similarity between the embedded features obtained by the embedding layer for image patches corresponding to the same category and the query prototype of the corresponding category is greater than the normalized inner product similarity between the embedded features and other query prototypes.

[0009] In some embodiments, the training process includes: acquiring a training set, the training set including multiple training full-slice images pre-labeled with slice-level labels, the slice-level labels being negative or positive; constructing a linear classifier containing a trainable weight matrix; executing the classification method for each training full-slice image in the training set, obtaining training embedding features corresponding to each image patch through the embedding layer, and obtaining a training attention score for each image patch during the global attention calculation; generating image patch-level pseudo-labels based on the slice-level labels of the training full-slice image and the image patches corresponding to the training attention scores, wherein the pseudo-label generation strategy includes: for training full-slice images with negative slice-level labels, setting all pseudo-labels of all image patches of the training full-slice image to 0; for training full-slice images with positive slice-level labels, setting the pseudo-labels of each image patch of the training full-slice image to 0; and for training full-slice images with positive slice-level labels, setting the pseudo-labels of all image patches of the training full-slice image to 0. The pseudo-labels of the top k% of image patches in terms of attention score are set to 1, and the pseudo-labels of the remaining image patches are set to 0. The linear classifier is used to perform image patch-level probability prediction on the training embedding features, and a guiding loss is calculated based on the cross-entropy between the probability prediction and the pseudo-label. The training loss is minimized to jointly optimize the embedding layer, the trainable weight matrix of the linear classifier, and the multiple query prototypes. This ensures that the embedding features of different image patches belonging to the same category obtained through the embedding layer are close to each other in the output space of the embedding layer, while the embedding features of image patches belonging to different categories are separated in the output space of the embedding layer. Furthermore, each of the multiple query prototypes falls into the corresponding category region in the output space of the embedding layer. The training process is performed iteratively in an end-to-end manner, with the classification method re-executed in each iteration to update the attention score and the pseudo-label.

[0010] In some embodiments, the spatial distance matrix is ​​constructed by calculating the Manhattan distance between every two image patches in the two-dimensional spatial coordinates; the topology-aware spatial bias matrix is ​​obtained by the following formula; the topology-aware spatial bias matrix is ​​obtained by the following formula:

[0011] Where S is the allocation matrix and D is the spatial distance matrix. Let be the bias matrix of the topology-aware space.

[0012] In some embodiments, the global attention calculation includes a multi-head attention mechanism and is performed using the following formula:

[0013] in, This represents the output of the global attention calculation, where Q, K, and V are the query matrix, key matrix, and value matrix obtained by linear projection of the merged feature sequence, respectively. Scaling factor This refers to the scaling scalar corresponding to each attention head in the multi-head attention mechanism. Let be the bias matrix of the topology-aware space.

[0014] According to one or more aspects of embodiments of this disclosure, a classification system for whole-slice images based on learnable feature merging and topology awareness is provided. The classification system includes: a feature extraction module configured to perform image patch segmentation on the whole-slice image to obtain multiple image patches, and to obtain an initial feature sequence, the initial feature sequence including two-dimensional spatial coordinates of a predetermined position of each image patch in the whole-slice image and image patch features characterizing the visual morphology of the image patch; and a merging module configured to obtain an allocation matrix for soft feature allocation based on the initial feature sequence, and to weightedly aggregate the image patch features into multiple merged features according to the allocation matrix, thereby obtaining a classification system based on the learned feature merging and topology awareness of the whole-slice image. A merged feature sequence consisting of multiple merged features, wherein the length of the merged feature sequence is less than the length of the initial feature sequence; a topology-aware module is configured to calculate a spatial distance matrix between the multiple image patches based on the two-dimensional spatial coordinates of each image patch, and to map the spatial distance matrix to a topology-aware spatial bias matrix reflecting the spatial distance relationship between the multiple merged features using the allocation matrix; a classification module is configured to perform global attention calculation on the merged feature sequence using the topology-aware spatial bias matrix as a negative bias term, and to predict the classification result of the whole slice image based on the features output by the global attention calculation.

[0015] According to one or more aspects of embodiments of the present disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program; wherein, when the processor executes the computer program, it implements the steps of the above-described classification method for full-slice images based on learnable feature merging and topology awareness.

[0016] According to embodiments of this disclosure, a learnable feature merging module compresses long sequences, significantly reducing the computational overhead of large-size images while improving the accuracy of classification and diagnosis. By introducing task-aware capabilities (such as guided branching and guided loss), the merging criteria are decoupled from simple local visual features, making it easier to merge truly redundant regions while preserving various diagnostic boundary features. An assignment matrix is ​​used to map the original spatial distances, forming a topologically aware spatial bias, and distance constraints are introduced in attention calculations to suppress erroneous connections, thereby perfectly preserving the multifocal spatial distribution. By effectively compressing redundant features, this scheme can significantly alleviate the memory overflow bottleneck caused by massive image blocks, facilitating the release of the application potential of large-parameter visual foundation models in extreme-scale image processing such as computational pathology. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0018] Figure 1 This is a flowchart of a method for classifying whole-slice images provided in an embodiment of the present invention.

[0019] Figure 2 This is a block diagram of the overall architecture of a classification system for whole-slice images provided in an embodiment of the present invention.

[0020] Figure 3 This is a conceptual comparison diagram of the effects of existing technologies and the merging strategy provided by this invention on the preservation of lesion boundaries.

[0021] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0023] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such order can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0024] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example is "performing at least one of step one and step two", which means the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0025] Various embodiments of this disclosure will now be described in detail. Numerous specific details are set forth in the following description in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure can be practiced without some of these specific details.

[0026] Figure 1 This is a flowchart of a method for classifying whole-slice images provided in an embodiment of the present invention. Figure 2 This is a block diagram of the overall architecture of a classification system for whole-slice images provided in an embodiment of the present invention. Figure 3 This is a conceptual comparison diagram of the effects of existing technologies and the merging strategy provided by this invention on lesion boundary preservation. Among them, Figure 3 In this context, 'a' represents the original image patch distribution. Figure 3 In the diagram, 'b' represents the existing token pruning strategy. Region 403 corresponds to situations where key information is lost or boundaries are blurred. Figure 3 In the diagram, c represents the token soft merging strategy of the present invention, and region 402 corresponds to the case where key boundaries are preserved.

[0027] Reference Figure 1 This embodiment provides a classification method for whole-slice images based on learnable feature merging and topology awareness. This classification method can be implemented using a framework called LTM-MIL (Learnable Token Merging - Multiple Instance Learning), which balances computational efficiency and preservation of pathological spatial topology when processing large-view-area WSI data. The classification method includes the following steps S100 to S400.

[0028] In step S100, the full slice image is segmented into multiple image blocks, and an initial feature sequence is obtained. The initial feature sequence includes two-dimensional spatial coordinates representing the positional relationship between the image block and the full slice image, and image block features representing the visual morphology of the image block.

[0029] Reference Figure 1 and Figure 2The feature extraction module (also known as the hybrid local feature extraction module) 100 is configured to perform step S100. Redundant descriptions are omitted here.

[0030] In some embodiments, the step of obtaining the initial feature sequence includes: determining the two-dimensional spatial coordinates of each image patch based on its segmentation position in the full slice image; extracting features from each image patch using a pre-trained visual encoder to obtain encoded features, and calculating image patch features using a sliding window attention mechanism, such that the image patch features are fused with the local morphological context of adjacent image patches; wherein, the calculation of the sliding window attention mechanism includes: applying two-dimensional rotational position encoding to the query features obtained by linear projection of the encoded features, calculating attention weights using a Sigmoid activation function, and weighting and aggregating the value features obtained by linear projection of the encoded features based on the attention weights to obtain image patch features.

[0031] Reference Figure 1 and Figure 2 The hybrid local feature extraction module 100 includes an input module 101, an image block segmentation and pre-trained encoder (also known as a pre-trained encoder) 102, a local attention submodule (also known as a sliding window attention submodule) 103, a two-dimensional rotation position encoder 104 (RoPE 2D) and an initial feature sequence generation module 105.

[0032] The input module 101 receives whole-slide images to be classified from an external source. For example, a whole-slide image (WSI) is a gigapixel-level pathological scan image, typically larger than 100,000 × 100,000 pixels.

[0033] Image block segmentation and a pre-trained encoder (also called a pre-trained encoder) 102 perform image block segmentation on the full slice image, uniformly dividing the full slice image into multiple non-overlapping image blocks. The segmentation position of each image block determines the two-dimensional spatial coordinates of that image block in the full slice image. In one embodiment, the two-dimensional spatial coordinates can be the pixel coordinates of a predetermined position point of the image block (e.g., the center point, the upper left point, or the lower right point) in the full slice image; in another embodiment, the two-dimensional spatial coordinates can also be the pixel coordinates of the image block in the segmentation grid (...). The row and column numbers in the matrix are used, for example, the coordinates of the image patch in the i-th row and j-th column are (i, j). Then, a pre-trained visual encoder (such as ResNet, ViT, or PathGen-CLIP, a basic visual model) is used to extract features from each image patch, resulting in encoded features (also called feature vectors). After encoding, a sequence is obtained. ,in Let D be the encoded feature of the i-th image patch. Here are the corresponding two-dimensional spatial coordinates, and N is the total number of image patches.

[0034] Subsequently, a pre-trained visual encoder is used to extract features from each image patch. To encode the interactions between adjacent tissues in the pathological image, the encoded features are fed into a local attention submodule (also known as a sliding window attention submodule) 103. The sliding window attention mechanism performs attention computation within a local neighborhood window, ensuring that the features of each image patch not only reflect its own visual morphology but also incorporate the morphological information (i.e., local morphological context) of its local neighborhood image patches. Within the local attention submodule 103, the encoded features are linearly projected to obtain query features. Key features Sum Value Characteristics ,in The weight matrix is ​​learnable; here, the standard linear projection operation in the Transformer self-attention mechanism is used. To preserve relative spatial information, a two-dimensional rotation position encoding (RoPE 2D) is applied to the input query features in the calculation of the sliding window attention mechanism. The calculation method is as follows: ,in The coordinates of the image patch are given in two dimensions. By using rotational position encoding, relative orientation information based on the spatial location of the image patch is embedded in the query features. Simultaneously, in the sliding window attention calculation, a sigmoid activation function is used to calculate the attention weights, preventing excessive concentration of attention weights on a few locations. In the standard Transformer attention mechanism, a softmax activation function is typically used to normalize the attention scores, ensuring that the sum of all attention weights is 1, resulting in competitive weight allocation. However, in this embodiment, a sigmoid activation function is used instead of softmax, allowing the attention weights at each location to be independently mapped to... This interval avoids the phenomenon of excessive concentration of attention weights caused by competitive normalization.

[0035] After processing by the local attention submodule 103, the initial feature sequence Z is output by the initial feature sequence generation module 105. Each element in the initial feature sequence Z corresponds to an image patch, which contains the image patch features enhanced by local morphological context and the corresponding two-dimensional spatial coordinates, forming a complete initial feature sequence.

[0036] Return to reference Figure 1 In step S200, an allocation matrix for soft feature allocation is obtained based on the initial feature sequence, and the image patch features are weighted and aggregated into multiple merged features according to the allocation matrix to obtain a merged feature sequence composed of multiple merged features. The length of the merged feature sequence is less than the length of the initial feature sequence.

[0037] Reference Figure 1 As shown in Figure 2, the merging module (also known as the semantically guided task-aware merging module) 200 is configured to execute step S200. Redundant descriptions are omitted here.

[0038] In some embodiments, the step of obtaining an allocation matrix for soft feature allocation based on an initial feature sequence and weighted aggregating image patch features into multiple merged features according to the allocation matrix includes: projecting the image patch features of each image patch in the initial feature sequence into corresponding embedded features through an embedding layer to obtain an embedding feature sequence composed of multiple embedded features; setting multiple learnable query prototypes and obtaining an allocation matrix by calculating the normalized inner product similarity between the embedded feature sequence and the multiple query prototypes; and weighted aggregating the image patch features based on the allocation matrix to obtain multiple merged features. The multiple query prototypes and the embedding layer including trainable parameters are jointly optimized through a training process so that the multiple query prototypes correspond to one of multiple categories in the full-slice image, and the normalized inner product similarity between the embedded features obtained by the embedding layer for image patches corresponding to the same category and the query prototype of the corresponding category is greater than the normalized inner product similarity between the embedded features and other query prototypes.

[0039] Reference Figure 1 and Figure 2 The merging module 200 includes: a decoupled embedding layer (also known as an embedding layer) 201, a task-aware guiding branch (also known as a linear classifier) ​​202, a query merging submodule 203, a feature merging submodule 205, and a merged feature sequence output module 206.

[0040] The decoupled embedding layer 201 projects the initial input feature sequence Z into the lightweight embedding space, calculated as follows: ,in and Let D be the trainable parameters, and E be the embedded feature sequence. The decoupled embedding layer 201 is essentially a standard fully connected linear layer that performs an affine transformation to map the features from the original dimension D to the embedding dimension. Because the original image patch features Z will be used for subsequent global attention calculation and final classification, if Z is directly used to calculate the merging assignment matrix S, the optimization objective of the merging decision and the optimization objective of the classification decision will be coupled in the same feature space. By projecting Z into an independent embedding space E through an embedding layer, and using E to calculate the assignment matrix S, the merging criteria are decoupled from the visual features, effectively separating the embedding spaces of tumors and stroma.

[0041] A task-aware guiding branch 202 is introduced to predict the probability at the image patch level. The task-aware guiding branch 202 contains a linear classifier with a trainable weight matrix. , for embedded features After linear transformation and softmax normalization, the output is the probability prediction of which category the image patch belongs to. This linear classifier corresponds to the standard linear classification head in deep learning, i.e., a single fully connected layer without nonlinear hidden layers. Supervised calculation is performed using pseudo-label guided loss. The training process for this guided loss will be described in detail below.

[0042] In the query merging submodule 203, M learnable query prototypes are set. Calculate the allocation matrix S204 using the following formula: ,in Let be the temperature parameter. In this formula, Perform the inner product operation between the embedded features and the query prototype. Perform temperature scaling and normalization. Perform probability normalization. Assignment matrix. Each element in This represents the probability weight of the i-th image patch being soft-assigned to the j-th merge group (corresponding to the j-th query prototype).

[0043] The feature merging submodule 205 performs weighted aggregation of image patch features Z based on the allocation matrix S, and calculates the final merged feature sequence. The merged feature sequence output module 206 outputs the results. This operation uses the transpose of S as the weight matrix to perform a weighted summation of the features of N image patches in Z, resulting in M ​​merged features. (Merged feature sequence) The length M is much smaller than the length N of the initial feature sequence Z ( This achieves efficient compression of feature sequences.

[0044] In some embodiments, the training process includes: acquiring a training set, which includes multiple pre-labeled full-slice images for training, where the slice-level labels are negative or positive; constructing a linear classifier containing a trainable weight matrix (i.e., the linear classifier in the task-aware guidance branch 202); performing a classification method on each full-slice image for training in the training set (i.e., performing a complete forward propagation of the full-slice image for training through steps S100, S200, and S300 in sequence), obtaining the embedding features (i.e., training embedding features) corresponding to each image patch through the embedding layer, and obtaining the attention score (i.e., training attention score) for each image patch during the global attention calculation; and generating image patch-level pseudo-labels for the corresponding image patches based on the slice-level labels and attention scores of the full-slice images for training. The pseudo-label generation strategy includes: for training full-slice images with negative slice-level labels, setting the pseudo-labels of all image patches in the training full-slice image to 0; for training full-slice images with positive slice-level labels, setting the pseudo-labels of the top k% of image patches in the training full-slice image with attention scores to 1, and setting the pseudo-labels of the remaining image patches to 0. A linear classifier is used to perform image patch-level probability prediction on the embedded features, and a guiding loss is calculated based on the cross-entropy between the probability prediction and the pseudo-labels. By minimizing the guiding loss, the trainable weight matrix of the embedding layer, the linear classifier, and multiple query prototypes are jointly optimized. This ensures that the embedded features obtained from different image patches belonging to the same category are close to each other in the output space of the embedding layer, while the embedded features obtained from different image patches belonging to different categories are separated in the output space of the embedding layer. Furthermore, each of the multiple query prototypes falls within the corresponding category region in the output space of the embedding layer. The training process is performed iteratively in an end-to-end manner, with the classification method re-executed in each iteration to update the attention scores and pseudo-labels. Specifically, in the early stages of training, attention scores are generated based on randomly initialized model parameters, resulting in low-quality pseudo-labels. As training iterates, model parameters are gradually optimized, and the attention scores generated by global attention calculation become increasingly accurate. The quality of pseudo-labels improves accordingly, and the supervisory signal guiding the loss also strengthens, forming a virtuous cycle of bootstrapping training. The optimization objective of the entire network is the total loss. ,in For slice-level classification loss, To guide losses. This is a weighting factor.

[0045] This soft merging mechanism differs from existing strategies such as lexical pruning that directly discard image patches. For example, this mechanism avoids the loss or blurring of key tumor boundaries as seen in existing pruning techniques (region 403 in Figure 3b). Figure 3 (Region 402 in c).

[0046] Return to reference Figure 1 In step S300, the spatial distance matrix between multiple image blocks is calculated based on the two-dimensional spatial coordinates of each image block, and the spatial distance matrix is ​​mapped to a topology-aware spatial bias matrix that reflects the spatial distance relationship between multiple merged features using an allocation matrix.

[0047] Referring to Figures 1 and 2, the topology-aware module (also known as the topology bias portion of the topology-aware global interaction and classification module 300) is configured to execute step S300. Redundant descriptions are omitted here.

[0048] In some embodiments, the spatial distance matrix is ​​constructed by calculating the Manhattan distance between every two image patches in two-dimensional spatial coordinates. The topology-aware spatial bias matrix is ​​obtained by the following formula: Where S is the assignment matrix and D is the spatial distance matrix. is the topology-aware space bias matrix.

[0049] Reference Figure 2 The topology sensing module includes a spatial distance matrix generation module 301 and a topology bias calculation submodule 302.

[0050] The two-dimensional spatial distance matrix generation module 301 determines the Manhattan two-dimensional distance matrix D between each pair of the original uncompressed image blocks, i.e. ,in and Let be the two-dimensional spatial coordinates of the i-th and j-th image patches, respectively. Manhattan distance ( Norm). Distance matrix. It fully depicts the spatial topological relationships between all the original image patches.

[0051] The topology bias calculation submodule 302 projects the spatial distance matrix D into the compressed space and uses the assignment matrix S to obtain the topology-aware spatial bias matrix (also known as the expected physical distance matrix). : This projection operation uses the allocation matrix S to... The original spatial distance matrix of dimension is mapped as A dimensional compressed spatial distance matrix, such that Each element in the table reflects the expected distance between the two corresponding merged features in the original physical space.

[0052] Return to reference Figure 1 In step S400, the topology-aware spatial bias matrix is ​​used as a negative bias term to perform global attention calculation on the merged feature sequence, and the classification result of the whole slice image is predicted based on the features output by the global attention calculation.

[0053] Referring to Figures 1 and 2, the classification module (also known as the global interaction and classification part of the topology-aware global interaction and classification module 300) is configured to execute step S400. Redundant descriptions are omitted here.

[0054] In this embodiment, the classification module includes a global attention submodule 303 and a MIL classifier 304.

[0055] In some embodiments, the global attention calculation includes a multi-head attention mechanism, and the topology-aware spatial bias matrix can be injected as a negative bias term (also known as a "negative penalty term") into the Transformer's global self-attention mechanism. For example, in the global attention submodule 303, the attention output can be calculated according to the following formula: in, , , These are the query matrix, key matrix, and value matrix obtained by linear projection of the merged feature sequence, respectively. Scaling factor This refers to the scaling scalar corresponding to each attention head in the multi-head attention mechanism. Let be the bias matrix of the topologically sensed space. Thus, even if two non-adjacent image patches are semantically very similar, if they are far apart in the original space (i.e., ...), they will not be considered adjacent. (If the value is large), the corresponding negative bias will also greatly suppress the erroneous attention connections between them.

[0056] Finally, the MIL classifier 304, based on global attention, calculates the feature predictions for the full-slice image, and after end-to-end joint optimization, outputs the global slice-level prediction results. The optimization objective is... ,in Slice-level classification loss (e.g., cross-entropy loss). To guide losses, This is the tradeoff coefficient used to balance the two loss terms.

[0057] Through the above steps, the classification method of this invention can maintain spatial topology and key pathological boundaries while reducing computational complexity. As a specific example of performance evaluation, applying the LTM-MIL method of this invention to some public datasets (e.g., Camelyon16, Camelyon17, BRACS) achieves superior classification performance metrics while reducing sequence length by 62.8% and FLOPs computation by 28.3%. Furthermore, due to its effective compression capability of histological redundancy information, which alleviates the memory overflow bottleneck, this architecture can also seamlessly adapt to high-dimensional visual base models with large parameter sets (e.g., PathGen-CLIP) in some embodiments, unlocking the application potential of large base models in computational pathology and thus improving the application value of practical clinical auxiliary diagnosis.

[0058] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0059] Reference Figure 4 An electronic device 400 according to embodiments of the present disclosure may include a processor 410 and a memory 420. The processor 410 may include (but is not limited to) a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a microprocessor, an application-specific integrated circuit (ASIC), etc. The memory 420 may store computer programs to be executed by the processor 410. The memory 420 includes high-speed random access memory and / or non-volatile computer-readable storage media. When the processor 410 executes the computer program stored in the memory 420, the classification method for full-slice images based on learnable feature merging and topology awareness, as described above, can be implemented.

[0060] Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store computer programs and any associated data, data files, and data structures in a non-transitory manner and to provide the computer programs and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer programs. In one example, the computer programs and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0061] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A classification method for whole-slice images based on learnable feature merging and topology awareness, characterized in that, The classification method includes: The whole slice image is segmented into multiple image blocks, and an initial feature sequence is obtained. The initial feature sequence includes two-dimensional spatial coordinates representing the positional relationship between the image block and the whole slice image, and image block features representing the visual morphology of the image block. Based on the initial feature sequence, an allocation matrix for soft feature allocation is obtained, and the image patch features are weighted and aggregated into multiple merged features according to the allocation matrix to obtain a merged feature sequence composed of the multiple merged features. The length of the merged feature sequence is less than the length of the initial feature sequence. Based on the two-dimensional spatial coordinates of each image patch, a spatial distance matrix is ​​calculated between the plurality of image patches, and the spatial distance matrix is ​​mapped using the allocation matrix to a topology-aware spatial bias matrix reflecting the spatial distance relationship between the plurality of merged features; and The topology-aware spatial bias matrix is ​​used as a negative bias term to perform global attention calculation on the merged feature sequence, and the classification result of the whole slice image is predicted based on the features output by the global attention calculation.

2. The classification method according to claim 1, characterized in that, The steps for obtaining the initial feature sequence include: The two-dimensional spatial coordinates are determined based on the segmentation position of each image block within the full slice image; and A pre-trained visual encoder is used to extract features from each image patch to obtain encoded features. A sliding window attention mechanism is then used to calculate the image patch features, which incorporate the local morphological context of adjacent image patches. The calculation of the sliding window attention mechanism includes: applying two-dimensional rotation position encoding to the query features obtained by linear projection of the encoded features, calculating attention weights using the Sigmoid activation function, and weighting and aggregating the value features obtained by linear projection of the encoded features based on the attention weights to obtain the image patch features.

3. The classification method according to claim 1, characterized in that, The step of obtaining an allocation matrix for soft feature allocation based on the initial feature sequence, and weighting and aggregating the image patch features into multiple merged features according to the allocation matrix includes: The initial feature sequence is projected into the image block features of each image block in the image block through the embedding layer, and the corresponding embedding features are projected into the embedding features to obtain an embedding feature sequence composed of multiple embedding features; Multiple learnable query prototypes are set up, and the assignment matrix is ​​obtained by calculating the normalized inner product similarity between the embedded feature sequence and the multiple query prototypes; and The image patch features are weighted and aggregated based on the allocation matrix to obtain the multiple merged features. The plurality of query prototypes and the embedding layer including trainable parameters are jointly optimized through a training process so that the plurality of query prototypes correspond to one of the plurality of categories in the whole slice image, and the normalized inner product similarity between the embedding features obtained by the embedding layer for image blocks corresponding to the same category and the query prototype of the corresponding category is greater than the normalized inner product similarity between the embedding features and other query prototypes.

4. The classification method according to claim 3, characterized in that, The training process includes: Obtain a training set, which includes multiple full-slice images for training that are pre-labeled with slice-level labels, wherein the slice-level labels are negative or positive; Construct a linear classifier containing a trainable weight matrix; The classification method is executed for each training full-slice image in the training set. The training embedding features corresponding to each image patch are obtained through the embedding layer, and the training attention score of each image patch is obtained during the global attention calculation. Based on the slice-level labels of the training full-slice image and the corresponding image patches with the training attention scores, image patch-level pseudo-labels are generated. The pseudo-label generation strategy includes: for training full-slice images with negative slice-level labels, setting the pseudo-labels of all image patches in the training full-slice image to 0; for training full-slice images with positive slice-level labels, setting the pseudo-labels of the top k% of image patches in each image patch with training attention scores to 1, and setting the pseudo-labels of the remaining image patches to 0; and... The linear classifier is used to perform image patch-level probability prediction on the training embedding features, and a guiding loss is calculated based on the cross-entropy between the probability prediction and the pseudo-label. The embedding layer, the trainable weight matrix of the linear classifier, and the multiple query prototypes are jointly optimized by minimizing the guiding loss. This ensures that the embedding features obtained from different image patches belonging to the same category are close to each other in the output space of the embedding layer, while the embedding features obtained from image patches belonging to different categories are separated in the output space of the embedding layer. Furthermore, each of the multiple query prototypes falls within the corresponding category region in the output space of the embedding layer. The training process is performed iteratively in an end-to-end manner, with the classification method being re-executed in each iteration to update the attention score and the pseudo-label.

5. The classification method according to claim 1, characterized in that, The spatial distance matrix is ​​constructed by calculating the Manhattan distance between every two image blocks in the two-dimensional spatial coordinates. The topology-aware spatial bias matrix is ​​obtained by the following formula: Where S is the allocation matrix and D is the spatial distance matrix. Let be the bias matrix of the topology-aware space.

6. The classification method according to claim 1, characterized in that, The global attention calculation includes a multi-head attention mechanism and is performed using the following formula: in, This represents the output of the global attention calculation, where Q, K, and V are the query matrix, key matrix, and value matrix obtained by linear projection of the merged feature sequence, respectively. Scaling factor This refers to the scaling scalar corresponding to each attention head in the multi-head attention mechanism. Let be the bias matrix of the topology-aware space.

7. A classification system for whole-slice images based on learnable feature merging and topology awareness, characterized in that, The classification system includes: The feature extraction module is configured to perform image block segmentation on the whole slice image to obtain multiple image blocks, and to obtain an initial feature sequence, wherein the initial feature sequence includes the two-dimensional spatial coordinates of the predetermined position of each image block in the whole slice image and image block features characterizing the visual morphology of the image block; The merging module is configured to obtain an allocation matrix for soft feature allocation based on the initial feature sequence, and to weight and aggregate the image patch features into multiple merged features according to the allocation matrix, thereby obtaining a merged feature sequence composed of the multiple merged features, wherein the length of the merged feature sequence is less than the length of the initial feature sequence. The topology-aware module is configured to calculate the spatial distance matrix between the plurality of image blocks based on the two-dimensional spatial coordinates of each image block, and to map the spatial distance matrix into a topology-aware spatial bias matrix that reflects the spatial distance relationship between the plurality of merged features using the allocation matrix. The classification module is configured to perform global attention calculation on the merged feature sequence using the topology-aware spatial bias matrix as a negative bias term, and predict the classification result of the full-slice image based on the features output by the global attention calculation.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program; wherein, when the processor executes the computer program, it implements the classification method for full-slice images as described in any one of claims 1-6.

Images