A method and system for classifying histopathological slide images based on vector quantization attention

Abstract

The present application belongs to the technical field of image recognition, and discloses a histopathological section image classification method and system based on vector quantization attention. The method comprises the following steps: obtaining the category label of multiple histopathological section images, dividing the tissue area of each histopathological section image to obtain multiple image blocks, and taking the image blocks as an example set as a bag; using a feature extraction model to extract the features of the examples in each bag to obtain the feature sequence of each bag; constructing a multiple example classification model based on a vector quantization attention module and a multi-scale convolution module, and training the multiple example classification model using the feature sequence and the category label; obtaining the target feature sequence of the histopathological section image to be classified; inputting the target feature sequence into the trained multiple example classification model to obtain the attention score of each feature vector, comparing the attention score with a preset threshold, and obtaining the disease area. The present application solves the problems of low classification accuracy and low diagnosis efficiency of histopathological images.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and more specifically, relates to a method and system for classifying histopathological slide images based on vector quantization attention. Background Technology

[0002] Cancer is a serious threat to human health. For almost all cancers, screening and early diagnosis can improve the survival rate and quality of life of cancer patients, and reduce the cost of cancer treatment and complications. Traditional cancer detection methods mainly rely on the microscopic observation and analysis of tissue sections by pathologists. This method is not only time-consuming and labor-intensive, but also depends on the level and experience of pathologists, and suffers from problems such as high subjectivity, low repeatability, and insufficient quantitative analysis.

[0003] In recent years, digital pathology and computer vision technologies have been widely applied and developed in the field of cancer detection. Whole-slide image (WSI) technology enables the digital storage and computer analysis of tissue pathology slide images. Unlike natural images, whole-slide images have high resolution, displaying cellular structures and pathological changes at different magnifications. Containing approximately one trillion pixels, the entire slide cannot be directly input into a convolutional neural network for detection, and pixel-level annotations are difficult to obtain. Diagnosing whole-slide images often faces situations where only the slide's category label is available, yet the identification and detection of cancerous regions within the slide is required.

[0004] Multiple-Instance Learning (MIL) is a weakly supervised learning method well-suited for solving the aforementioned problems. A full-view digital slice is defined as a "bag," and all cropped image patches are defined as "instances" within that bag. Researchers typically build models to classify bags containing multiple instances. The model extracts instance features and maps them to labels or scores. A threshold is set to classify instances into cancerous tissue instances and normal tissue instances, and the corresponding cancer regions are visualized at the scale of the original slice. In recent research, some researchers have used attention mechanisms, multi-scale structures, clustering algorithms, graph networks, and residual connections to construct analyses for different types of cancer slices, such as breast cancer and colon cancer, achieving efficient and accurate cancer detection and segmentation. However, these algorithms often only use bag-level clustering, ignoring instance similarity, or they use traditional clustering methods when constructing instance similarity metrics, resulting in overly complex models and difficult optimization. They fail to effectively combine attention mechanisms with instance similarity relationships. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for classifying histopathological slide images based on vector quantization attention. This solves the problem that existing multi-instance classification models, when processing histopathological slide images containing only category labels, pay low attention to the similarity of instances, resulting in poor detection of disease areas and thus affecting the efficiency of doctors' diagnoses.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for classifying histopathological slide images based on vector quantization attention is provided, comprising: a training phase S1: obtaining category labels for multiple histopathological slide images, extracting the tissue region for each histopathological slide image, dividing the tissue region into multiple image blocks, and grouping the multiple image blocks as an example set into a bag, thus multiple histopathological slide images correspond to multiple bags; S2: using a feature extraction model to extract features from the examples within each bag, thereby obtaining the feature vector of each example in each bag, and obtaining the feature sequence of each bag based on the position of the example in the bag; S3: based on a vector quantization attention module and multiple... The scaled convolution module constructs a multi-instance classification model, using the feature sequence of each bag as input and the category label corresponding to that bag as the optimization objective to train the multi-instance classification model and obtain a trained multi-instance classification model. In the application stage, steps S1 and S2 are performed on the pathological slide image of the tissue to be classified to obtain the target feature sequence of the pathological slide image of the tissue to be classified. The target feature sequence is input into the trained multi-instance classification model to obtain the attention score of the example corresponding to each feature vector in the target feature sequence. The attention score of each example is compared with a preset threshold to obtain normal examples and abnormal examples. Based on the position of the abnormal examples in the pathological slide image of the tissue to be classified, the disease region is determined.

[0007] Preferably, the step S1 of extracting the tissue region for each histopathological slide image specifically involves: obtaining a mask for the tissue region of each histopathological slide image; cropping the histopathological slide image and its mask to generate multiple image blocks and corresponding masks; calculating the proportion of cell tissue in each image block based on the mask; and selecting image blocks with a proportion greater than a preset threshold as an example set into a package.

[0008] Preferably, obtaining the mask of the tissue region in each histopathological slide image specifically involves using one of the following methods to segment the histopathological slide image into tissue regions and background regions: Otsu's algorithm, histogram bimodal method, iterative threshold segmentation, watershed algorithm, seed region growth, region splitting and merging, active contour method, HSV color space-based segmentation, LAB color space-based segmentation, or clustering in color space.

[0009] Preferably, the feature extraction model sequentially includes a ResNet50 network structure with the last convolutional module removed, and an adaptive average pooling layer that pools the 1024×14×14 image features into a 1024-dimensional feature vector.

[0010] Preferably, step S3 further includes performing a linear transformation on the feature sequence to achieve dimensionality reduction, and dividing the dimensionality-reduced feature sequence into subsequences according to positional order, and using the subsequences to train the multi-example classification model; in a further preferred embodiment, the subsequences are mutually exclusive.

[0011] Preferably, the vector quantization attention module includes a labeling submodule and an attention calculation submodule. The labeling submodule is used to add a label vector before the first feature vector of the subsequence to form a combined sequence. The combined sequence is then input into the attention calculation submodule to obtain the corresponding vector quantization loss, the labeled vector, and the attention score of each example in the combined sequence.

[0012] Preferably, the vector quantization attention module further includes a post-processing submodule, which is used to concatenate multiple labeled vectors to form a feature matrix, average multiple vector quantization losses to obtain an average vector quantization loss, and combine multiple attention scores to obtain a target attention score for the corresponding example.

[0013] Preferably, the multi-scale convolution module includes three one-dimensional convolutional layers. The number of input channels of each one-dimensional convolutional layer is the same as the number of sub-sequences divided from the feature sequence of each bag. The number of input and output channels is 1, the stride is 1, and the kernel sizes of the three one-dimensional convolutional layers are 7, 5, and 3, respectively, and the padding sizes are 3, 2, and 1, respectively.

[0014] Preferably, the weighted total loss L of the multi-instance classification model is:

[0015]

[0016] in, The coefficient of cross-entropy loss. For average vector quantization loss, This represents the cross-entropy loss.

[0017] A second aspect of this application provides a histopathological slide image classification system based on vector quantization attention, comprising: a training unit; a bag acquisition module: used to acquire category labels for multiple histopathological slide images, extract tissue regions for each histopathological slide image, divide the tissue regions into multiple image blocks, and group the multiple image blocks as an example set into a bag, thus multiple histopathological slide images correspond to multiple bags; a feature sequence acquisition module: used to extract features from the examples in each bag using a feature extraction model, thereby obtaining the feature vector of each example in each bag, and obtaining the feature sequence of each bag based on the position of the example in the bag; and a training module: used to train the system based on the vector quantization attention module and multiple... The scaled convolution module constructs a multi-instance classification model, using the feature sequence of each bag as input and the category label corresponding to that bag as the optimization objective to train the multi-instance classification model and obtain a trained multi-instance classification model. The application unit executes the bag acquisition module and the feature sequence acquisition module on the pathological slide image to be classified to obtain the target feature sequence of the pathological slide image to be classified. The target feature sequence is input into the trained multi-instance classification model to obtain the attention score of each feature vector corresponding to the example in the target feature sequence. The attention score of each example is compared with a preset threshold to obtain normal examples and abnormal examples. Based on the position of the abnormal examples in the pathological slide image to be classified, the disease region is determined.

[0018] In summary, compared with the prior art, the histopathological slide image classification method based on vector quantization attention provided by the present invention has the following beneficial effects:

[0019] 1. This application constructs a multi-instance classification model based on a vector quantization attention module and a multi-scale convolution module. The vector quantization attention module uses the vector quantization method to express the feature vectors of all instances as nearest neighbors, thereby enabling the focus and recognition of the similarity between instances in the same or different packages, realizing implicit clustering and feature expression among similar features, and significantly improving classification accuracy.

[0020] 2. This application adopts a method of dividing long sequence features into subsequences for separate calculation, and uses a labeling submodule to label them to generate learnable label vectors. The label vectors are used to represent the information of different subsequences, which saves computational resources.

[0021] 3. The multi-scale convolution module of this application includes three one-dimensional convolutional layers. The number of input channels of each one-dimensional convolutional layer is the same as the number of sub-sequences divided from the feature sequence of each bag. The multi-scale structure realizes the information transmission and feature fusion of different sub-sequences. Attached Figure Description

[0022] Figure 1This is a schematic diagram illustrating the steps of the histopathological slide image classification method based on vector quantization attention in an embodiment of this application;

[0023] Figure 2 This is a flowchart illustrating the histopathological slide image classification method based on vector quantization attention, as described in an embodiment of this application.

[0024] Figure 3 This is a schematic diagram illustrating the training of the classification model in an embodiment of this application;

[0025] Figure 4 This is image data of a pathological section of breast tissue from an embodiment of this application;

[0026] Figure 5 This is an attention visualization of a pathological section image of breast tissue from an embodiment of this application;

[0027] Figure 6 This is a schematic diagram of the vector quantization attention module mechanism in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0029] The first aspect of this application provides a method for classifying histopathological slide images based on vector quantization attention, such as... Figure 1 and Figure 2 As shown, it includes the training phase and the application phase.

[0030] The training phase includes the following steps S1 to S3.

[0031] S1: Obtain the category labels of multiple histopathological slide images, extract the tissue region for each histopathological slide image, divide the tissue region to obtain multiple image blocks, and take the multiple image blocks as an example set into a package, so that multiple histopathological slide images correspond to multiple packages.

[0032] Multiple histopathological slide images can be from different or the same patients in the same organ, and the slides are recorded as electronic images using a magnifying glass. The presence or type of lesion in the slides can serve as a category label, which can be manually labeled.

[0033] In a further preferred embodiment, the extraction of the tissue region for each of the histopathological slide images specifically involves:

[0034] Obtain the mask of the tissue region in each histopathological slide image, crop the histopathological slide image and its mask to generate multiple image blocks and corresponding masks, calculate the proportion of cell tissue in each image block based on the mask, and filter out image blocks with a proportion greater than a preset threshold as an example set into a package.

[0035] In a further preferred scheme, one of the following methods is used to segment a tissue pathology slide image into tissue regions and background regions: Otsu's method, histogram bimodal method, iterative threshold segmentation, watershed algorithm, seed region growth, region splitting and merging, active contour method, HSV color space-based segmentation, LAB color space-based segmentation, and clustering in color space.

[0036] Multiple image patches are generated by sliding window cropping of the tissue region. The proportion of cell tissue in each image patch is calculated, and image patches with a proportion greater than a preset threshold are selected as examples.

[0037] In this embodiment, the Otsu algorithm is preferred for segmenting tissue pathology slide images, as defined below:

[0038]

[0039] Where g represents the inter-class variance, which in this embodiment corresponds to the cell tissue region and the background region. and These represent the proportion of pixels in the cellular tissue region and the background region to the total image, respectively. and Let g be the average gray value of the cellular tissue region and the background region, respectively. Otsu's method seeks an optimal threshold that maximizes g, with values ​​less than and greater than this threshold representing the cellular tissue region and the background region, respectively. This optimal threshold can be calculated by iterating through all possible gray values, calculating the inter-class variance g for each gray value, and then selecting the gray value corresponding to the largest inter-class variance g as the optimal threshold.

[0040] After segmenting the histopathological slide image using the Otsu algorithm, binarization is performed using the optimal threshold mentioned above to convert the histopathological slide image into a binary image, which is then used as the mask for the histopathological slide image. Using an L×L sliding window to crop the histopathological slide image, multiple L×L image blocks can be obtained, and their masks are also cropped to L×L. If all pixels in the mask corresponding to a certain image block are black, then the image block only contains the background and should be removed; conversely, if all pixels are white, then it only contains cellular tissue and should be retained. For image blocks that contain both background and cellular tissue, the proportion of white pixels in the mask is calculated to determine whether to retain or remove them; in this embodiment, this proportion is set to 85%.

[0041] Image blocks whose proportion of cell tissue pixels to the total number of pixels in the image block is greater than a preset proportion are taken as an example set and grouped into a package. Therefore, multiple tissue pathology slide images correspond to multiple packages, and the number of examples N in each package is not necessarily the same.

[0042] S2: Use a feature extraction model to extract features from the examples in each bag, thereby obtaining the feature vector of each example in each bag, and obtaining the feature sequence of each bag based on the position of the example in the bag.

[0043] Using feature extraction models f (x), input the example for each package f (x) Perform feature extraction to obtain N feature vectors, each with dimension D. Then the feature sequence of this bag is... x ∈N×D.

[0044] In a further optimized scheme, the feature extraction model f (x) is a modified ResNet50 pre-trained on the natural image dataset ImageNet. The modified network structure is a ResNet50 network structure with the last convolutional module removed, and an adaptive average pooling layer that can pool the image features of dimension 1024×14×14 into a 1024-dimensional feature vector. Then D is 1024.

[0045] S3: Construct a multi-instance classification model based on the vector quantization attention module and the multi-scale convolution module. Take the feature sequence of each bag as input and the category label corresponding to the bag as the optimization objective to train the multi-instance classification model to obtain the trained multi-instance classification model.

[0046] like Figure 3 As shown, the multi-instance classification model of this application g ( x Includes feature encoding module e ( xVector quantization attention module a ( x and multi-scale convolution modules c ( x )

[0047] Feature encoding module e ( x It contains a linear layer, an activation function layer, and a random deactivation layer, where the deactivation probability of the random deactivation layer is 0.25. e ( x The feature sequence H∈N×D is encoded into the feature sequence h∈N×E.

[0048] The vector quantization attention module includes a labeling submodule and an attention calculation submodule. The labeling submodule is used to add a label vector to the first feature vector of the subsequence to form a combined sequence. The combined sequence is then input into the attention calculation submodule to obtain the corresponding vector quantization loss, the labeled vector, and the attention score of each example in the combined sequence.

[0049] like Figure 6 As shown, the vector quantization attention module a ( x The input is a feature sequence h∈N×E. First, the feature sequence h∈N×E is divided into NS subsequences of roughly equal length and mutually disjoint. Assume the number of feature vectors in each subsequence is S. If N is not divisible by NS, the number of feature vectors S in the last subsequence may differ from the other subsequences, but this application will use S to represent the number of feature vectors in each subsequence. A trainable label matrix token is defined, where token∈NS×E. The label matrix token can be decomposed into individual trainable label vectors tokeni, each tokeni∈1×E. The labeling submodule adds the corresponding tokeni before the first feature vector of each subsequence hi∈S×E, forming a combined sequence. ∈(S+1)×E.

[0050] Sequential input combination sequence The attention submodule outputs this combined sequence. Attention score Ai∈S×1 corresponding to the feature vector of the subsequence, and the labeled vector of the subsequence. Vector quantization loss of ∈1×E and subsequence ,in, for After the feature information of the subsequence is extracted by the vector quantization attention module, the labeled vector is separated from its head.

[0051] By calculating each subsequence using the attention calculation submodule, we obtain NS Ai and NS [unclear] ... and NS vector quantization loss NS Ai and NS The target attention score A∈N×1 and the feature matrix T∈NS×E of the feature sequence are obtained by combining the feature vectors of the feature sequence, and the loss is quantized from the NS vectors. The average vector quantization loss of the feature sequence is obtained by averaging. .

[0052] The attention calculation submodule performs the following process for vector quantization of attention for each subsequence:

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] Where X is an input subsequence with feature dimension E. and It is a trainable parameter matrix, Q, K, and V are the query, key, and value matrices after linear transformation, and Z is the discrete encoding matrix after vector quantization encoding VQ(•). Where is the dimension of the bond, B is the trainable uniform band matrix, DWC(·) is depthwise convolution, and LN(·) is layer normalization. It is the attention score of the vector portion without removing the labels. It is the attention output matrix. Y is the result of residual connection between the attention output matrix after layer normalization and the input vector. Y is the output of vector quantization attention calculation of the subsequence, and the labeled vector can be separated from its head.

[0063] The vector quantization attention module also includes a post-processing submodule, which is used to concatenate multiple labeled vectors to form a feature matrix, average multiple vector quantization losses to obtain the average vector quantization loss, and combine multiple attention scores to obtain the target attention score for the corresponding example.

[0064] The core idea of ​​vector quantization attention is to represent the key matrix using a finite set of encodings, thereby reducing the scale and complexity of softmax operations in attention computation and performing implicit clustering. To address the low-rank problem of the clustered matrix, a strip-shaped trainable matrix is ​​added as a bias term for attention, focusing on sequence relationships within a finite length and acting as a positional encoding. Depthwise separable convolutions are performed on the value matrix as a compensation term to enrich the feature representation of the attention output matrix. Finally, attention scores are calculated through linear layers and multiplied by the input to achieve weighted summation of the input vector. In actual computation, to obtain the example attention scores Ai∈S×1 and the label information tokeni∈1×E for the corresponding subsequences of the combined sequence hi∈(S+1)×E, the following needs to be removed: The first value, which is the value corresponding to the label vector, and only retain the first value. The first vector is the vector corresponding to the marker vector.

[0065] The definition of vector quantization coding VQ(•) is:

[0066]

[0067] in, It is the input vector, which in this embodiment is the key matrix K. It is the output vector, which in this embodiment is the discrete encoding matrix Z. It is the encoded vector in the vector quantization encoding table. It is a trainable vector quantization encoding table, NC is the number of codes in the vector quantization encoding table, and ||•|| is the norm of the vector, usually the Euclidean norm. The process of vector quantization is to find the encoding vector that is closest to the input vector and use it to replace the input vector.

[0068] The vector quantization attention module calculates the Euclidean distance between the vector-quantized feature vector and the original feature vector as the vector quantization loss. Therefore, vector quantization loss The definition is as follows:

[0069]

[0070]

[0071] Where Num is the number of input vectors, which in this embodiment is the number of eigenvectors contained in the key matrix, and is also equivalent to the number of eigenvectors contained in the combined sequence, i.e., S+1. yes The i-th input vector, In this embodiment, K refers to the process of calculating the vector quantization attention for each subsequence. yes The output vector after vector quantization according to exist The order in can be combined as , In this embodiment, Z, sg(·) represents the gradient cutoff, and d(·) is the distance metric, typically Euclidean distance or squared error, in the process of calculating the vector quantization attention for each subsequence. The coefficient of the second term is given, and the first term of the loss represents the coefficient of the first term. and Closer, the second item indicates that... and Closer.

[0072] A trainable uniform banded matrix B can be defined as:

[0073]

[0074] in, For matrix B The element in the i-th row and j-th column, where b represents bandwidth and c represents trainable parameters.

[0075] Suppose that the depthwise separable convolution (DWC) consists of a depthwise convolution part and a pointwise convolution part. The input of the depthwise separable convolution is... The output is Then it can be defined as:

[0076]

[0077] in, This is the input feature map, with a shape of C1×H×W, where C1 is the number of input channels, and H and W are the height and width of the feature map, respectively. Indicates the output channel is A depthwise convolution operation with a kernel size of N×N and padding size of N divided by 2; It is the bias term of the depthwise convolution, and its shape is 1× ; It is the output of depthwise convolution, and its shape is ×H×W; For output channel A pointwise convolution operation with a kernel size of 1×1 and padding size of 0; It is the bias term of the depthwise convolution, and its shape is 1× ; It is the output of pointwise convolution, and its shape is ×H×W; This layer-by-layer batch normalization can make feature output smoother and model training more stable. This application will... The dimension is transformed from 1×S×E to 1×E×S to accommodate the computation of depthwise separable convolution. After obtaining the computation result, it is then transformed back to the original dimension and subjected to depthwise convolution. Here... This is equivalent to extending V to the first dimension. In the process of calculating the vector quantization attention for each subsequence, V is calculated using DWC(·). It needs to be converted back to the original shape of V.

[0078] In a further optimized scheme, the multi-scale convolution module uses three one-dimensional convolutional layers, each with NS input channels, 1 output channel, and a stride of 1. The kernel sizes are 7, 5, and 3, and the padding sizes are 3, 2, and 1, respectively. This numerical design ensures that the feature matrix T remains unchanged in feature dimension after passing through the multi-scale convolution module, and is only compressed to 1 in sequence dimension, thereby realizing information transfer and feature fusion of subsequences.

[0079] In a further preferred embodiment, step S3 can be further divided into the following steps S31 to S33.

[0080] S31: Select feature sequences and their true class labels of multiple bags as training sets. When performing a single iteration on the training set, select the feature sequence of one bag and input it into the multi-instance classification model to obtain the attention score of each instance vector in the bag, the predicted classification probability of the bag, and the average vector quantization loss.

[0081] S32: Calculate the cross-entropy between the predicted classification probability of the package and the true class label to obtain the cross-entropy loss. ;

[0082] S33: Obtain the weighted total loss according to the following formula. During a single iteration on the training set, the classification model is trained using backpropagation based on the weighted total loss.

[0083]

[0084] in, The coefficient of cross-entropy loss. For average vector quantization loss, This represents the cross-entropy loss.

[0085] After obtaining the weighted total loss L, backpropagation is performed to train the multi-instance classification model. The network is trained on the training set for one epoch and then validated on the validation set. The average loss on the validation set is calculated and recorded. If the average loss of the current multi-instance classification model on the validation set is smaller than the average loss of the previous multi-instance classification model, the multi-instance classification model is saved and recorded. Otherwise, no record is made and training continues until no multi-instance classification model has an average loss on the validation set smaller than the previous multi-instance classification model within a certain period. Then, training is terminated, and the weights of the multi-instance classification model that performs best on the validation set are saved.

[0086] In a further optimized scheme, a multi-instance classification model is trained, backpropagation is performed using stochastic gradient descent, the Adam optimizer is used, the initial learning rate is 0.00001, the weight decay coefficient is 0.0001, the batch size is 1, and the observation period for validation loss is 50.

[0087] Application phase

[0088] Images of tissue pathology sections to be classified (e.g., Figure 4 (For the breast tissue pathological section image in the image), execute steps S1 and S2 to obtain the target feature sequence of the tissue pathological section image to be classified; after executing step S1 on the tissue pathological section image to be classified, obtain the target package, and execute step S2 on the target package to obtain the target feature sequence.

[0089] The target feature sequence is input into the trained multi-instance classification model to obtain the attention score of each feature vector corresponding to the example in the target feature sequence. The attention score of each example is compared with a preset threshold to obtain normal examples and abnormal examples. Based on the position of the abnormal example in the pathological slice image of the tissue to be classified, the disease area is determined.

[0090] In a further preferred embodiment, the attention scores of the examples of the target package are normalized using a maximum-minimum value, and the attention scores of the examples of the target package are visualized using a color table mapping method, for example... Figure 5 Attention visualization of breast tissue pathological section images.

[0091] In this embodiment, the feature extraction model f The image size in the hyperparameters of g(x) is set to 256; the hyperparameters of the classification model g(x) are set as follows:

[0092]

[0093] Automatic classification algorithms for histopathological slide images are currently in the theoretical stage. Most existing histopathological slide image diagnostic algorithms are based on a single attention pooling method and have high hardware requirements. This application improves upon existing deep learning methods by using vector quantization to enhance the attention mechanism, performing implicit clustering of examples, dividing the attention process into multiple subsequences for separate calculations, and using a trainable label matrix to record subsequence information. Finally, a multi-scale structure is employed for subsequence information transfer and feature fusion. With relatively low hardware memory usage, the model achieves good classification accuracy, with an AUC of 0.9301 on the Camelyon16 dataset used in the experiments. This can provide effective assistance to doctors.

[0094] A second aspect of this application provides a histopathological slide image classification system based on vector quantization attention, comprising:

[0095] Training Unit

[0096] Package acquisition module: used to acquire category labels of multiple histopathological slide images, extract the tissue region for each histopathological slide image, divide the tissue region to obtain multiple image blocks, and use the multiple image blocks as an example set into a package, so multiple histopathological slide images correspond to multiple packages;

[0097] Feature sequence acquisition module: Used to extract features from the examples in each bag using a feature extraction model, thereby obtaining the feature vector of each example in each bag, and obtaining the feature sequence of each bag based on the position of the example in the bag;

[0098] Training module: Used to build a multi-instance classification model based on the vector quantization attention module and the multi-scale convolution module. It takes the feature sequence of each bag as input and the category label corresponding to the bag as the optimization objective to train the multi-instance classification model to obtain the trained multi-instance classification model.

[0099] Application Unit

[0100] The system executes a package acquisition module and a feature sequence acquisition module to obtain the target feature sequence of the tissue pathology slide image to be classified.

[0101] The target feature sequence is input into the trained multi-instance classification model to obtain the attention score of each feature vector corresponding to the example in the target feature sequence. The attention score of each example is compared with a preset threshold to obtain normal examples and abnormal examples. Based on the position of the abnormal example in the pathological slice image of the tissue to be classified, the disease area is determined.

[0102] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for classifying histopathological slide images based on vector quantization attention, characterized in that, include: Training phase S1: Obtain the category labels of multiple histopathological slide images, extract the tissue region for each histopathological slide image, divide the tissue region to obtain multiple image blocks, and take the multiple image blocks as an example set into a package, then the multiple histopathological slide images correspond to multiple packages. S2: Use a feature extraction model to extract features from the examples in each bag, thereby obtaining the feature vector of each example in each bag, and obtaining the feature sequence of each bag based on the position of the example in the bag; S3: Construct a multi-instance classification model based on the vector quantization attention module and the multi-scale convolution module. Take the feature sequence of each bag as input and the category label corresponding to the bag as the optimization objective to train the multi-instance classification model to obtain the trained multi-instance classification model. Application phase Perform steps S1 and S2 on the pathological slide image of the tissue to be classified to obtain the target feature sequence of the pathological slide image of the tissue to be classified. The target feature sequence is input into the trained multi-instance classification model to obtain the attention score of each feature vector corresponding to the example in the target feature sequence. The attention score of each example is compared with a preset threshold to obtain normal examples and abnormal examples. Based on the position of the abnormal example in the pathological slice image of the tissue to be classified, the disease area is determined. Step S3 further includes performing a linear transformation on the feature sequence to achieve dimensionality reduction, and dividing the dimensionality-reduced feature sequence into subsequences according to positional order, using the subsequences to train the multi-example classification model; the subsequences are mutually exclusive. The vector quantization attention module includes a labeling submodule and an attention calculation submodule. The labeling submodule is used to add a label vector to the first feature vector of the subsequence to form a combined sequence. The combined sequence is then input into the attention calculation submodule to obtain the corresponding vector quantization loss, the labeled vector, and the attention score of each example in the combined sequence. The vector quantization attention module also includes a post-processing submodule, which is used to concatenate multiple labeled vectors to form a feature matrix, average multiple vector quantization losses to obtain the average vector quantization loss, and combine multiple attention scores to obtain the target attention score for the corresponding example.

2. The histopathological slide image classification method based on vector quantization attention according to claim 1, characterized in that, The step S1, which involves extracting the tissue region from each of the histopathological slide images, specifically involves: Obtain the mask of the tissue region in each histopathological slide image, crop the histopathological slide image and its mask to generate multiple image blocks and corresponding masks, calculate the proportion of cell tissue in each image block based on the mask, and filter out image blocks with a proportion greater than a preset threshold as an example set into a package.

3. The histopathological slide image classification method based on vector quantization attention according to claim 2, characterized in that, The specific steps to obtain the mask for the tissue region in each histopathological slide image are as follows: One of the following methods is employed to segment histopathological slide images into tissue regions and background regions: Otsu's method, histogram bimodal method, iterative threshold segmentation, watershed algorithm, seed region growth, region splitting and merging, active contour method, HSV-based color space segmentation, LAB-based color space segmentation, and clustering in color space.

4. The histopathological slide image classification method based on vector quantization attention according to claim 1, characterized in that, The feature extraction model includes, in sequence, a ResNet50 network structure with the last convolutional module removed, and an adaptive average pooling layer that transforms 1024×14×14 image features into 1024-dimensional feature vectors.

5. The histopathological slide image classification method based on vector quantization attention according to claim 1, characterized in that, The multi-scale convolution module includes three one-dimensional convolutional layers. The number of input channels of each one-dimensional convolutional layer is the same as the number of sub-sequences divided by the feature sequence of each bag. The number of output channels is 1, and the stride is 1. The kernel sizes of the three one-dimensional convolutional layers are 7, 5, and 3, and the padding sizes are 3, 2, and 1, respectively.

6. The histopathological slide image classification method based on vector quantization attention according to claim 1, characterized in that, The weighted total loss L of the multi-instance classification model is: in, The coefficient of cross-entropy loss. For average vector quantization loss, This represents the cross-entropy loss.

7. A histopathological slide image classification system based on vector quantization attention, characterized in that, include: Application Unit The package acquisition module and the feature sequence acquisition module are executed on the pathological slide images of the tissues to be classified to obtain the target feature sequence of the pathological slide images of the tissues to be classified. The target feature sequence is input into the trained multi-instance classification model to obtain the attention score of each feature vector corresponding to the example in the target feature sequence. The attention score of each example is compared with a preset threshold to obtain normal examples and abnormal examples. Based on the position of the abnormal example in the pathological slice image of the tissue to be classified, the disease area is determined. It also includes training units: Package acquisition module: used to acquire category labels of multiple histopathological slide images, extract the tissue region for each histopathological slide image, divide the tissue region to obtain multiple image blocks, and use the multiple image blocks as an example set into a package, so multiple histopathological slide images correspond to multiple packages; Feature sequence acquisition module: Used to extract features from the examples in each bag using a feature extraction model, thereby obtaining the feature vector of each example in each bag, and obtaining the feature sequence of each bag based on the position of the example in the bag; Training module: Used to build a multi-instance classification model based on the vector quantization attention module and the multi-scale convolution module. It takes the feature sequence of each bag as input and the category label corresponding to the bag as the optimization objective to train the multi-instance classification model to obtain the trained multi-instance classification model. The method also includes performing a linear transformation on the feature sequence to achieve dimensionality reduction, dividing the dimensionality-reduced feature sequence into subsequences according to positional order, and using the subsequences to train the multi-instance classification model; the subsequences are mutually exclusive. The vector quantization attention module includes a labeling submodule and an attention calculation submodule. The labeling submodule is used to add a label vector to the first feature vector of the subsequence to form a combined sequence. The combined sequence is then input into the attention calculation submodule to obtain the corresponding vector quantization loss, the labeled vector, and the attention score of each example in the combined sequence. The vector quantization attention module also includes a post-processing submodule, which is used to concatenate multiple labeled vectors to form a feature matrix, average multiple vector quantization losses to obtain the average vector quantization loss, and combine multiple attention scores to obtain the target attention score for the corresponding example.

Images