WSI image classification system based on hierarchical deep learning

Abstract

The invention discloses a WSI image classification system based on hierarchical deep learning. The system comprises an input unit, a processor and a display unit which are connected in sequence, the sample library is in communication connection with the processor, and a WSI image classification model is integrated on the processor; in the first stage, a WSI image classification model is trained through samples in a sample library until the recognition precision of the WSI image classification model reaches a set standard, and training of the WSI image classification model is completed; and a second stage: inputting the to-be-classified WSI image into the trained WSI image classification model through an input unit, outputting a classification result of the to-be-classified WSI image, and displaying the classification result. By designing a layered deep neural network to learn WSI image features step by step, the model can pay attention to local and global features at the same time, and rich high-quality feature representation can be extracted from the image, so that only slice-level labeling is needed, more refined labeling is not needed, the requirement for a data set is reduced, and the efficiency of the model is improved. And the labeling workload and cost are obviously reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and more specifically, the present invention relates to a WSI image classification system based on hierarchical deep learning. Background Art

[0002] In the traditional method, the doctor needs to manually adjust the different positions of the observation slice through the microscope to observe the tissue status. The process is complicated and subjective, and it is easy to miss or misjudge. Different doctors may have different observation results for the same slice.

[0003] With the development of medical imaging technology, the emergence of full-view digital slides (WSI) has solved some problems, but pathologists still need to view them from the beginning. With the improvement of computer hardware and software performance and the introduction of new technologies, it has become possible to use computer-aided diagnosis technology to assist pathologists in diagnosing WSI images, but it still faces some challenges.

[0004] First, WSI images have an ultra-high resolution, generally reaching 150,000x150,000 pixels, which makes WSI images unable to be directly used in current deep learning methods; second, a large number of WSI images lack annotations, resulting in a small number of samples that can be used for supervised learning. Summary of the invention

[0005] The present invention provides a WSI image classification system based on hierarchical deep learning, aiming to improve the above problems.

[0006] The present invention is implemented as follows: a WSI image classification system based on hierarchical deep learning, the system comprising:

[0007] An input unit, a processor and a display unit connected in sequence;

[0008] The sample library is connected to the processor in communication, wherein the WSI image classification model is integrated on the processor;

[0009] Phase 1: The WSI image classification model is trained using samples in the sample library until the recognition accuracy of the WSI image classification model reaches the set standard, and the training of the WSI image classification model is completed;

[0010] The second stage: the WSI image to be classified is input into the trained WSI image classification model through the input unit, and the WSI image classification model outputs the classification result of the WSI image to be classified and displays it through the display unit.

[0011] Furthermore, the system also includes:

[0012] The image preprocessing module is used to receive the WSI image, divide the foreground area in the WSI image into several sub-image blocks, extract the information of each sub-image module, including the image identifier of the WSI image and the coordinates in the WSI image, and store the sub-image blocks and their information in the sample library.

[0013] Furthermore, the WSI image classification model includes:

[0014] The first vision transformer model and the second vision transformer model are connected in sequence,

[0015] The output of the second vision transformer model is connected to the classifier;

[0016] The first vision transformer model consists of an image embedding module, a position embedding module, a multi-head self-attention module, and a layer normalization module connected in sequence;

[0017] The second vision transformer model consists of a linear layer, a position embedding module, a multi-head self-attention module and a layer normalization module.

[0018] Furthermore, the first-layer DMAE framework and the second-layer DMAE framework are used to perform unsupervised learning on the first vision transformer model and the second vision transformer model respectively.

[0019] Furthermore, the first-layer DMAE framework uses the first vision transformer model as the student model and the teacher model. The student module is connected to the decoder and the sub-image blocks are loaded into the student module and the teacher module.

[0020] The second-layer DMAE framework uses the second vision transformer model as the student model and teacher model. The student module is connected to the decoder, and the features output by the student model in the first-layer DMAE framework are input into the student module and teacher module in the second-layer DMAE framework.

[0021] Furthermore, the decoder consists of a linear layer, a position embedding module, a multi-head self-attention module, a layer normalization module and a linear layer connected in sequence.

[0022] Furthermore, the loss function L of the first-layer DMAE framework and the second-layer DMAE framework is DMAE It is expressed as:

[0023] L DMAE =λL contrastive +(1-λ)L reconstructive ;

[0024] Where λ represents the equilibrium parameter, L contrastive is the cross entropy loss between the student model and the teacher model, L reconstructive is the mean squared error loss between the decoder output and the student model input.

[0025] Furthermore, the cross entropy loss L between the student model and the teacher model is contrastive It is expressed as:

[0026]

[0027] Among them, B represents the batch size, x i Represents the teacher model and student model input, p(x i ) represents the output of the teacher model, q(x i ) represents the output of the student model, and τ represents the temperature parameter.

[0028] Furthermore, the mean square error loss L between the decoder output and the student model input is reconstructive It is expressed as:

[0029]

[0030] Among them, B represents the batch size, s(x i ) represents the decoder output, x i Represents the input of the student model.

[0031] Furthermore, the classifier consists of a linear layer, a Relu activation function, a dropout layer, and a linear layer connected in sequence.

[0032] The WSI image classification system based on layered deep learning provided by the present invention has the following beneficial technical effects:

[0033] (1) The present invention designs a hierarchical deep neural network to learn WSI image features step by step, so that the model can focus on local and global features at the same time, and can extract rich high-quality feature representations from the image, so that only slice-level annotation is required, and more refined annotation is not required, which reduces the requirements for the data set and significantly reduces the annotation workload and cost;

[0034] (2) An unsupervised learning framework was introduced, which enabled the model to learn WSI image features even when the dataset lacked effective labels. A small number of samples were used to train the classifier, which greatly expanded the data space available for model training. At the same time, the model was constructed using a self-attention mechanism, which could effectively capture long-distance dependencies in pathological images and improve the accuracy of classification results.

[0035] (3) Not only can the classification prediction results of WSI images be obtained, but also the corresponding attention heat map can be obtained, highlighting the areas where lesions may occur for reference by pathologists, thus accelerating the diagnosis process; BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the structure of a WSI image classification system based on layered deep learning provided by an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the preprocessing process of a WSI image provided in an embodiment of the present invention, wherein (a) is loading of a sub-image block of a WSI image, (b) is an image after the WSI image is converted to the HSV space and subjected to median blurring and binarization processing, (c) is contour extraction, (d) is background area removal, (e) foreground area is divided into sub-image blocks, and (f) is information extraction and storage of sub-image blocks;

[0038] Figure 3 A schematic diagram of the structure of a WSI image classification model provided by an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of the structure of the first layer DMAE framework provided in an embodiment of the present invention;

[0040] Figure 5 Attention heat maps corresponding to the WSI images provided in the embodiments of the present invention, wherein (a) and (b) are the first WSI image and its corresponding attention heat maps, respectively, (c) and (d) are the second WSI image and its corresponding attention heat maps, respectively, and (e) and (f) are the third WSI image and its corresponding attention heat maps, respectively. Implementation

[0042] The specific implementation modes of the present invention are further explained in detail below by describing the embodiments with reference to the accompanying drawings, so as to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

[0043] Figure 1 A schematic diagram of the structure of a WSI image classification system based on hierarchical deep learning provided in an embodiment of the present invention. For ease of explanation, only the part related to the embodiment of the invention is shown. The system includes:

[0044] An input unit, a processor and a display unit connected in sequence;

[0045] The sample library is connected to the processor in communication, wherein the processor is integrated with a multi-layer deep learning WSI image classification model;

[0046] Phase 1: The WSI image classification model is trained using samples in the sample library until the recognition accuracy of the WSI image classification model reaches the set standard, and the training of the WSI image classification model is completed;

[0047] The second stage: the WSI image to be classified is input into the trained WSI image classification model through the input unit, and the WSI image classification model outputs the classification result of the WSI image to be classified and displays it through the display unit.

[0048] In an embodiment of the present invention, the system further includes:

[0049] The image preprocessing module is used to receive the WSI image, divide the foreground area in the WSI image into several sub-image blocks, extract the information of each sub-image module, the information of the sub-image module includes the image identifier of the WSI image and the coordinates in the WSI image, and store the sub-image blocks and their information in the sample library.

[0050] Receive the WSI image, convert the WSI image to the HSV space, extract the contours of the foreground area and the background area after median blur and binarization, remove the background area, and retain the foreground area; segment the foreground area into image blocks of a set size and without overlap, and then segment the image blocks into sub-image blocks of a set size and without overlap, extract and save the image identifiers of each sub-image block, including: the file identifier of the WSI image and the coordinates of the sub-image block in the WSI image. Figure 2 As shown, the WSI image is loaded, and after the foreground area is separated after median blur and binarization processing, the foreground area is first segmented into multiple 4096*4096 image blocks, and then each 4096*4096 image block is segmented into multiple 256*256 sub-image blocks. The segmented sub-image blocks are based on the file identifier of the WSI image and the coordinates of the sub-image blocks in the WSI image, and their coordinates are stored as h5 files.

[0051] Figure 3 The schematic diagram of the structure of the WSI image classification model provided in the embodiment of the present invention shows only the part related to the embodiment of the present invention for the convenience of description. The WSI image classification model includes:

[0052] The first vision transformer model and the second vision transformer model are connected in sequence,

[0053] The output of the second vision transformer model is connected to the classifier;

[0054] The first vision transformer model consists of a patch embedding module, a pos embedding module, a multi-head self-attention module, and a layer norm module connected in sequence.

[0055] The second vision transformer model consists of a linear layer, a position embedding module (posembed), a multi-head self-attention module (Multi head self-attention) and a layer normalization module (layer norm).

[0056] In the embodiment of the present invention, the classifier is composed of a linear layer, a Relu activation function, a dropout layer and a linear layer connected in sequence.

[0057] In the embodiment of the present invention, the first vision transformer model and the second vision transformer model use the DMAE framework for unsupervised learning, and the classifier performs supervised learning through samples, and annotates the collected WSI images of the specified part (breast) as normal or abnormal, thereby forming samples.

[0058] In an embodiment of the present invention, the present invention adopts a first-layer DMAE framework and a second-layer DMAE framework to perform unsupervised learning on the first vision transformer model and the second vision transformer model respectively, wherein:

[0059] The first-layer DMAE framework uses the first vision transformer model as the student model and the teacher model. The student module is connected to the decoder and the sub-image blocks are loaded into the student module and the teacher module.

[0060] The second-layer DMAE framework uses the second vision transformer model as the student model and the teacher model. The student module is connected to the decoder, and the features output by the student model in the first-layer DMAE framework are input into the student module and the teacher module in the second-layer DMAE framework.

[0061] The decoder consists of a linear layer, a position embedding module, a multi-head self-attention module, a layer normalization module, and a linear layer, which are connected in sequence.

[0062] In the embodiment of the present invention, the first-layer DMAE framework receives an image block input of 256*256 size and outputs an image category label CLS token. The output of the first layer is continuously sent to the second-layer DMAE framework in batches of 256 to obtain an actual 4096*4096 range image block category label output, and finally obtain a series of CLS tokens representing WSI image features; specifically, the first-layer DMAE framework is as follows: Figure 4 As described above, for the input sub-image block, it is first handed over to the first-layer DMAE framework for self-supervised learning. When its training converges, its weight is frozen and the update is stopped. Then the CLS token output by this layer is aggregated and passed to the second-layer DMAE framework for self-supervised learning. This layered learning framework allows the model to process only a part of the data at a time, reducing the training pressure. It also allows the model to focus on local and global features at the same time.

[0063] During training, the input is passed to the student and teacher models at the same time, and the cross entropy loss L of the student model and the teacher model is calculated. contrastive , expressed as:

[0064]

[0065] Among them, B represents the batch size, x i Represents the teacher model and student model input, p(x i ) represents the output of the teacher model, q(x i ) represents the output of the student model, τ represents the temperature parameter; then the output of the student model is input to the decoder for restoration, and the mean square error loss L between the output of the decoder and the input of the student model is calculated reconstructive , expressed as:

[0066]

[0067] Among them, B represents the batch size, s(x i ) represents the decoder output, x i Represents the input of the student model. The mean square error loss plus the cross entropy loss is the final loss L of the framework. DMAE , expressed as:

[0068] L DMAE =λL contrastive +(1-λ)L reconstructive;

[0069] Among them, λ represents the balance parameter. The smaller λ is, the faster the model learns, and the larger λ is, the stronger the model's anti-interference ability is. During inference, only the student model is used. Using the above self-supervised learning method, a large number of unlabeled WSI images can be effectively used for learning.

[0070] In the embodiment of the present invention, the sub-image loading process is specifically as follows:

[0071] The corresponding sub-image block data is read from the corresponding WSI image according to the coordinates of the sub-image block in the WSI image, instead of storing the segmented image blocks in advance, thus avoiding occupying a large amount of storage space and reducing frequent disk access operations.

[0072] In an embodiment of the present invention, the WSI image classification model further includes:

[0073] The attention matrix acquisition module is connected to the multi-head self-attention module in the second vision transformer model.

[0074] Fusion unit, connected to the attention matrix acquisition module;

[0075] The attention matrix acquisition module obtains the attention matrix of the multi-head self-attention module in the second vision transformer model - removes a certain proportion of low-attention areas according to the threshold, reconstructs the attention matrix into the WSI image size, and interpolates the missing areas;

[0076] The fusion unit draws different colors at the corresponding positions according to the attention size in the attention matrix, superimposes it on the WSI image, and outputs an attention heat map. The attention heat map can cover the areas where lesions may occur in the WSI image, which can help doctors quickly locate and reduce the diagnosis time. Figure 5 As shown in the figure, three WSI images are selected and the attention heat map output by the WSI image classification model is output.

[0077] The present invention has been described exemplarily. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A WSI image classification system based on hierarchical deep learning, characterized in that: The system comprises: An input unit, a processor and a display unit connected in sequence; A sample library is connected to the processor in communication, wherein the processor is integrated with a WSI image classification model; Phase 1: The WSI image classification model is trained using samples in the sample library until the recognition accuracy of the WSI image classification model reaches the set standard, and the training of the WSI image classification model is completed; The second stage: the WSI image to be classified is input into the trained WSI image classification model through the input unit, and the WSI image classification model outputs the classification result of the WSI image to be classified and displays it through the display unit.

2. The WSI image classification system based on hierarchical deep learning as claimed in claim 1, characterized in that: The system further comprises: The image preprocessing module is used to receive the WSI image, divide the foreground area in the WSI image into several sub-image blocks, extract the information of each sub-image module, including the image identifier of the WSI image and the coordinates in the WSI image, and store the sub-image blocks and their information in the sample library.

3. The WSI image classification system based on hierarchical deep learning as claimed in claim 1, characterized in that: The WSI image classification model includes: The first vision transformer model and the second vision transformer model are connected in sequence, The output of the second vision transformer model is connected to the classifier; The first vision transformer model consists of an image embedding module, a position embedding module, a multi-head self-attention module, and a layer normalization module connected in sequence; The second vision transformer model consists of a linear layer, a position embedding module, a multi-head self-attention module and a layer normalization module.

4. The WSI image classification system based on hierarchical deep learning as claimed in claim 3, characterized in that: The first-layer DMAE framework and the second-layer DMAE framework are used to perform unsupervised learning on the first vision transformer model and the second vision transformer model respectively.

5. The WSI image classification system based on hierarchical deep learning as claimed in claim 4, characterized in that: The first-layer DMAE framework uses the first vision transformer model as the student model and teacher model. The student module is connected to the decoder, and the sub-image blocks are loaded into the student module and the teacher module. The second-layer DMAE framework uses the second vision transformer model as the student model and teacher model. The student module is connected to the decoder, and the features output by the student model in the first-layer DMAE framework are input into the student module and the teacher module in the second-layer DMAE framework.

6. The WSI image classification system based on hierarchical deep learning as claimed in claim 4, characterized in that: The decoder consists of a linear layer, a position embedding module, a multi-head self-attention module, a layer normalization module and a linear layer connected in sequence.

7. The WSI image classification system based on hierarchical deep learning as described in claim 4 is characterized in that: The loss function L of the first layer DMAE framework and the second layer DMAE framework DMAE It is expressed as: THE DMAE =λL contrastive +(1-λ)L reconstructive ; Where λ represents the equilibrium parameter, L contrastive is the cross entropy loss between the student model and the teacher model, L reconstructive is the mean squared error loss between the decoder output and the student model input.

8. The WSI image classification system based on layered deep learning according to claim 7 is characterized in that: The cross entropy loss L between the student model and the teacher model contrastive It is expressed as: Among them, B represents the batch size, x i Represents the teacher model and student model input, p(x i ) represents the output of the teacher model, q(x i ) represents the output of the student model, and τ represents the temperature parameter.

9. The WSI image classification system based on layered deep learning according to claim 7, characterized in that: The mean square error loss L between the decoder output and the student model input reconstructive It is expressed as: Among them, B represents the batch size, s(x i ) represents the decoder output, x i Represents the input of the student model.

10. The WSI image classification system based on hierarchical deep learning as claimed in claim 3, characterized in that: The classifier consists of a linear layer, a Relu activation function, a dropout layer, and a linear layer connected in sequence.

Images