Histopathology full-slice image analysis system and method based on double-branch cooperative training deep network model

Abstract

The invention belongs to the field of computer vision, and particularly relates to a histopathology full-slice image analysis system and method based on a double-branch cooperative training deep network model, and the method comprises the steps: obtaining a histopathology full-slice image data set with an image level label, and carrying out the segmentation preprocessing; block image coding based on a pathology basic large model; establishing a double-branch cooperative training deep network model based on the slice map; using a histopathology full-slice sample data set with an image level label to train a network model in an off-line manner; and predicting the histopathology full-slice image by using the trained network model, and outputting an analysis result of the full-slice image. According to the method, the correlation between the full-slice images is modeled by introducing the slice images to assist the deep network model, the defect that the correlation between the full slices is neglected in an existing histopathology full-slice image analysis method can be made up, and the histopathology full-slice analysis precision of the deep network model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to the technical field of histopathology whole-slice image analysis, and specifically to a histopathology whole-slice image analysis system and method based on a dual-branch collaborative training deep network model. Background Art

[0002] Traditional histopathology full-slice image analysis requires professional pathologists to view full-slice images under a microscope, find some fields of interest, and then use pathology expertise to analyze the tissue morphology under the fields of view, and finally draw a diagnosis conclusion of the lesion at the slice level. This traditional method has many problems. Due to the consideration of diagnostic accuracy, the number of films that doctors read every day is limited. With the promotion of cancer screening, the shortage of pathologists is increasing, and doctors are under great pressure to read films. At the same time, the accuracy of pathology full-slice analysis is closely related to the experience of the doctor who reads the film, and will also be affected by the subjective views and physical fatigue of the doctor who reads the film at that time. With the development of computer technology, it will become a trend to combine computers and artificial intelligence methods to assist doctors in the analysis of pathology full slices, which will help relieve the work pressure of pathologists and improve the quality of diagnosis, and has broad application prospects.

[0003] In recent years, with the continuous development and improvement of slice scanner technology, computer hardware and artificial intelligence algorithms, the development of deep learning in image processing, especially in recognition and classification, has become more and more efficient, and the application of graph convolutional neural networks in image processing has become more and more extensive. At the same time, due to the flexibility and efficiency of its composition strategy, it has also shown strong competitiveness in pathological image classification. Using deep learning methods based on graph convolutional neural networks for medical pathological image classification can not only reduce the burden on medical workers, but also improve diagnostic efficiency and improve the medical health level of the general public.

[0004] At present, the mainstream methods of deep learning-related histopathology whole-slice image analysis still focus on the correlation between block images within the whole slice, ignoring the possible correlation between whole slices. The commonalities and differences between whole slices will also have considerable contributions to downstream tasks. Based on this consideration, this application designs a histopathology whole-slice image analysis system and method based on a dual-branch collaborative training deep network model, which further improves the accuracy of these methods based on the past multi-instance learning paradigm method. Summary of the invention

[0005] The purpose of the present invention is to provide a histopathology whole-slice image analysis system and method based on a dual-branch collaborative training deep network model, so as to quickly and accurately realize histopathology whole-slice image analysis (including downstream tasks such as cancer subtype classification, cancer grade prediction, and patient survival prediction based on whole slices), and to make up for the deficiency of existing histopathology whole-slice image analysis methods that ignore the correlation between whole slices, and to improve the analysis accuracy of deep network models for histopathology whole slices.

[0006] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0007] A method for analyzing whole-slice images of histopathology based on a dual-branch collaborative training deep network model comprises the following steps:

[0008] Step 1: Obtain a tissue pathology full-slice image dataset D with image category labels and perform preprocessing, divide the image dataset D into blocks and remove the block images without tissue areas to obtain the divided full-slice image B i , in represents the jth block image in the i-th full-slice image, and n represents the total number of block images divided from the i-th full-slice image;

[0009] Step 2: Use the pathological basis model f to analyze the whole slice image B i Encode the block images and generate d for each block image p The feature vector of Get the encoded full-slice image

[0010] Step 3: Establish a dual-branch collaborative training deep network model, wherein the dual branches include a graph learning branch and a multi-instance branch, and transform the encoded full-slice image B i As a sample input to the model for training, generate prediction p of multiple instance branches i and the prediction q of the graph learning branch i ;

[0011] Step 4: The prediction p of the multi-instance branch i and the prediction q of the graph learning branch i Substituting the loss value into the pre-constructed loss function, weighting the loss value and then back-propagating to train the deep network model, to obtain an optimized deep network model;

[0012] Step 5: Input the full slice image to be analyzed into the deep network model and take the prediction result q of the image learning branch i As a result of whole-slide image analysis.

[0013] Furthermore, step 1 specifically includes:

[0014] Step 1.1. Obtain a histopathology image dataset with image category labels:

[0015] D={(B1,Y1),(B2,Y2),...,(B i ,Y i ),...,(B N ,Y N )}, where B i represents the i-th histopathology full-slice sample image, Y i represents the category label of the corresponding i-th tissue pathology full-slice image, and N represents the number of full-slice image samples in the dataset;

[0016] Step 1.2: Divide the acquired full-slice image data set into several fixed-size block images according to the non-overlapping sliding window method, and obtain the divided full-slice sample image after removing the background block image without tissue.

[0017] Furthermore, the dual-branch collaborative training deep network model includes a multi-instance backbone network g MIL , node cache queue, adaptive graph building module, graph convolutional network for slice graph, graph learning branch classifier and multi-instance classifier μ MIL .

[0018] Furthermore, the multi-instance backbone network g MIL For converting the encoded full slice image X i Aggregate into full slice feature s i , specifically:

[0019]

[0020] The multi-instance classifier μ MIL Used to receive the full slice feature s i And generate the prediction p corresponding to the full slice i , specifically:

[0021] p i =μμ MIL (s i )

[0022] Among them, μ MIL With multi-instance backbone network g MIL Supporting facilities.

[0023] Furthermore, the node cache queue is composed of a storage space of length L, which is expressed as Used to store full-slice features s encountered by the network during trainingi .

[0024] Furthermore, the adaptive graph construction module includes two linear layers and a K nearest neighbor clustering module; during training, all full-slice features s stored in the node cache queue are i It is input into the adaptive graph construction module, mapped to the hidden layer through the linear layer and constructed as a hypergraph of the slice graph through K nearest neighbor clustering. Each node in the slice graph represents a full slice, and the node feature is the feature representation of the slice.

[0025] Furthermore, the graph convolution network for slice graphs includes two layers of hypergraph convolution layers and a channel attention module with centering operation, which is used to transfer information between nodes / full slices and update the feature representation of nodes / full slices, and the updated feature representation of nodes / full slices is sent to the graph learning branch classifier to output the prediction result q of the graph learning branch. i .

[0026] Furthermore, in step 4, the loss value is weighted and then back-propagated to train the deep network model to obtain an optimized deep network model, including:

[0027] The downstream task loss of multiple instance branches is obtained by using the loss functions shown in formula (1), formula (2) and formula (3). Downstream task loss of graph learning branch And the branch distillation loss between the two branches

[0028]

[0029] The three loss values ​​are weighted and then back-propagated to train the entire slice graph-based dual-branch collaborative training deep network model.

[0030] A histopathology whole-slice image analysis system based on a dual-branch collaborative training deep network model, used to implement the above-mentioned histopathology whole-slice image analysis method, comprising:

[0031] The whole slice image data acquisition module is used to acquire the tissue pathology whole slice image dataset D with image category labels and preprocess it to obtain the divided whole slice image B i ;

[0032] The encoding module is used to encode the whole slice image B using the pathological basis large model f i The block images are encoded to obtain the encoded full slice image X i ;

[0033] The full-slice image analysis module is used to establish a dual-branch collaborative training deep network model, train the model, and perform image analysis on the full-slice image to be analyzed, and generate prediction p for multiple instance branches. i and the prediction q of the graph learning branch i , take the prediction result q of the graph learning branch i As a result of whole-slice image analysis and output;

[0034] The loss optimization module is used to weight the loss value through the loss function and then back-propagate to train the deep network model to obtain an optimized deep network model.

[0035] The beneficial effects of the present invention are:

[0036] (1) The present invention constructs a slice diagram to explicitly establish an information path between the feature representations of the whole slice, thereby making up for the deficiency of the existing histopathology whole slice image analysis method that ignores the correlation between the whole slices, and can view the connection relationship between the whole slices through the visualized slice diagram, thereby ensuring the interpretability of the method;

[0037] (2) The present invention uses a node cache queue module to achieve end-to-end dual-branch (graph learning branch and multi-instance branch) collaborative training, improves the integrity of the method, and saves the computational and time overhead of the training model;

[0038] (3) The present invention can be applied to various existing histopathology whole-slice analysis methods based on the multi-instance learning paradigm, and can stably improve the accuracy of these methods while being highly generalizable;

[0039] (4) The present invention can be applied to various downstream tasks of whole-slice analysis of histopathology, including cancer subtype classification, cancer grading, patient survival prediction, and gene mutation prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the analysis method of the present invention;

[0041] Figure 2 It is a dual-branch collaborative training network module diagram of the present invention;

[0042] Figure 3 This is a training sample image of the tissue pathology whole-slice image of the present invention. DETAILED DESCRIPTION

[0043] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0044] Example 1

[0045] like Figure 1 As shown, in this embodiment, a dual-branch collaborative training method for histopathology full-slice image analysis based on slice graph is proposed, which mainly uses the graph structure to model the correlation between full-slice images, combines with the graph convolution network, constructs the information path between the full-slice feature representations, and then combines the knowledge distillation technology to migrate the information between the block images in the full slice concerned on the multi-instance branch to the graph branch, and finally realizes the reasoning of various downstream tasks with less computational complexity by relying only on the graph branch, including the following steps:

[0046] Step 1: Obtain a tissue pathology full-slice image dataset D with image category labels and perform preprocessing to obtain a full-slice image B after segmentation and removal of the background area without tissue i , in represents the jth block image in the i-th full-slice image, and n represents the total number of block images divided from the i-th full-slice image. In practice, n may be different for different full-slice images;

[0047] Step 1.1: Obtain a histopathology image dataset with image category labels:

[0048] D={(B1,Y1),(B2,Y2),...,(B i ,Y i ),...,(B N ,Y N )}, where B i represents the i-th histopathology full-slice sample image, Y i represents the 1-hot category label of the corresponding i-th histopathology full slide image, and N represents the number of image samples in the dataset;

[0049] Step 1.2: Divide the acquired full-slice image data set into several fixed-size block images according to the non-overlapping sliding window method, and obtain the divided full-slice sample image after removing the background block image without tissue.

[0050] Combination Figure 3 The dataset used in this example is for non-small cell carcinoma in lung cancer. The downstream task corresponding to the label is cancer subtype classification. It contains two types of full-slice images: 467 lung adenocarcinomas (LUAD) and 477 lung squamous cell carcinomas (LUSC). The full slices are divided into block images of 256 pixels × 256 pixels at a magnification of 20×, as shown in Figure 3As shown, this is a training sample diagram of a whole-slice image of histopathology. The labels of all data are confirmed by professional pathologists. 70% of all data are used as training sets and 30% of the data are used for testing.

[0051] Step 2: Use the pathological basis model f to encode the block images and generate d for each block image. p The feature vector of Get the encoded full-slice image T represents matrix transpose; the pathological basis model used in this embodiment is PLIP, and only the encoder of its visual modality is used to encode the feature vector dimension d of the block image p =512.

[0052] Step 3: Establish a dual-branch collaborative training deep network model based on slice graph, which includes a multi-instance backbone network, a multi-instance classifier, a node cache queue, an adaptive graph construction module, a slice graph-oriented graph convolutional network, and a graph learning branch classifier, a total of 6 modules, such as Figure 2 As shown; the encoded full slice image X described in step 2 i As a sample input into the deep network model for training, the prediction p of the multi-instance branch is finally generated from the two branches. i And the prediction q of the slice graph learning branch i ;

[0053] Step 3.1: The multi-instance backbone network g MIL and its matching multi-instance classifier μ MIL Any existing histological pathology whole-slice analysis network based on multi-instance learning can be used, and it is only required that the network has the ability to generate feature representations of the whole slice; the multi-instance backbone network and its supporting multi-instance classifier used in this embodiment are taken from ABMIL: its multi-instance backbone network includes a linear layer and an attention module, and its multi-instance classifier also includes only a linear layer. The specific operation of the multi-instance backbone network is as follows:

[0054] H i =ReLU(X i W1),

[0055] a i =SofTmax(ReLU(H i W2)W3),

[0056] s i =a i H i ,

[0057] Among them, H iis the intermediate layer output of the multi-instance backbone network, representing the intermediate state of the block image feature vector after a linear layer transformation. is the learnable weight matrix of the linear layer, The superscript represents the shape of the matrix. The linear layer bias is omitted in the formula to keep it simple. ReLU represents the rectified linear unit activation function. i For ABMIL's attention score, That is, the feature representation of the corresponding full slice i generated by ABMIL; in this embodiment, d s =500; The specific operation of the multi-instance classifier is as follows:

[0058] p i =s i W4,

[0059] in is the learnable weight of the linear layer in the classifier, C is the number of categories of the downstream task, in this embodiment, C=2, p i This is the prediction of multiple instance branches.

[0060] Step 3.2: The node cache queue node is composed of a storage space of length L, represented as Used to store the full slice feature representation s encountered by the network during training i , is dynamically updated during training, and the node cache queue is locally saved together with the model weight after training. In this embodiment, L=3072, and the queue update strategy is a first-in-first-out strategy, that is, the current training sample will be kept at the front of the queue.

[0061] Step 3.3: The adaptive graph construction module includes two linear layers and a K-nearest neighbor clustering module. During training, all samples stored in the node cache queue are input into this module, and its feature representation is mapped to the hidden layer through the linear layer and the hypergraph of the slice graph is constructed through K-nearest neighbor clustering. Each node in the graph represents a full slice sample, and the specific operations are as follows:

[0062] ε=KNN(ReLU(B N W5)W6),

[0063]

[0064] in is the learnable weight in the linear layer, KNN represents K nearest neighbor operation, and each node and the K nodes closest to the node (including itself) together form a hyperedge; in this embodiment, K=12.

[0065] Step 3.4, the graph convolution network for the slice graph includes two hypergraph convolution layers and a channel attention module with a centering operation for information transfer between nodes / full slices and updating the feature representation of nodes / full slices; in this embodiment, the hypergraph convolution layer does not change the dimension of the node feature, and its operation can be expressed as follows:

[0066]

[0067]

[0068] in represents the hypergraph node features of the slice graph before the i-th hypergraph convolutional layer, represents the updated hypergraph node features after the first hypergraph convolutional layer, HGC represents the hypergraph convolution operation, and Concat represents the matrix concatenation operation. Finally, the node feature U is obtained after concatenation along the channel. Then, the information aggregation of different graph convolution scale information is balanced through the channel attention with centering operation. The specific operations are as follows:

[0069]

[0070]

[0071] in is the learnable weight in the linear layer, Mean represents the mean operation, and the updated node / full slice feature representation is obtained. Then the updated node / full slice feature representation is sent to the graph learning branch classifier for prediction of downstream tasks. The classifier also contains only one linear layer; the specific operations are as follows:

[0072]

[0073]

[0074] in is the learnable weight in the linear layer, and u represents the current training sample (in the queue The forefront ), and finally output the prediction result q of the graph learning branch i .

[0075] Step 4: i and the q i Substitute into the pre-constructed loss functions shown in equations (1), (2), and (3), and calculate the downstream task losses of the multi-instance branches respectively: Downstream task loss of slice graph learning branch And the knowledge distillation loss between branches (branch distillation loss) The three loss values ​​are weighted and then back-propagated to train the entire slice graph-based dual-branch collaborative training deep network model.

[0076]

[0077] Step 5: Inference phase, the full slice image to be analyzed is input into the deep network model, and the prediction result q of the image learning branch is taken i as the prediction result of the model.

[0078] A histopathology whole-slice image analysis system based on a dual-branch collaborative training deep network model, used to implement the above-mentioned histopathology whole-slice image analysis method, comprising:

[0079] The whole slice image data acquisition module is used to acquire the tissue pathology whole slice image dataset D with image category labels and preprocess it to obtain the divided whole slice image B i ;

[0080] The encoding module is used to encode the whole slice image B using the pathological basis large model f i The block images are encoded to obtain the encoded full slice image X i ;

[0081] The full-slice image analysis module is used to establish a dual-branch collaborative training deep network model, train the model, and perform image analysis on the full-slice image to be analyzed, and generate prediction p for multiple instance branches. i and the prediction q of the graph learning branch i , take the prediction result q of the graph learning branch i As a result of whole-slice image analysis and output;

[0082] The loss optimization module is used to weight the loss value through the loss function and then back-propagate to train the deep network model to obtain an optimized deep network model.

[0083] An electronic device includes a processor and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image analysis method as described above.

[0084] A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned image analysis method.

[0085] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0086] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0087] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0088] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for analyzing whole-slice images of histopathology based on a dual-branch collaborative training deep network model, characterized in that: The following steps are involved: Step 1: Obtain a tissue pathology full-slice image dataset D with image category labels and perform preprocessing, divide the image dataset D into blocks and remove the block images without tissue areas to obtain the divided full-slice image B i , in represents the jth block image in the i-th full-slice image, and n represents the total number of block images divided from the i-th full-slice image; Step 2: Use the pathological basis model f to analyze the whole slice image B i Encode the block images and generate d for each block image p The feature vector of Get the encoded full-slice image Step 3: Establish a dual-branch collaborative training deep network model, wherein the dual branches include a graph learning branch and a multi-instance branch, and transform the encoded full-slice image X i As a sample input to the model for training, generate prediction p of multiple instance branches i and the prediction q of the graph learning branch i ; Step 4: The prediction p of the multi-instance branch i and the prediction q of the graph learning branch i Substituting the loss value into the pre-constructed loss function, weighting the loss value and then back-propagating to train the deep network model, to obtain an optimized deep network model; Step 5: Input the full slice image to be analyzed into the deep network model and take the prediction result q of the image learning branch i As a result of whole-slide image analysis.

2. The histopathological whole-slice image analysis method according to claim 1, characterized in that: Step 1 specifically includes: Step 1.

1. Obtain a histopathology image dataset with image category labels: D={(B1,Y1),(B2,Y2),...,(B i ,Y i ),...,(B N ,Y N )}, where B i represents the i-th histopathology full-slice sample image, Y i represents the category label of the corresponding i-th tissue pathology full-slice image, and N represents the number of full-slice image samples in the dataset; Step 1.2: Divide the acquired full-slice image data set into several fixed-size block images according to the non-overlapping sliding window method, and obtain the divided full-slice sample image after removing the background block image without tissue.

3. The histopathological whole-slice image analysis method according to claim 1, characterized in that: The dual-branch collaborative training deep network model includes a multi-instance backbone network g MIL , node cache queue, adaptive graph building module, graph convolutional network for slice graph, graph learning branch classifier and multi-instance classifier μ MIL .

4. The histopathological whole-slice image analysis method according to claim 3, characterized in that: The multi-instance backbone network g MIL For converting the encoded full slice image X i Aggregate into full slice feature s i , specifically: The multi-instance classifier μ MIL Used to receive the full slice feature s i And generate the prediction p corresponding to the full slice i , specifically: p i =μ MIL (s i ) Among them, μ MIL With multi-instance backbone network g MIL Supporting facilities.

5. The histopathological whole-slice image analysis method according to claim 4, characterized in that: The node cache queue consists of a storage space of length L, expressed as Used to store full-slice features s encountered by the network during training i .

6. The histopathological whole-slice image analysis method according to claim 5, characterized in that: The adaptive graph construction module includes two linear layers and a K-nearest neighbor clustering module; All full-slice features s stored in the node cache queue during training i It is input into the adaptive graph construction module, mapped to the hidden layer through the linear layer and constructed as a hypergraph of the slice graph through K nearest neighbor clustering. Each node in the slice graph represents a full slice sample, and the node feature is the feature representation of the slice.

7. The histopathological whole-slice image analysis method according to claim 6, characterized in that: The graph convolutional network for slice graphs includes two layers of hypergraph convolutional layers and a channel attention module with centering operation, which is used to transfer information between nodes / full slices and update the feature representation of nodes / full slices, and send the updated node / full slice features to the graph learning branch classifier to output the prediction result q of the graph learning branch. i .

8. The histopathological whole-slice image analysis method according to claim 1, characterized in that: In step 4, the loss value is weighted and then back-propagated to train the deep network model to obtain an optimized deep network model, including: The downstream task loss of multiple instance branches is obtained by using the loss functions shown in formula (1), formula (2) and formula (3). Downstream task loss of graph learning branch And the branch distillation loss between the two branches The three loss values ​​are weighted and then back-propagated to train the entire slice graph-based dual-branch collaborative training deep network model.

9. A histopathological whole-slice image analysis system based on a dual-branch collaborative training deep network model, used to implement the histopathological whole-slice image analysis method according to any one of claims 1 to 8, characterized in that: include: The whole slice image data acquisition module is used to acquire the tissue pathology whole slice image dataset D with image category labels and preprocess it to obtain the divided whole slice image B i ; The encoding module is used to encode the whole slice image B using the pathological basis large model f i The block images are encoded to obtain the encoded full slice image X i ; The full-slice image analysis module is used to establish a dual-branch collaborative training deep network model, train the model, and perform image analysis on the full-slice image to be analyzed, and generate prediction p for multiple instance branches. i and the prediction q of the graph learning branch i , take the prediction result q of the graph learning branch i As a result of whole-slice image analysis and output; The loss optimization module is used to weight the loss value through the loss function and then back-propagate to train the deep network model to obtain an optimized deep network model.

Images