A method and electronic device for generating panoramic pathology scanning image descriptors

Abstract

The present invention relates to a panoramic pathology scanning image descriptor generation method and electronic device. The present invention labels the panoramic pathology scanning image by category and pathological state, assigns a global pathology label, and cuts it into a number of color blocks at equal intervals, and obtains the descriptor of the color block through a computer vision model; the descriptor matrix composed of the descriptors of all color blocks of a complete panoramic pathology scanning image and its global pathology label is regarded as an instance, and all instances are input into a sparse global attention model framework to train a panoramic pathology scanning image classification model; the panoramic pathology scanning image classification model is used to obtain the color block's contribution score to the global pathology label of the entire panoramic pathology scanning image, and the global descriptor is obtained by dot multiplication with the descriptor matrix of the color block. The present invention integrates the pathology image deep learning descriptor extraction model with the sparse global attention method to achieve the extraction of global descriptors for panoramic pathology scanning images from different data sources.

Description

Technical Field

[0001] The present invention belongs to the field of image data information mining, and in particular relates to a method for generating a panoramic pathology scanning image descriptor and an electronic device. Background Art

[0002] Pathology provides an objective basis for the diagnosis, classification and determination of treatment plans for diseases, as well as the judgment of disease progression, prognosis and efficacy. As a bridge between basic medicine and clinical medicine, pathology plays an extremely important role in the diagnosis and treatment of diseases (especially tumors). With the rapid development of the medical big data industry, various medical institutions have stored a large number of panoramic pathology scanning images (WSI). These pathology image data are rich in a large amount of data information and can provide valuable objective basis for clinical diagnosis and precision medical decision-making. However, this type of data storage volume is large and it is difficult to retrieve and classify as quickly and efficiently as structured data.

[0003] At present, great progress has been made in data mining of panoramic pathology scan images through artificial intelligence. However, most of the existing pathology artificial intelligence analysis models are often only targeted at a single task, lacking a framework that can simultaneously perform segmentation, classification, and visual retrieval around pathology images. At the same time, due to the huge size of panoramic pathology scan images, existing analysis schemes remain at the color block level, making it difficult to fully utilize the global information of the image. Image descriptors are a quantitative processing result of image content, and their related algorithms can be well applied to image visual detection, classification, segmentation, recognition, and retrieval tasks by abstractly representing the image. Therefore, obtaining global image descriptors for panoramic pathology scan images is the primary solution to solve the bottleneck of pathology image data mining.

[0004] Traditional deep networks are currently the main method for obtaining image descriptors. This type of method only represents a single content, does not fully describe the information of key points, and cannot capture the deep semantic information of huge panoramic pathology scan images. Therefore, it is impossible to complete the panoramic pathology scan image retrieval task using only the descriptors of the fully connected layers of the deep network. Summary of the invention

[0005] The purpose of the present invention is to provide a panoramic pathology scanning image descriptor generation method and electronic device, which combines a computer vision model with a sparse global attention mechanism to extract global-level image descriptors for panoramic pathology scanning images (WSI) of different sources and formats, and provide a structured data basis for subsequent pathology diagnosis and text report tasks.

[0006] The present invention is implemented by the following technical scheme: A method for generating a panoramic pathology scanning image descriptor comprises the following steps:

[0007] Step 1: Obtain paraffin HE-stained sections of various types and pathological conditions, convert them into panoramic pathological scanning images (WSI) and perform quality control;

[0008] Step 2: Pathologists annotate the panoramic pathology scan image (WSI) by category and pathological status and assign a global pathology label;

[0009] Step 3: Cut the panoramic pathology scan image into several color blocks at equal distances, perform quality control and assign spatial information to obtain a color block set;

[0010] Step 4: Input the color block set into the computer vision model framework for training, output the computer vision model after parameter adjustment, and perform the computer vision model effectiveness test;

[0011] Step 5: Input the color block set obtained in step 3 into the selected computer vision model in the fully connected layer to obtain the descriptor matrix of the color block;

[0012] Step 6: The descriptor matrix composed of the descriptors of all color blocks of a complete panoramic pathology scan image and its global pathology label are regarded as an instance, and all instances are input into the sparse global attention model framework to train a panoramic pathology scan image classification model based on sparse global attention that can achieve panoramic pathology scan image level classification;

[0013] Step 7: Obtain the global pathology label contribution score of the color block to the entire panoramic pathology scan image through the panoramic pathology scan image classification model. The descriptor matrix of the color block is multiplied by the global pathology label contribution score and then averaged to generate a global descriptor at the panoramic pathology scan image level of a fixed size.

[0014] Preferably, in step three, the panoramic pathology scan image is subjected to background removal processing, the tissue region image in the panoramic pathology scan image is retained, and the image is equidistantly divided into a number of color blocks with the same physical distance according to the highest resolution, and then quality control is performed, and spatial position information is given to each color block to obtain a color block set.

[0015] Preferably, in step 4, the color block set is divided into a training set and a validation set, wherein the training set is input into the computer vision model to fine-tune it, and the validation set is used to verify the performance of the computer vision model.

[0016] Preferably, the computer vision model is a ResNet50 model.

[0017] Preferably, in step five, the color block set obtained in step three is color standardized, resized to a specified size, input into a fine-tuned computer vision model, image information is captured through a feature extraction module, and the final fully connected layer output is used as a descriptor of the color block to obtain the descriptors of the color blocks of the entire panoramic pathology scan image, thereby forming a descriptor matrix of the color blocks.

[0018] Preferably, in step six, the descriptor of each color block of a panoramic pathology scan image is regarded as an instance, and all instances and corresponding global pathology labels are input into a panoramic pathology scan image classification model based on a sparse global attention framework according to the actual pathology task. The classification performance of the panoramic pathology scan image classification model is tested using an external validation set to obtain the global attention distribution.

[0019] Preferably, the sparse global attention model framework comprises an initialized fully connected layer, an attention network, and a classifier. The sparse global attention model regards a complete panoramic pathology scan image and its global label as an instance, and the descriptors of all color blocks of the panoramic pathology scan image as tokens that constitute the instance. The parameters are adjusted through training and the contribution of each token to the global classification of the panoramic pathology scan image is calculated. The global descriptor is output in the fully connected layer, and the predicted label of the panoramic pathology scan image is output in the classifier.

[0020] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the computer program executes the method for generating a panoramic pathology scanning image descriptor.

[0021] The present invention also provides a computer-readable storage medium, which stores computer instructions for the panoramic pathology scanning image descriptor generation method.

[0022] The sparse global attention mechanism can process large data sets, help the model capture detailed information of the image more effectively, and only focus on a part of the elements in the sequence, usually the most important or most relevant elements to the current task, ignoring unimportant information. The present invention combines the deep learning descriptor extraction model of pathological images with the sparse global attention method to extract global descriptors for panoramic pathological scanning images from different data sources. This strategy can meet various pathological tissue classification tasks and provide a basis for related technologies such as tissue microenvironment exploration and intelligent pathological medical report generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the present invention.

[0024] Figure 2 A sample diagram for fine-tuning the ResNet50 model.

[0025] Figure 3 To verify the performance of fine-tuning the ResNet50 model.

[0026] Figure 4 Schematic diagram of the sparse global attention model.

[0027] Figure 5 Figure 2 is the operating characteristic curve of the panoramic pathology scan image classification model.

[0028] Figure 6 Attach a manually annotated image to the original image;

[0029] Figure 7 To manually annotate the mask;

[0030] Figure 8 This is the attention distribution diagram.

[0031] Fig. 9 Make a classification tree analysis diagram for high-dimensional space dimensionality reduction clustering of global descriptors.

[0032] Fig.10 Clustering graph for global descriptors. DETAILED DESCRIPTION

[0033] In order to more clearly understand the above-mentioned purpose, descriptors and advantages of the present invention, the present invention is further described in detail below in conjunction with embodiments. The specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] Reference Figure 1 , a method for generating a panoramic pathology scanning image descriptor comprises the following steps:

[0035] Step 1: Obtain paraffin HE-stained sections of various types and pathological conditions, convert them into panoramic pathological scanning images (WSI) and perform quality control;

[0036] Step 2: Pathologists annotate the panoramic pathology scan image (WSI) by category and pathological status and assign a global pathology label;

[0037] Step 3: Cut the panoramic pathology scan image into several color blocks at equal distances, perform quality control and assign spatial information to obtain a color block set;

[0038] Step 4: Input the color block set into the computer vision model framework for training, output the computer vision model after parameter adjustment, and perform the computer vision model effectiveness test;

[0039] Step 5: Input the color block set obtained in step 3 into the selected computer vision model in the fully connected layer to obtain the descriptor matrix of the color block;

[0040] Step 6: The descriptor matrix composed of the descriptors of all color blocks of a complete panoramic pathology scan image and its global pathology label are regarded as an instance, and all instances are input into the sparse global attention model framework to train a panoramic pathology scan image classification model based on sparse global attention that can achieve panoramic pathology scan image level classification;

[0041] Step 7: Obtain the global pathology label contribution score of the color block to the entire panoramic pathology scan image through the panoramic pathology scan image classification model. The descriptor matrix of the color block is multiplied by the global pathology label contribution score and then averaged to generate a global descriptor at the panoramic pathology scan image level of a fixed size.

[0042] Furthermore, the paraffin HE stained sections of various categories and different pathological states collected in step one should be determined according to the specific goals. The paraffin HE stained sections of each category and different pathological state should come from different individuals and the number should not be less than 30. The source of the pathological tissue should be clear, without damage, folding, clear image, staining pollution and impurities, and digitized by a slice scanner and saved in image formats such as svs, tif, and png. In this embodiment, the goal is to establish a panoramic pathology scanning image survival global descriptor for breast cancer lymph node metastasis. Therefore, paraffin HE stained pathological tissue sections of breast lymph node cancer metastasis tissue and corresponding normal tissue are collected, and paraffin HE stained pathological tissue sections without tissue, damaged tissue overlap, and serious staining pollution are eliminated, and panoramic pathology scanning images with clear and good image quality are retained. A total of 661 paraffin HE stained sections were collected ( Figure 2 ), including 57 breast cancer images, 60 breast non-cancer images, 55 lung adenocarcinoma images, 57 lung adenocarcinoma images, 51 pancreatic cancer images, 52 pancreatic non-cancer images, 52 gastric adenocarcinoma images, 55 gastric adenocarcinoma images, 54 colon adenocarcinoma images, 60 colon adenocarcinoma images, 55 rectal adenocarcinoma images, and 53 rectal adenocarcinoma images. They were digitized into panoramic pathology scan images (Whole Slide Images, WSI) using a pathology slide scanner; at the same time, the public Camelyon 16 dataset was downloaded as a subsequent external validation set.

[0043] In this embodiment, a professional pathology expert is invited to annotate the tumor area on the panoramic pathology scan image in which the tumor is clearly present by using an annotation tool, and the annotation file is saved in the form of an annotation image in tif format.

[0044] Further, in step 3, the panoramic pathology scan image is subjected to background removal processing, the tissue region image in the panoramic pathology scan image is retained, and the image is divided into several color blocks with the same physical distance according to the highest resolution, and then quality control is performed, and the spatial position information of each color block is given to obtain a color block set. In this embodiment, the panoramic pathology scan image is divided into color blocks with an actual physical distance of 128μm*128μm with reference to the maximum resolution of the panoramic pathology scan image, and the cut color blocks are subjected to quality control. The quality control content includes: 1. Background removal, setting the three-channel threshold of the image, and removing the color blocks with pure white, pure black or close to pure white and pure black; 2. Size control, determining the cutting size with reference to the maximum resolution of the panoramic pathology scan image, assuming that the "40×" resolution uses 512 pixels*512 pixels, and the "20×" uses 256 pixels*256 pixels, and the color blocks with inconsistent sizes after segmentation are removed; 3. Clarity control, it is generally believed that when the clarity threshold is set at 150, the texture and color of the color block are still meaningful, and the color blocks with a clarity lower than 150 are removed. Finally, a total of 6,974,233 valid color blocks are obtained, which is the color block set.

[0045] Further, in step 4, the color block set is divided into a training set and a validation set, wherein the training set is input into the computer vision model to fine-tune it, and the validation set is used to verify the performance of the computer vision model. In this embodiment, the ResNet50 model framework in the computer vision model is selected for reference only, which can capture different degrees of detail features. Other computer vision models can be selected according to actual task needs, but the model should be adjusted according to the task parameters and the performance should meet the two-classification AUC-ROC above 0.95 and the multi-classification task AUC-ROC above 0.90.

[0046] In step 4 of this embodiment, since the annotated image and the corresponding panoramic pathology scan image are in equal proportion, the color block corresponding to the annotated area can be screened out from the numerous color blocks by using the annotated area position information, and a total of 311,243 breast cancer color blocks, 651,244 breast non-cancerous color blocks, 327,564 lung adenocarcinoma color blocks, 601,455 lung adenocarcinoma non-cancerous color blocks, 274,324 pancreatic cancer color blocks, 524,481 pancreatic non-cancerous color blocks, 299,572 gastric adenocarcinoma color blocks, 600,447 gastric non-cancerous color blocks, and 1,343 colon adenocarcinoma color blocks are screened out. There were 250,114 color blocks of colorectal adenocarcinoma, 642,202 color blocks of non-cancerous color of colon adenocarcinoma, 260,154 color blocks of rectal adenocarcinoma, and 579,564 color blocks of non-cancerous color of rectal adenocarcinoma, among which the non-tumor areas of panoramic pathology scan images (WSI) determined to be tumors were not included in the non-cancerous color blocks; then a random algorithm was used, and the random number of the random algorithm was set to '123', 50,000 color blocks of different states of the above-mentioned different tissues were selected to construct a training set, and a total of 2*50,000 color blocks entered the training queue, and 40,000 color blocks were selected to construct a test set, and a total of 2*50,000 color blocks entered the test queue.

[0047] In step 4 of this embodiment, the randomly created training set is input into the ResNet50 model for training, wherein the learning rate is set to 0.0001, the training and testing ratio within the ResNet50 model is 4:1, the number of color blocks entering the ResNet50 model in each batch is 1024, and the color blocks entering the ResNet50 model are resized to 224 pixels * 224 pixels * 3 channels. The training round is set to 20, and finally a fine-tuned ResNet50 model is obtained; the test set is used to verify the performance of the optimal fine-tuned ResNet50 model, and the AUC-ROC is the area under the ROC curve. The AUC-ROC value is 0.959, and the comparison results of the multi-classification predicted labels and the true label results are as follows Figure 3 ,The results show that the model can well capture the image details between ,color blocks of different tissue origins and different pathological states, and thus has ,excellent judgment ability for each color block.

[0048] In order to further verify the ability of the fine-tuned ResNet50 model to capture multi-dimensional details of pathological images and to demonstrate the subsequent generation process of global descriptors and the representativeness and high density of information in global descriptors, the public Camelyon 16 dataset was selected as an external verification of the full-process operation and application of the present invention. The Camelyon 16 dataset consists of 159 slices without metastasis and 111 slices with tumor metastasis and their corresponding annotations of breast cancer lymph node metastasis by professional pathologists.

[0049] Furthermore, in step five, the color block set obtained in step three is color standardized, resized to a specified size, input into the fine-tuned computer vision model, and captured through a feature extraction module. The output of the final fully connected layer is used as the descriptor of the color block to obtain the descriptor of the color block of the entire panoramic pathology scan image, thereby forming a descriptor matrix of the color block.

[0050] Furthermore, in step six, the descriptor of each color block of a panoramic pathology scan image is regarded as an instance. According to the actual pathology task, all instances and corresponding global pathology labels are input into a panoramic pathology scan image classification model based on a sparse global attention framework. The classification performance of the panoramic pathology scan image classification model is tested using an external validation set to obtain the global attention distribution.

[0051] The sparse global attention model framework described in this embodiment is built based on the principle of multiple instance learning. Figure 4 As shown in FIG. 1 , it includes an initialized fully connected layer, an attention network, and a classifier. The initialized fully connected layer compresses the descriptor matrix parameters of the input N*1024-dimensional color block to N*512 dimensions and then inputs the attention network. N is the number of instance color blocks. The attention network includes self-attention device I, self-attention device II, and self-attention weight calculator. The first self-attention device changes the descriptor matrix parameters of the color block from N*512 to N*256 through the fully connected layer, and uses the Tanh activation function to output N*256 data. The second self-attention device changes the descriptor matrix parameters of the block from N*512 to N*256 through the fully connected layer, and uses the Sigmoid activation function to output N*256 data. The self-attention weight calculator The weight calculator outputs N*256 data by performing a dot product of the N*256 data output by the first self-attention device and the N*256 data output by the second self-attention device, and then inputs the data into the fully connected layer of the self-attention weight calculator to output the classification contribution of each color block. The classification contribution of each color block is normalized by the softmax normalization function to obtain the global pathology label contribution score of each color block relative to the entire panoramic pathology scan image, and the 512-dimensional descriptor output by the initialized fully connected layer of each color block is multiplied by its corresponding global pathology label contribution score to obtain a modified N*512 color block descriptor matrix. Finally, the modified N*512 color block descriptor matrix is ​​added in the same column to obtain a 512-dimensional global descriptor. In essence, the global attention model regards a complete panoramic pathology scan image and its global label as an instance, and the descriptors of all color blocks of the panoramic pathology scan image as the tokens that make up the instance. It adjusts the parameters through training and calculates the contribution of each token to the global classification of the panoramic pathology scan image. It outputs the global descriptor in the fully connected layer and the predicted label of the panoramic pathology scan image in the classifier.

[0052] 60 slices without metastasis and 60 slices with tumor metastasis were randomly selected from the Camelyon 16 dataset. After these 120 slices were subjected to image segmentation and quality control processes, N color blocks were obtained for each slice. After being input into the fine-tuned ResNet50 model, no classification processing was performed, but a descriptor matrix with a length of N*1024 characters was directly output from the last fully connected layer. The descriptor matrix with a length of N*1024 characters of a slice was regarded as an instance. Then, the instances of the above 120 slice transformations and their corresponding slice labels were input into the sparse global attention model framework for attention training, in which the model random number was set to '123', the training and testing ratio within the model was 5:1, and the model training rounds were set to 10. Finally, a panoramic pathology scanning image classification model based on sparse global attention was obtained. The best model in the iterative process was the 9th iteration model, and its internal validation set AUC-ROC value was 0.9953.

[0053] The remaining slices of the Camelyon 16 dataset (159 slices without metastasis and 111 slices with tumor metastasis) were input into the trained panoramic pathology scan image classification model to obtain the category judgment, judgment probability, attention weight, and panoramic pathology scan image global descriptor (512 characters in length) of these slices. The panoramic pathology scan image classification model was verified using slices other than the training set as an external validation set, and the AUC-ROC value (area under the operating characteristic curve) was 0.9953 ( Figure 5 ), it is believed that the panoramic pathology scan image classification model has good accuracy in judging the global descriptor of panoramic pathology scan images. The attention weight calculated for each slice is used to draw a spatial heat map and compared with the annotations of breast cancer lymph node metastases by professional pathologists in the Camelyon 16 dataset. It is found that the attention focus area ( Figure 8 ) matches the marked area ( Figure 6 and Figure 7 ), so the focus of the sparse global attention model is consistent with the clinical and pathological diagnosis logic. The 512-character length of the panoramic pathology scan image global descriptor generated by all instances is subjected to high-dimensional space dimensionality reduction clustering analysis ( Fig. 9 and Fig.10 ), including Group 0-Group 6, further verified the ability of ResNet50 model to capture color block details and the panoramic pathology scanning image global descriptor after attention correction of the panoramic pathology scanning image classification model can well represent the slice, thereby realizing the structuring, enrichment and differentiation of panoramic pathology scanning image information.

[0054] In one embodiment, the present invention further provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the computer program executes the method for generating a panoramic pathology scanning image descriptor.

[0055] In one embodiment, the present invention further provides a computer-readable storage medium storing computer instructions for the panoramic pathology scanning image descriptor generation method.

[0056] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for generating a panoramic pathology scanning image descriptor, characterized in that: The following steps are involved: Step 1: Obtain paraffin HE-stained sections of various types and pathological conditions, convert them into panoramic pathological scanning images and perform quality control; Step 2: Annotate the panoramic pathology scan image by category and pathological status and assign a global pathology label; Step 3: Cut the panoramic pathology scan image into several color blocks at equal distances, perform quality control and assign spatial information to obtain a color block set; Step 4: Input the color block set into the computer vision model framework for training, output the computer vision model after parameter adjustment, and perform a computer vision model performance test; Step 5: Input the color block set obtained in step 3 into the selected computer vision model in the fully connected layer to obtain the descriptor matrix of the color block; Step 6: The descriptor matrix composed of the descriptors of all color blocks of a complete panoramic pathology scan image and its global pathology label are regarded as an instance, and all instances are input into the sparse global attention model framework to train a panoramic pathology scan image classification model based on sparse global attention that can realize panoramic pathology scan image level classification; the sparse global attention model framework includes an initialized fully connected layer, a self-attention network, and a classifier. The sparse global attention model regards a complete panoramic pathology scan image and its global label as an instance, and the descriptors of all color blocks of the panoramic pathology scan image are regarded as tokens constituting the instance. The parameters are adjusted through training and the contribution of each token to the global classification of the panoramic pathology scan image is calculated, and the global descriptor is output in the fully connected layer, and the predicted label of the panoramic pathology scan image is output in the classifier; Step 7: Obtain the global pathology label contribution score of the color block to the entire panoramic pathology scan image through the panoramic pathology scan image classification model. The descriptor matrix of the color block is multiplied by the global pathology label contribution score and then averaged to generate a global descriptor at the panoramic pathology scan image level of a fixed size.

2. The method for generating a panoramic pathology scanning image descriptor according to claim 1, characterized in that: In step three, the panoramic pathology scan image is subjected to background removal processing, the tissue area image in the panoramic pathology scan image is retained, and it is equidistantly divided into several color blocks with the same physical distance according to the highest resolution, and then quality control is performed. At the same time, spatial position information is given to each color block to obtain a color block set.

3. The method for generating a panoramic pathology scanning image descriptor according to claim 1, characterized in that: In step 4, the color patch set is divided into a training set and a validation set, wherein the training set is input into the computer vision model to fine-tune it, and the validation set is used to verify the effectiveness of the computer vision model.

4. The method for generating a panoramic pathology scanning image descriptor according to claim 1, wherein: The computer vision model is a ResNet50 model.

5. The method for generating a panoramic pathology scanning image descriptor according to claim 1, wherein: In step five, the color block set obtained in step three is color standardized, resized to a specified size, and input into the fine-tuned computer vision model. The image information is captured through the feature extraction module, and the output of the final fully connected layer is used as the descriptor of the color block to obtain the descriptor of the color block of the entire panoramic pathology scan image, forming a descriptor matrix of the color block.

6. The method for generating a panoramic pathology scanning image descriptor according to claim 1, wherein: In step six, the descriptor of each color block of a panoramic pathology scan image is regarded as an instance. According to the actual pathology task, all instances and corresponding global pathology labels are input into a panoramic pathology scan image classification model based on a sparse global attention framework. The classification performance of the panoramic pathology scan image classification model is tested using an external validation set to obtain the global attention distribution.

7. An electronic device, comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the processor executes the computer program, the computer program executes the panoramic pathology scanning image descriptor generation method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions for executing the method for generating a panoramic pathology scanning image descriptor according to any one of claims 1 to 6.

Images