Multi-color-domain pathological slide artifact detection method based on comparative learning

Abstract

The invention relates to the technical field of digital image processing, and discloses a multi-color-domain pathological slide artifact detection method based on comparative learning, which comprises the following steps of: 1, cutting all full-slice images into a plurality of sub-image blocks by utilizing a pathological full-slice cutting algorithm; 2, designing and realizing a first feature extractor; 3, for the pathological sub-image blocks screened out in the step 2, converting the RGB color domain image into HSV and LAB color domain images by using an OpenCV tool; 4, designing a second feature extractor, and extracting key features from the three color domain pathological images obtained in the step 3; 5, constructing a twin network structure, and introducing a joint loss function; and step 6, in a test stage, repeating the operations of the step 1 and the step 2, obtaining an original RGB color domain image, sending the obtained image into the second feature extractor trained in the step 4, processing the image by the classifier, and finally obtaining a classification prediction result of the pathological image block.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and more specifically to a multi-color domain pathology film artifact detection method based on contrast learning. Background Art

[0002] In recent years, breast cancer, colorectal cancer and lung cancer have all shown a significant upward trend in my country. Clinically, tumors can be detected through a variety of ways, such as X-ray examination, color Doppler ultrasound examination, MRI examination and histopathological examination, among which histopathological examination is recognized as the gold standard for tumor diagnosis.

[0003] Histopathology examination refers to the observation of magnified pathological tissue sections by pathologists through a microscope. However, the rapid development of digital pathology scanners has enabled pathology sections to be permanently converted into high-resolution digital images. In the daily work of many hospital pathology departments, the digitization of histopathology sections has become an indispensable part. At the same time, digitized pathology images are also widely used in pathology reading exchange meetings, teaching, and examinations. Digitalized histopathology sections are also used in the research of computer-aided diagnosis systems, which has promoted the application and development of histopathology.

[0004] In the process of making pathological sections, unexpected artifacts may be introduced due to staff errors, such as folds, cracks, looseness, knife marks, and tissue bubbles. On the one hand, these artifacts will cause the original tissue area to be covered or destroyed, thus affecting the pathologist's reading and diagnosis. On the other hand, they will also affect the performance of CAD and even cause serious misdiagnosis.

[0005] In the current quality control process of pathology, for artifact detection in pathology sections, most hospitals will arrange personnel to conduct spot checks under a microscope, and the proportion of spot checks on the full pathology sections is generally no more than half. On the one hand, the hospital sends a large number of sections for inspection every day, the quality control cost is high and there are not enough staff; on the other hand, the effect of microscopic detection is poor and there are subjective differences.

[0006] With the development of deep learning technology, researchers have developed a variety of algorithms to automatically identify and classify artifacts in pathological images. These algorithms are usually based on convolutional neural networks (CNNs) and can learn the characteristics of artifacts from pathological images. However, artifact detection methods based on deep learning still face some challenges. First, the diversity and complexity of artifacts require CNN models to have a high degree of generalization ability to identify artifacts in pathological images with different staining methods and from different sampling sites. In addition, the reasoning performance of the model is also a hot topic in current research, and doctors and technicians hope to get results quickly in actual operations. For artifact detection, most existing methods have poor color perception of pathological images stained with hematoxylin-eosin (H&E), and thus cannot distinguish artifacts from normal tissues well. In addition, due to the problem of insufficient data, most methods only distinguish whether the image block contains artifacts. The causes of different artifacts are different, so it is of practical significance to carry out multi-classification tasks for artifacts in image blocks. Secondly, the existing methods have weak color perception ability for H&E stained pathological images, and some artifacts are easily confused with normal tissues, such as mild folds, cracks, and looseness. Summary of the invention

[0007] In order to overcome the shortcomings of the prior art, the present invention provides a multi-color domain pathology film artifact detection method based on contrast learning, which has the advantage of high artifact recognition accuracy.

[0008] To achieve the above object, the present invention provides the following technical solution: a multi-color domain pathology film preparation artifact detection method based on contrast learning, comprising the following steps: Step 1: Using the pathological full-slice cropping algorithm, all full-slice images are cropped into several sub-image blocks; Step 2: Design and implement the first feature extractor; Step 3: For the pathological sub-image blocks selected in step 2, use OpenCV tools to convert the RGB color domain image into HSV and LAB color domain images; Step 4: design a second feature extractor to extract key features from the three color domain pathological images obtained in step 3; Step 5: Construct the twin network structure and introduce the joint loss function; Step 6: In the testing phase, repeat the operations of steps 1 and 2 to obtain the original RGB color domain image, send the obtained image to the second feature extractor trained in step 4, and process it by the classifier to finally obtain the classification prediction result of the pathological image block.

[0009] As a preferred technical solution of the present invention, in step 1, the specific steps are: Step 1.1, define the folder path to be imported, install and configure the OpenSlide tool, and then load the pathology full-slice image; Step 1.2: In the image preprocessing stage, use the read_region method of OpenSlide to obtain the required sub-image blocks.

[0010] As a preferred technical solution of the present invention, in step 1.2, it is necessary to input coordinate parameters, level parameters, and region size parameters, wherein the coordinate parameters are specified in the form of a tuple, including x and y coordinates, to determine the position of the image block in the full slice; the level parameter is to specify the pyramid level of the pathological full slice; the region size parameter is to define the pixel size of the image block to be acquired, including but not limited to 256*256 pixels, 512*512 pixels, and 1024*1024 pixels.

[0011] As a preferred technical solution of the present invention, step 2 comprises the following steps: Step 2.1, design the first feature extractor, use MobileNetV2 as the network body, pre-train MobileNetV2 on ImageNet and save the weight parameter file; Step 2.2, modify the network structure of MobileNetV2, replace the activation function ReLU with the Leaky ReLU function, and add the ECA attention module between the 3*3 depth separable convolution and the 1*1 convolution; Step 2.3: Migrate the pre-trained model parameters to the optimized model architecture and perform further fine-tuning on a complete dataset containing valid and invalid pathology full sections; Step 2.4: After fine-tuning, the first feature extractor of the model can efficiently identify and distinguish valid and invalid image blocks, and then cache the valid image blocks in the specified file directory.

[0012] As a preferred technical solution of the present invention, in step 3, the specific steps are: Step 3.1, starting from all the RGB color domain image patches screened in step 2, ensure that there is a set of high-quality image patches as a basis; Step 3.2: In the Dataset class of the deep learning algorithm, redefine a subclass; Step 3.3, in the __getitem__ method of the newly defined Dataset subclass, use the cvtColor function of the OpenCV library to convert the RGB image into HSV and LAB color domain images; Step 3.4: The return value of the __getitem__ method includes the original RGB image object, the converted HSV image object, and the LAB image object.

[0013] As a preferred technical solution of the present invention, the specific method of step 4 is as follows: Step 4.1, introduce the second feature extractor GhostNetV2, and separate its classifier part; Step 4.2: The pathological images in the three color domains obtained by processing in step 3 are used as input and sent to an independent GhostNetV2 feature extractor.

[0014] As a preferred technical solution of the present invention, in step 5, the specific steps are: Step 5.1, compare the three acquired feature vectors in pairs and calculate the cosine similarity; Step 5.2: splice the classifier obtained in step 4.1 to the RGB color domain branch, output the features obtained by the branch to the classifier, and use the cross entropy loss function for constraint; Step 5.3: Based on the similarity loss function and cross entropy loss function obtained in the previous steps, a joint loss function is constructed to constrain the network training; Step 5.4: Input the actual data into the constructed network model, perform ten-fold cross validation, and select the best model in the validation to save.

[0015] As a preferred technical solution of the present invention, in step 5.1, the specific loss function calculation formula is as follows: ; ; ; ; Among them, the RGB feature vector is , the HSV eigenvector is , the LAB feature vector is ; For normalization operation, is the cosine similarity standard function, is the numerical control coefficient, , and are the similarity values ​​calculated pairwise, is the average of the above three values.

[0016] As a preferred technical solution of the present invention, in step 5.2, the cross entropy loss function formula is as follows: ; Among them, in the formula, In Represents cross entropy, and by default represents multivariate cross entropy; Indicates the total number of images, Represents the current image, It is a sample The label only takes values ​​in the range , Represents the predicted probability value, and its value range is , is the standard logarithmic function; In step 5.3, the specific formula of the joint loss function is as follows: .

[0017] As a preferred technical solution of the present invention, step 6 comprises the following steps: Step 6.1: In the testing phase, firstly repeat the operations of steps 1 to 2 to obtain the original RGB color domain image; Step 6.2, input the acquired RGB color domain image into the model trained in steps 1 to 5; the model uses its learned feature extraction and classification capabilities to conduct in-depth analysis of the image block, and the classifier outputs the prediction result of the pathological image block; Step 6.3, after completing the classification prediction, save the classification results of each image block and its coordinates in the original image; Step 6.4: Perform statistical analysis on the image blocks in the entire pathology full slice, calculate the number and proportion of each type of artifact image blocks; determine whether the proportion of each artifact exceeds the preset range by setting a threshold; if it exceeds the threshold, the pathology full slice is judged to be unqualified.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention has a better effect of detecting artifacts, and uses the contrast learning idea and multi-color domain learning of images to achieve high-precision automatic detection of pathological film artifacts; by introducing three color domains of RGB, HSV and LAB, and sending them to the feature extractor with shared parameters, image features that are easier to perceive color differences are extracted; using the improved MobileNetV2 as the first feature extractor, invalid background images are effectively filtered out, while the second feature extractor GhostNetV2 focuses on extracting multi-color features of valid images; parameter transfer learning of the feature extractor is realized through the ImageNet natural image data set, which accelerates the convergence of model training; in the training process, a joint loss function is designed, combining cross entropy loss and similarity contrast loss, to enhance the contrast learning ability of the network, thereby improving the discrimination of pathological images; finally, the trained network model parameters are fixed, and the RGB color domain image blocks are obtained according to the training process, and are input into the trained network to identify various artifacts in the image; not only the efficiency of automatic artifact detection of tissue pathology images is improved, but also its reliability is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the steps of the present invention; Figure 2 is a flow chart of step 1 of the present invention; Figure 3 is a flow chart of step 2 of the present invention; Figure 4 is a flow chart of step 3 of the present invention; Figure 5 is a flow chart of step 4 of the present invention; Figure 6 is a flow chart of step 5 of the present invention; Figure 7 is a flow chart of step 6 of the present invention; Figure 8 This is a flow chart of the training phase of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figures 1 to 8 , any one embodiment of the present invention is described below for further explanation: A method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning comprises the following steps: Step 1: Using the pathological full-slice cropping algorithm, all full-slice images are cropped into several sub-image blocks; Step 2: Design and implement the first feature extractor; its task is to filter out invalid pathological sub-image blocks, thereby improving the quality and efficiency of subsequent analysis; Step 3: For the pathological sub-image blocks selected in step 2, use OpenCV tools to convert the RGB color domain image into HSV and LAB color domain images; this conversion process not only enriches the feature information of the image, but also provides more dimensional data for subsequent feature extraction and analysis. At the same time, the image blocks of these three color domains are temporarily stored for subsequent processing; Step 4: design a second feature extractor specifically for extracting key features from the three color domain pathology images obtained in step 3; Step 5: Construct the twin network structure and introduce the joint loss function; optimize the training process of the overall network. The twin network improves the consistency of feature extraction and classification through collaborative learning, while the joint loss function ensures the stability and effectiveness of network training. Step 6: In the testing phase, repeat the operations of steps 1 and 2 to obtain the original RGB color domain image, send the obtained image to the second feature extractor trained in step 4, and process it by the classifier to finally obtain the classification prediction result of the pathological image block.

[0022] Among them, in step 1, the specific steps are: Step 1.1, define the folder path to be imported, and install and configure the OpenSlide tool, which is the key tool for processing pathology full-slide images. Then load the pathology full-slide image to prepare for subsequent image block cutting; Step 1.2: In the image preprocessing stage, use the read_region method of OpenSlide to obtain the required sub-image blocks.

[0023] In step 1.2, you need to input coordinate parameters, level parameters, and area size parameters; Coordinates: Specified in the form of a tuple, mainly containing x and y coordinates, determining the position of the image block in the full slice; Level: specifies the pyramid level of the pathology full slice, which affects the resolution and details of the acquired image patch; Region size: defines the pixel size of the image block to be acquired, which is preset to 256*256 pixels in the present invention. Optionally, the size of the small image cut in step 2 can be selected to be 512*512, 1024*1024 pixels, etc.

[0024] Among them, in step 2, the purpose is to effectively filter out invalid image blocks, and the specific method is as follows: Step 2.1, design the first feature extractor, use MobileNetV2 as the network body, pre-train MobileNetV2 on ImageNet and save the weight parameter file; Step 2.2, modify the network structure of MobileNetV2, replace the activation function ReLU with the Leaky ReLU function, which helps to expand the range of the ReLU function, which helps to adjust the zero gradient problem of negative values; add the ECA attention module between the 3*3 depth separable convolution and the 1*1 convolution. The ECA attention mechanism uses 1D convolution to efficiently implement local cross-channel interaction, which can avoid dimensionality reduction to further improve computational efficiency; Step 2.3: Migrate the pre-trained model parameters to the optimized model architecture and further fine-tune it on a complete dataset containing valid and invalid pathology full slides; this process involves applying the weights of the existing model to the improved model framework and then training it on a specific dataset to optimize the performance and accuracy of the model; this approach ensures that the model can be more accurate and reliable when processing specific pathology images; Step 2.4: After fine-tuning, the first feature extractor of the model can efficiently identify and distinguish valid and invalid image blocks, and then cache the valid image blocks in the specified file directory.

[0025] Among them, the purpose of step 3 is to obtain a multi-color domain image of a valid sub-image block, and the specific method is as follows: Step 3.1, starting from all the RGB color domain image patches screened in step 2, ensure that there is a set of high-quality image patches as a basis; Step 3.2: Redefine a subclass in the Dataset class of the deep learning algorithm; this step is to enable the model to better process and understand image data in different color domains; Step 3.3, in the __getitem__ method of the newly defined Dataset subclass, use the cvtColor function of the OpenCV library to convert the RGB image into HSV and LAB color domain images; Step 3.4: The return value of the __getitem__ method includes the original RGB image object, the converted HSV image object, and the LAB image object. In this way, the model can process images in multiple color domains simultaneously within a unified framework, enhancing its ability to understand and analyze image features.

[0026] Among them, in step 4, the purpose is to extract key features from the pathological images of three different color domains (RGB, HSV, LAB) obtained in step 3. The specific steps are as follows: Step 4.1, introduce the second feature extractor GhostNetV2, and separate its classifier part; Step 4.2: The pathological images in the three color domains (RGB, HSV, and LAB) obtained in step 3 are used as input and sent to an independent GhostNetV2 feature extractor. Through this process, we can extract unique feature vectors from each color domain, providing rich information for subsequent classification and identification.

[0027] Among them, in step 5, the purpose is to design a twin network structure and introduce a joint loss function. The specific steps are as follows: Step 5.1, compare the three acquired feature vectors in pairs and calculate the cosine similarity; The specific loss function calculation formula is as follows: ; ; ; ; Among them, the RGB feature vector is , the HSV eigenvector is , the LAB feature vector is ; For normalization operation, is the cosine similarity standard function, is the numerical control coefficient, , and are the similarity values ​​calculated pairwise, is the average of the above three values.

[0028] Step 5.2: splice the classifier obtained in step 4.1 to the RGB color domain branch, output the features obtained by the branch to the classifier, and use the cross entropy loss function for constraint; The cross entropy loss function formula is as follows: ; Among them, in the formula, In Represents cross entropy, and by default represents multivariate cross entropy; Indicates the total number of images, Represents the current image, It is a sample The label only takes values ​​in the range , Represents the predicted probability value, and its value range is , is the standard logarithmic function.

[0029] Step 5.3: Based on the similarity loss function and cross entropy loss function obtained in the previous steps, a joint loss function is constructed to constrain the network training; The specific formula of the joint loss function is as follows: .

[0030] Step 5.4: Input the actual data into the constructed network model, perform ten-fold cross validation, and select the best model in the validation to save.

[0031] Wherein, step 6 comprises the following steps: Step 6.1: In the testing phase, firstly repeat the operations of steps 1 to 2 to obtain the original RGB color domain image; Step 6.2, input the acquired RGB color domain image into the model trained in steps 1 to 5; the model uses its learned feature extraction and classification capabilities to conduct in-depth analysis of the image block, and the classifier outputs the prediction result of the pathological image block; Step 6.3, after completing the classification prediction, save the classification results of each image block and its coordinates in the original image; Step 6.4: Perform statistical analysis on the image blocks in the entire pathology full slice, and calculate the number and proportion of each type of artifact image blocks. By setting a reasonable threshold, determine whether the proportion of each artifact exceeds the preset range. If it exceeds the threshold, the pathology full slice is judged as unqualified, thereby achieving effective monitoring of the quality of the pathology image.

[0032] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0033] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-color domain pathology film artifact detection method based on contrast learning, characterized in that: The steps include: Step 1: Using the pathological full-slice cropping algorithm, all full-slice images are cropped into several sub-image blocks; Step 2: Design and implement the first feature extractor; Step 3: For the pathological sub-image blocks selected in step 2, use OpenCV tools to convert the RGB color domain image into HSV and LAB color domain images; Step 4: design a second feature extractor to extract key features from the three color domain pathological images obtained in step 3; Step 5: Construct the twin network structure and introduce the joint loss function; Step 6: In the testing phase, repeat the operations of steps 1 and 2 to obtain the original RGB color domain image, send the obtained image to the second feature extractor trained in step 4, and process it by the classifier to finally obtain the classification prediction result of the pathological image block.

2. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 1, characterized in that: In step 1, the specific steps are: Step 1.1, define the folder path to be imported, install and configure the OpenSlide tool, and then load the pathology full-slice image; Step 1.2: In the image preprocessing stage, use the read_region method of OpenSlide to obtain the required sub-image blocks.

3. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 2, characterized in that: In step 1.2, it is necessary to input coordinate parameters, level parameters, and region size parameters, wherein the coordinate parameters are specified in the form of a tuple, including x and y coordinates, to determine the position of the image block in the full slice; the level parameter is to specify the pyramid level of the pathological full slice; the region size parameter is to define the pixel size of the image block to be acquired, including but not limited to 256*256 pixels, 512*512 pixels, and 1024*1024 pixels.

4. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1, design the first feature extractor, use MobileNetV2 as the network body, pre-train MobileNetV2 on ImageNet and save the weight parameter file; Step 2.2, modify the network structure of MobileNetV2, replace the activation function ReLU with the Leaky ReLU function, and add the ECA attention module between the 3*3 depth separable convolution and the 1*1 convolution; Step 2.3: Migrate the pre-trained model parameters to the optimized model architecture and perform further fine-tuning on a complete dataset containing valid and invalid pathology full sections; Step 2.4: After fine-tuning, the first feature extractor of the model can efficiently identify and distinguish valid and invalid image blocks, and then cache the valid image blocks in the specified file directory.

5. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 1, characterized in that: In step 3, the specific steps are: Step 3.1, starting from all the RGB color domain image patches screened in step 2, ensure that there is a set of high-quality image patches as a basis; Step 3.2: In the Dataset class of the deep learning algorithm, redefine a subclass; Step 3.3, in the __getitem__ method of the newly defined Dataset subclass, use the cvtColor function of the OpenCV library to convert the RGB image into HSV and LAB color domain images; Step 3.4: The return value of the __getitem__ method includes the original RGB image object, the converted HSV image object, and the LAB image object.

6. The method for detecting artifacts in multi-color domain pathological film preparation based on contrastive learning according to claim 1, characterized in that: The specific steps of step 4 are as follows: Step 4.1, introduce the second feature extractor GhostNetV2, and separate its classifier part; Step 4.2: The pathological images in the three color domains obtained by processing in step 3 are used as input and sent to an independent GhostNetV2 feature extractor.

7. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 1, characterized in that: In step 5, the specific steps are: Step 5.1, compare the three acquired feature vectors in pairs and calculate the cosine similarity; Step 5.2: splice the classifier obtained in step 4.1 to the RGB color domain branch, output the features obtained by the branch to the classifier, and use the cross entropy loss function for constraint; Step 5.3: Based on the similarity loss function and cross entropy loss function obtained in the previous steps, a joint loss function is constructed to constrain the network training; Step 5.4: Input the actual data into the constructed network model, perform ten-fold cross validation, and select the best model in the validation to save.

8. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 7, characterized in that: In step 5.1, the specific loss function calculation formula is as follows: ； ； ； ； Among them, the RGB feature vector is , the HSV eigenvector is , the LAB feature vector is ; For normalization operation, is the cosine similarity standard function, is the numerical control coefficient, , and are the similarity values ​​calculated pairwise, is the average of the above three values.

9. The method for detecting artifacts in multi-color domain pathological film preparation based on contrast learning according to claim 7, characterized in that: In step 5.2, the cross entropy loss function formula is as follows: ； Among them, in the formula, In Represents cross entropy, and by default represents multivariate cross entropy; Indicates the total number of images, Represents the current image, It is a sample The label only takes values ​​in the range , Represents the predicted probability value, and its value range is , is the standard logarithmic function; In step 5.3, the specific formula of the joint loss function is as follows: 。 10. The method for detecting pathological film artifacts in multiple color domains based on contrastive learning according to claim 1, characterized in that: The step 6 comprises the following steps: Step 6.1: In the testing phase, firstly repeat the operations of steps 1 to 2 to obtain the original RGB color domain image; Step 6.2, input the acquired RGB color domain image into the model trained by steps 1 to 5; The model uses its learned feature extraction and classification capabilities to conduct in-depth analysis of the image blocks, and the classifier outputs the prediction results of the pathological image blocks; Step 6.3, after completing the classification prediction, save the classification results of each image block and its coordinates in the original image; Step 6.4, statistically analyzing the image blocks in the entire pathological full slice, and calculating the number and proportion of various artifact image blocks; By setting a threshold, it is determined whether the proportion of each artifact exceeds a preset range; If it exceeds the threshold, the pathological full section is judged as unqualified.

Images