Pathological image artifact fine-grained classification method based on double-branch fusion network
Through the combination of the dual-branch fusion network and the HSV edge detection module, the limitations of deep learning methods when distinguishing pathological image artifacts with high feature similarity are solved, the fine-grained classification accuracy of artifacts is improved, and the similar artifacts are realized Efficient identification.
Patent Information
- Application Number
- CN202411129032.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-16
AI Technical Summary
Existing deep learning methods show limitations when distinguishing pathological image artifacts with high feature similarity, making it difficult to pay full attention to global information, resulting in insufficient recognition accuracy.
Using a method based on a dual-branch fusion network, combining convolutional neural network (CNN) and visual transformer (Transformer), the interaction and fusion of local and global information is enhanced through the edge detection module and cross attention mechanism of HSV, and the interaction and fusion of local and global information are enhanced, and artifacts are improved. fine-grained classification capability.
It significantly improves the recognition accuracy of similar artifacts, enhances the model's ability to recognize subtle features such as cracks, and maintains the integrity of image color information, and improves the fine-grained classification accuracy of pathological image artifacts.
Smart Images

Figure CN118968178A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to digital image processing, histopathological image artifact recognition and computer-aided technology, and relates to a pathological image artifact fine-grained classification method. Background Art
[0002] Histopathological image analysis provides accurate pathological information for tumor diagnosis and treatment, and is recognized by experts in this field as the gold standard for tumor diagnosis. Before digitization, histopathological images need to go through complex steps such as specimen collection, cutting, and staining. This process is cumbersome, has a low degree of automation, and is affected by the subjective operation of pathological technicians, which is prone to unexpected defects (called "artifacts"). Artifacts are mainly divided into three categories: 1) tissue-level artifacts, such as knife marks and looseness; 2) slice-level artifacts, such as bubbles and ink; 3) scanning-level artifacts, such as defocus. These artifacts may prevent pathologists from observing normal tissues when observing and judging real lesions, thus leading to misdiagnosis or missed diagnosis, which in turn affects the patient's treatment and prognosis. In addition, histopathological images with artifacts also have a negative impact on computer-aided diagnosis systems. Therefore, it is particularly important to detect artifacts in the completed histopathological images.
[0003] At present, there are three main methods for artifact recognition in pathological images: manual quality inspection, machine learning, and deep learning. Manual quality inspection is the most commonly used method in medical institutions. Since a large number of tissue pathology sections are prepared every day, clinical pathology departments can only conduct manual sampling at a certain ratio, which cannot cover all newly prepared sections. In addition, due to the lack of unified evaluation criteria, the judgment results between different doctors may be different. In contrast, machine learning and deep learning methods provide a more efficient and objective solution. Early artifact recognition technologies mostly used machine learning algorithms, which usually rely on the color features of the image or artificially designed features. The color feature-based method is effective in identifying artifacts with obvious color differences (such as tissue folding and ink stains), but the recognition power is insufficient when the color difference is small. The method based on artificially designed features not only increases the complexity of algorithm design, but also makes it difficult to ensure the universality of the designed features.
[0004] Deep learning methods provide a more adaptive solution. Artifact recognition methods based on deep learning have significantly surpassed traditional machine learning methods in terms of accuracy and generalization. Many scholars have applied convolutional neural networks (CNNs) to artifact recognition in histopathological images, such as using ResNet101, YOLOV5, and Densenet121 models as feature extractors. These feature extractors perform well in capturing local features such as color and texture, but they lack the ability to effectively process global information such as large-area macroscopic topological structures. In order to solve the above problems, some scholars have tried to introduce attention mechanisms. However, these attention mechanisms are limited by the size of the receptive field, which means that they are subject to certain limitations when processing global information, and thus the performance improvement and gain they can bring are also limited. In recent years, the Transformer model has made significant progress in the field of vision. Its core self-attention mechanism gives the model a broad global receptive field, greatly improving its ability to process image information. Guo et al. introduced a multi-head attention mechanism to classify artifacts, but its local information and global information were only simply integrated at the end of the model, and the performance improvement brought was very limited.
[0005] Existing research has achieved good results in identifying and distinguishing artifacts with large morphological and color differences, such as ink, tissue folding, blur, etc. However, there are many types of artifacts, and some artifacts have high similarity, such as Figure 1 Four artifacts are shown. By observing the local areas marked by red boxes, it can be found that there are significant similarities between them: irregular cracks are distributed in the tissues of these areas. If the model only focuses on these details, it will be difficult to distinguish different artifacts. However, observing the overall distribution of cracks, it can be found that there are obvious differences between them. For example, loose cracks are distributed more densely and in inconsistent directions; cracks in fissures mostly extend in the same direction. Therefore, in order to effectively distinguish these artifacts, the model needs to understand its overall pattern and structure while being sensitive to the local features of the defects. However, existing deep learning methods often find it difficult to fully pay attention to their global information when distinguishing these similar artifacts, which urgently requires us to conduct more in-depth research.
[0006] With the rapid advancement of deep learning technology in the field of computer vision, convolutional neural networks (CNNs) have made significant progress in the application of artifact recognition in pathological images. Foucart et al. used a deep residual network to extract information from rough labels for artifact recognition and segmentation in hematoxylin-eosin (H&E) and immunohistochemistry (IHC) whole slice images (WSIs). Haghighat et al. used a multi-task deep neural network based on ResNet18 to evaluate the quality of WSI block by block at 5x magnification. However, the network models used in these studies, such as deep residual networks and ResNet18, have certain limitations in processing global information.
[0007] To overcome this challenge, Li et al. proposed a two-stage CNN method, which effectively weighted the features of different channels by introducing a channel attention mechanism and an attention residual block. However, this method does not solve the problem of limited receptive field. To this end, Shakarami et al. proposed a new CNN architecture TCNN, which enhances the model's ability to capture the global contextual information of the image by applying dilated convolutions in the last few layers of the convolutional layer. Pizarro et al. (2023) stacked multiple convolutional layers to increase their receptive field layer by layer. The above-mentioned CNN-based improvements have broadened the network's receptive field to a certain extent, enabling it to capture global information more effectively. However, the receptive field of CNN is still limited by the size of the convolution kernel and the depth of the network architecture. In particular, as the depth of the network increases, the risk of losing detailed information also increases. These are issues that need to be further addressed in current research.
[0008] With the introduction of Vision Transformer (ViT), the application of Transformer architecture in the field of computer vision has attracted widespread attention. Thanks to the self-attention mechanism, the receptive field of Transformer can immediately cover the entire input image. Guo et al. proposed the BIQA model, the innovation of which is that it combines a DINet-based saliency module and an image block module (patch block) using multi-head self-attention and pyramid position encoding generator, which is specifically used to extract the correlation and spatial information between image patches. The model combines CNN-based and Transformer-based network modules to process local artifact features and global artifact features respectively. However, the model only integrates local information and global information through simple splicing before the final classification head. This method fails to give full play to the advantages of the combination of the two, resulting in limited performance improvement.
[0009] In response to this, Chen et al. and Han et al. successively proposed mobile-former and Dyn-Perceiver. They both adopted the parallel design of CNN and Transformer, and realized the feature fusion between the two through a bidirectional bridge structure. However, the feed-forward network (FFN) used in the Transformer branch of Mobile-Former does not perform well in the parallel framework, and the input of the MobileNet sub-block depends on the output of the Transformer sub-block, which limits the independent parallel performance of the two branches. Dyn-Perceiver focuses more on dynamic early exit.
[0010] At present, all the mentioned artifact recognition technologies mainly distinguish artifacts with significant feature differences. However, for artifacts with high feature similarity, the existing recognition methods show limitations in their ability to distinguish. Among the current artifact recognition methods based on deep learning, some methods borrow strategies for color processing in the field of traditional machine learning, but compared with traditional methods, these deep learning methods have significantly reduced their dependence on color. For example, TCNN uses a color mapping layer in the initial layer, which converts the color space of the image from the RGB domain to the HSV domain, but this method only performs a simple color domain conversion. The AR-Classifier proposed by Ke et al. further utilizes complementary information from different color spaces (such as LAB, HED, and HSV), but it also leads to an exponential increase in the amount of computation. Summary of the invention
[0011] In order to overcome the shortcomings of the existing technology, the present invention provides a fine-grained classification method for pathological image artifacts based on a dual-branch fusion network, which fully fuses and interacts the local and global information of artifacts to achieve fine-grained classification of similar artifacts on pathological images; this dual-branch architecture consists of two key parts: a convolutional neural network (CNN) branch and a visual transformer (Transformer) branch, the former is used to fully extract local information of texture and color, and the latter is used to capture global information such as the overall distribution law of cracks; on the basis of this architecture, in order to enhance the network's attention to crack features, an edge detection module based on HSV is constructed; in addition, in order to further enhance the interaction and fusion of local and global information, a cross-attention mechanism is designed, which can promote the effective integration of information between different regions, thereby improving the model's recognition accuracy for similar artifacts.
[0012] The technical solution adopted by the present invention to solve its technical problem is:
[0013] A fine-grained classification method for pathological image artifacts based on a dual-branch fusion network includes the following steps:
[0014] Step 1: First, obtain the whole slice WSI of the tissue pathology image with artifacts, and have the pathologist professionally annotate the artifact area to obtain the annotated data of the artifact area;
[0015] Step 2: Perform OTSU threshold segmentation on the WSI obtained in step 1 to distinguish the foreground and background in the image, and segment the foreground image in the marked area into small images of N×N pixels;
[0016] Step 3: Use the HSV-based edge detection module to enhance the crack features in the image block obtained in step 2 to highlight the crack features in the artifact;
[0017] Step 4: Perform data augmentation on the image patches obtained in step 3 to increase the generalization ability of the model and reduce overfitting;
[0018] Step 5: Perform preliminary feature extraction and adjust the data dimension of the data obtained in step 4 to ensure that it can be adapted and input into the two processing branches respectively;
[0019] Step 6: Input the data converted in step 5 into the dual-branch network, and obtain the final artifact fine-grained classification result through classification processing.
[0020] Furthermore, in step 2, the whole slice is cut into pixel sizes suitable for mainstream convolutional neural network processing, and the process is as follows:
[0021] Step 2.1: Use the OpenSlide module to read the pathological full-slice image and simultaneously obtain the relevant annotation data. This step ensures that the image data and its corresponding annotation information can be accurately loaded, laying the foundation for subsequent processing;
[0022] Step 2.2: Use the OTSU threshold segmentation algorithm to process the image to identify and extract the foreground area in the image, that is, the part containing the tissue structure;
[0023] Step 2.3: Generate a mask map of the annotated area based on the existing annotation information; then, perform a logical AND operation on the foreground area mask and the annotated area mask to accurately locate the tissue area within the annotated area; finally, evenly segment the selected tissue area according to the standard size of N×N pixels.
[0024] Preferably, in step 2, N×N selects 256×256, 512×512 or 1024×1024 pixels.
[0025] Furthermore, in step 3, an HSV-based edge detection module is used to enhance the crack information in the image while ensuring that the color information of the image is not lost. The process is as follows:
[0026] Step 3.1: Use the OpenCV library to load the image file from the specified path and perform color space conversion to convert the image from BGR to HSV color space. This conversion helps to process the hue, saturation, and brightness information of the image more intuitively.
[0027] Step 3.2: In the HSV color space, decompose the image into three independent channels: hue H, saturation S, and brightness V. Apply Gaussian blur technique to the brightness channel to smooth the image and reduce noise interference, creating a clearer background for subsequent edge recognition.
[0028] Step 3.3: Apply the Canny edge recognition algorithm to the brightness channel after Gaussian blur processing to accurately identify the edge lines in the image;
[0029] Step 3.4: Use the edge obtained by Canny edge recognition as a mask to extract the edge area in the original image;
[0030] Step 3.5: Perform a weighted merge of the original image and the image in the edge area to highlight the edge while retaining the color information of the original image.
[0031] Furthermore, in step 4, data enhancement is used to improve the adaptability of the model to diverse image content. The process is as follows:
[0032] Step 4.1: Perform a series of transformation operations on the image to achieve data augmentation, including rotation, horizontal or vertical flipping, scaling, cropping, and random brightness adjustment;
[0033] Step 4.2: Convert the enhanced image into a four-dimensional tensor format suitable for deep learning model input. This conversion ensures that the image data can be effectively processed and analyzed by the model.
[0034] In step 5, image features are initially extracted and data dimensions are adjusted so that the data can be adapted to two different network branches. The process is as follows:
[0035] Step 5.1: Input the data obtained in step 4 into a stem module, which is composed of several convolutional layers. The core function of this module is to perform preliminary feature extraction and dimension conversion on the input image. After this step, the data will be ready to enter the CNN branch.
[0036] Step 5.2: Perform further dimension conversion on the output data obtained in step 5.1 and convert it into a three-dimensional data format suitable for the Transformer branch. This conversion is to meet the Transformer branch's requirements for specific dimensions of input data and ensure that the data can be correctly fed into the Transformer network for processing.
[0037] In step 6, a dual-branch network structure is used to simultaneously capture local details and global context information of the image, and the recognition ability of the model is improved through feature fusion and interaction. The process is as follows:
[0038] Step 6.1: Input the data processed in step 5 into the CNN branch and the Transformer branch respectively. The two branches are implemented using the MobileNet and MG-former (MobileGLU-Transformer) architectures respectively.
[0039] Step 6.2: Perform feature extraction in the MobileNet branch and the MG-former branch respectively. The MG-former module is a neural network module optimized on the basis of Transformer, which is composed of an improved feed-forward layer MobileGLU and a linear focused attention.
[0040] Step 6.3: While the MobileNet branch and the MG-former branch are learning features independently, the feature interaction between the two branches is realized through the FLCA module. The FLCA module receives two sets of input features: when FLCA is the first FLCA module from top to bottom, the current branch features correspond to the features of the MG-former branch, and the other branch features correspond to the features of the MobileNet branch. Each set of features is first batch normalized; then, the current branch features are linearly projected W Q Generate the query vector Q, and the other branch is linearly projected W K , W V Generate the key vector K and the value vector V, and then apply the focus similarity (F) to Q and K to calculate the attention weight. The focus similarity calculation process is shown in formula (1). In formula (2), x **p is to perform a p-power operation on each element in x, focusing on the function f p It can help focus on similar features. The norm of the features before and after mapping is the same, that is, ||x||=||f p (x)||, indicating that the function only adjusts the feature direction; then the obtained attention weight is multiplied by V element by element, and then added element by element with Q after deep convolution DWC processing to generate the updated current branch feature. This process not only enhances the expressiveness of the feature, but also promotes the information interaction between the two branches;
[0041]
[0042] in,
[0043] Step 6.4: Finally, the features of the two branches are fused (concatenation) to generate a comprehensive feature vector, and the final classification result is output based on it.
[0044] In the step 6.2, the MG-former module is composed of a MobileGLU module and a linear focused attention. In the MG-former module, BN is a batch normalization operation, DWC (depthwise conv) is a depth convolution, gelu is an activation function, LN is a layer normalization operation, and FL-Att (Focused Linear Attention) is a linear focused attention mechanism; the MobileGLU processing flow is as follows: first, the input feature map is batch normalized, and then channel mixing is performed through a 1*1 convolution layer, and a GELU activation function is applied to increase nonlinearity; then, the feature map is further subjected to a 3*3 depth convolution to further extract spatial features, and is again subjected to batch normalization and GELU activation; next, the processed feature map is matrix multiplied with the initial input feature map after dimension transformation, and is added to the initial input feature map after processing through a 1*1 convolution layer; after MobileGLU, layer normalization and a linear focused attention mechanism are applied to enhance feature representation capabilities and generate the final feature map.
[0045] The technical concept of the present invention is: adopting the idea of parallel framework, but optimizing the interaction timing between CNN branch and Transformer branch. In order to deepen the interaction and fusion of features, a linear focused cross attention module, FocusedLinearCrossAttention (FLCA), is designed to improve the efficiency and effect of feature interaction mechanism. In addition, in order to enhance the feature extraction capability of Transformer branch in parallel framework, MobileGLU is designed based on the inverted residual structure and gating mechanism (GLU) in MobileNet2 to replace the feedforward layer in the traditional self-attention mechanism, in order to achieve better performance.
[0046] Among the artifacts to be distinguished by the present invention, there is a significant difference in color between the cracks and the surrounding tissues, which is the key point of distinction. An HSV-based edge detection module is proposed. The module first converts the image from the RGB domain to the HSV domain, and then introduces the Canny edge recognizer to enhance the boundary information between the cracks and normal tissues in the image, thereby improving the ability to distinguish artifacts.
[0047] In view of the complexity of similar artifacts in pathological images, a dual-branch fusion network structure was introduced and improved. This structure optimizes the integration of local and global information, so that the model can not only capture image details, but also deeply understand its overall pattern, significantly improving the accuracy of classification. And through the designed HSV-based edge detection module, this module significantly enhances the model's ability to recognize subtle features such as cracks, while ensuring the integrity of image color information and maintaining the model's high sensitivity to artifacts. It can effectively solve the shortcomings of existing technologies in fine-grained classification of similar artifacts, and provide a more efficient and accurate tool for artifact recognition in tissue pathology images.
[0048] The beneficial effects of the present invention are mainly manifested in: fully fusing and interacting the local and global information of artifacts to achieve fine-grained classification of similar artifacts on pathological images and improve artifact recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 are examples of artifacts. The types of artifacts in each row from top to bottom are loose, tissue bubbles, cracks, and knife marks.
[0050] Figure 2 It is a flow chart.
[0051] Figure 3 It is a double-branch network diagram.
[0052] Figure 4 This is a schematic diagram of the MG-former module.
[0053] Figure 5 It is the diagram of the cross-attention FLCA module. DETAILED DESCRIPTION
[0054] The present invention will be further described below in conjunction with the accompanying drawings.
[0055] Reference Figure 2 to Figure 5 , a fine-grained classification method for pathological image artifacts based on a dual-branch fusion network, comprising the following steps:
[0056] Step 1: First, obtain a whole slide image (WSI) of tissue pathology images with artifacts, and have a pathologist professionally annotate the artifact area to obtain the annotated data of the artifact area;
[0057] Step 2: Perform OTSU (Otsu's Thresholding) threshold segmentation operation on the WSI obtained in step 1 to distinguish the foreground and background in the image, and segment the foreground image in the marked area into small images of 256×256 pixels;
[0058] Optionally, the size of the small image cut in step 2 can be selected to be 512×512, 1024×1024 pixels or the like.
[0059] In step 2, the whole slice is cut into pixel sizes suitable for mainstream convolutional neural network processing, and the process is as follows:
[0060] Step 2.1: Use the OpenSlide module to read the pathological full-slice image and simultaneously obtain the relevant annotation data. This step ensures that the image data and its corresponding annotation information can be accurately loaded, laying the foundation for subsequent processing;
[0061] Step 2.2: Use the OTSU threshold segmentation algorithm to process the image to identify and extract the foreground area in the image, that is, the part containing the tissue structure;
[0062] Step 2.3: Generate a mask map of the annotated area based on the existing annotation information. Then, perform a logical AND operation on the foreground area mask and the annotated area mask to accurately locate the tissue area within the annotated area. Finally, the selected tissue area is evenly segmented according to the standard size of 256×256 pixels.
[0063] Step 3: Use the HSV-based edge detection module to enhance the crack features in the image block obtained in step 2 to highlight the cracks and other features in the artifacts;
[0064] In step 3, an HSV-based edge detection module is used to enhance the crack information in the image while ensuring that the color information of the image is not lost. The process is as follows:
[0065] Step 3.1: Use the OpenCV library to load the image file from the specified path and perform color space conversion to convert the image from BGR to HSV color space. This conversion helps to process the hue, saturation, and brightness information of the image more intuitively;
[0066] Step 3.2: In the HSV color space, decompose the image into three independent channels: hue (H), saturation (S), and brightness (V). Apply Gaussian blur technique to the brightness channel to smooth the image and reduce noise interference, creating a clearer background for subsequent edge recognition.
[0067] Step 3.3: Apply the Canny edge recognition algorithm to the brightness channel after Gaussian blur processing to accurately identify the edge lines in the image;
[0068] Step 3.4: Use the edge obtained by Canny edge recognition as a mask to extract the edge area in the original image;
[0069] Step 3.5: Perform a weighted merge of the original image and the image in the edge area to highlight the edge while retaining the color information of the original image.
[0070] Step 4: Perform data augmentation on the image patches obtained in step 3 to increase the generalization ability of the model and reduce overfitting;
[0071] In step 4, the purpose is to improve the adaptability of the model to diverse image content through data enhancement. The process is as follows:
[0072] Step 4.1: Perform a series of transformation operations on the image to achieve data augmentation, including but not limited to rotation, horizontal or vertical flipping, scaling, cropping, and random brightness adjustment;
[0073] Step 4.2: Convert the enhanced image into a four-dimensional tensor format suitable for deep learning model input. This conversion ensures that the image data can be effectively processed and analyzed by the model.
[0074] Step 5: Perform preliminary feature extraction and adjust the data dimension of the data obtained in step 4 to ensure that it can be adapted and input into the two processing branches respectively;
[0075] In step 5, the purpose is to preliminarily extract image features and adjust the data dimension so that the data can adapt to two different network branches. The process is as follows:
[0076] Step 5.1: Input the data obtained in step 4 into a stem module, which consists of several convolutional layers. The core function of this module is to perform preliminary feature extraction and dimension conversion on the input image. After this step, the data will be ready to enter the CNN branch;
[0077] Step 5.2: Perform further dimension conversion on the output data obtained in step 5.1 and convert it into a three-dimensional data format suitable for the Transformer branch. This conversion is to meet the Transformer branch's requirements for specific dimensions of input data and ensure that the data can be correctly fed into the Transformer network for processing.
[0078] Step 6: Input the data converted in step 5 into the dual-branch network, and obtain the final artifact fine-grained classification result through classification processing.
[0079] In step 6, a dual-branch network structure is used to simultaneously capture local details and global context information of the image, and the recognition ability of the model is improved through feature fusion and interaction. Figure 3 As shown, the process is as follows:
[0080] Step 6.1: Input the data processed in step 5 into the CNN branch and the Transformer branch respectively. In the present invention, these two branches are implemented using the MobileNet and MG-former (MobileGLU-Transformer) architectures respectively.
[0081] Step 6.2: Perform feature extraction in the MobileNet branch and the MG-former branch. The MG-former module is a neural network module optimized on the basis of Transformer, which is mainly composed of an improved feedforward layer MobileGLU and a linear focused attention. Figure 4 As shown in the figure, in this module, BN is a batch normalization operation, DWC (depthwise conv) is a depth convolution, gelu is an activation function, LN is a layer normalization operation, and FL-Att (Focused Linear Attention) is a linear focused attention mechanism.
[0082] The MobileGLU processing flow is as follows: First, the input feature map is batch normalized, followed by channel mixing through a 1*1 convolution layer, and the GELU activation function is applied to increase nonlinearity. After that, the feature map is further convolved with a 3*3 depthwise convolution to extract spatial features, and then batch normalized and GELU activated again. Next, the processed feature map is matrix multiplied with the initial input feature map after dimension transformation, and then processed through a 1*1 convolution layer and added to the initial input feature map. After the MobileGLU module, layer normalization and linear focus attention mechanism are applied to enhance feature representation capabilities and generate the final feature map.
[0083] Step 6.3: While the MobileNet branch and the MG-former branch are learning features independently, the FLCA module is used to implement feature interaction between the two branches, such as Figure 5 As shown, the FLCA module receives two sets of input features: Figure 3 When the first FLCA module from top to bottom is used in , the current branch features correspond to the features of the MG-former branch, while the other branch features correspond to the features of the MobileNet branch. Each set of features is first batch normalized. Subsequently, the current branch features are linearly projected W Q Generate the query vector Q, and the other branch is linearly projected W K , W V Generate the key vector K and the value vector V, and then apply the focus similarity (F) to Q and K to calculate the attention weight. The focus similarity calculation process is shown in formula (1). In formula (2), x**p is to perform a p-power operation on each element in x, focusing on the function f p It can help focus on similar features. The norm of the features before and after mapping is the same, that is, ||x||=||f p (x)||, indicating that the function only adjusts the feature direction. Then the obtained attention weight is multiplied by V element by element, and then added element by element with Q after deep convolution (DWC) to generate the updated current branch feature. This process not only enhances the expressiveness of the feature, but also promotes the information interaction between the two branches.
[0084]
[0085] in,
[0086] Step 6.4: Finally, the features of the two branches are fused (concatenation) to generate a comprehensive feature vector, and the final classification result is output based on it.
[0087] The solution of this embodiment introduces a dual-branch architecture that integrates CNN and Transformer in view of the characteristics of artifacts. To improve the feature extraction effect of the Transformer branch, the present invention designs a MobileGLU based on the inverted residual structure and the gated mechanism GLU to replace the traditional feedforward layer, and combines it with the linear focus attention to form an MG-former to replace the Transformer, which significantly improves the recognition accuracy of the model; to improve the interaction and fusion of local and global information, the present invention designs a linear focus cross attention mechanism FLCA, which introduces a linear focus function in the cross attention to promote the effective integration of information between different regions; to enhance the network's attention to crack features, an HSV-based edge detection module is constructed. This module performs edge recognition while converting the color domain to the HSV color space, significantly enhancing the model's ability to recognize subtle features such as cracks while retaining the image color information.
[0088] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.
Claims
1. A fine-grained classification method for pathological image artifacts based on a dual-branch fusion network, characterized in that: The method comprises the following steps: Step 1: First, obtain the whole slice WSI of the tissue pathology image with artifacts, and have the pathologist professionally annotate the artifact area to obtain the annotated data of the artifact area; Step 2: Perform OTSU threshold segmentation on the WSI obtained in step 1 to distinguish the foreground and background in the image, and segment the foreground image in the marked area into small images of N×N pixels; Step 3: Use the HSV-based edge detection module to enhance the crack features in the image block obtained in step 2 to highlight the crack features in the artifact; Step 4: Perform data augmentation on the image patches obtained in step 3 to increase the generalization ability of the model and reduce overfitting; Step 5: Perform preliminary feature extraction and adjust the data dimension of the data obtained in step 4 to ensure that it can be adapted and input into the two processing branches respectively; Step 6: Input the data converted in step 5 into the dual-branch network, and obtain the final artifact fine-grained classification result through classification processing.
2. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as claimed in claim 1, characterized in that: In step 2, the whole slice is cut into pixel sizes suitable for mainstream convolutional neural network processing, and the process is as follows: Step 2.1: Use the OpenSlide module to read the pathological full-slice image and simultaneously obtain the relevant annotation data; Step 2.2: Use the OTSU threshold segmentation algorithm to process the image to identify and extract the foreground area in the image, that is, the part containing the tissue structure; Step 2.3: Generate a mask map of the annotated area based on the existing annotation information; then, perform a logical AND operation on the foreground area mask and the annotated area mask to accurately locate the tissue area within the annotated area; finally, evenly segment the selected tissue area according to the standard size of N×N pixels.
3. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as described in claim 1 or 2, characterized in that: In step 2, N×N selects 256×256, 512×512 or 1024×1024 pixels.
4. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as described in claim 1 or 2, characterized in that: In step 3, an HSV-based edge detection module is used to enhance the crack information in the image while ensuring that the color information of the image is not lost. The process is as follows: Step 3.1: Use the OpenCV library to load the image file from the specified path and perform color space conversion to convert the image from BGR to HSV color space; Step 3.2: In HSV color space, decompose the image into three independent channels: hue H, saturation S and brightness V, and apply Gaussian blur technique to the brightness channel; Step 3.3: Apply the Canny edge recognition algorithm to the brightness channel after Gaussian blur processing to accurately identify the edge lines in the image; Step 3.4: Use the edge obtained by Canny edge recognition as a mask to extract the edge area in the original image; Step 3.5: Perform a weighted merge of the original image and the image in the edge area to highlight the edge while retaining the color information of the original image.
5. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as described in claim 1 or 2, characterized in that: In step 4, data enhancement is used to improve the adaptability of the model to diverse image content. The process is as follows: Step 4.1: Perform a series of transformation operations on the image to achieve data augmentation, including rotation, horizontal or vertical flipping, scaling, cropping, and random brightness adjustment; Step 4.2: Convert the enhanced image into a 4D tensor format suitable for deep learning model input.
6. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as described in claim 1 or 2, characterized in that: In step 5, image features are initially extracted and data dimensions are adjusted so that the data can be adapted to two different network branches. The process is as follows: Step 5.1: Input the data obtained in step 4 into a stem module, which is composed of several convolutional layers. The core function of this module is to perform preliminary feature extraction and dimension conversion on the input image. After this step, the data will be ready to enter the CNN branch. Step 5.2: Perform further dimension conversion on the output data obtained in step 5.1 and convert it into a three-dimensional data format suitable for the Transformer branch.
7. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as described in claim 1 or 2, characterized in that: In step 6, a dual-branch network structure is used to simultaneously capture local details and global context information of the image, and the recognition ability of the model is improved through feature fusion and interaction. The process is as follows: Step 6.1: Input the data processed in step 5 into the CNN branch and the Transformer branch respectively. These two branches are implemented using the MobileNet and MG-former architectures respectively. Step 6.2: Perform feature extraction in the MobileNet branch and the MG-former branch respectively. The MG-former module is a neural network module optimized on the basis of Transformer, which consists of an improved feed-forward layer MobileGLU and linear focus attention. Step 6.3: While the MobileNet branch and the MG-former branch are learning features independently, the feature interaction between the two branches is realized through the FLCA module. The FLCA module receives two sets of input features: when FLCA is the first FLCA module from top to bottom, the current branch features correspond to the features of the MG-former branch, and the other branch features correspond to the features of the MobileNet branch. Each set of features is first batch normalized; then, the current branch features are linearly projected W Q Generate the query vector Q, and the other branch is linearly projected W K , W V Generate the key vector K and the value vector V, and then apply the focus similarity (F) to Q and K to calculate the attention weight. The focus similarity calculation process is shown in formula (1). In formula (2), x **p is to perform a p-power operation on each element in x, focusing on the function f p It can help focus on similar features. The norm of the features before and after mapping is the same, that is, ||x||=||f p (x)||, indicating that the function only adjusts the feature direction; then the obtained attention weight is multiplied by V element by element, and then added element by element with Q processed by deep convolution DWC to generate the updated current branch feature; in, Step 6.4: Finally, the features of the two branches are fused to generate a comprehensive feature vector, and the final classification result is output based on it.
8. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network as claimed in claim 7, characterized in that: In the step 6.2, the MG-former module is composed of a MobileGLU module and a linear focus attention, wherein BN is a batch normalization operation, DWC is a deep convolution, gelu is an activation function, LN is a layer normalization operation, and FL-Att is a linear focus attention mechanism; the MobileGLU processing flow is as follows: first, a batch normalization operation is performed on the input feature map, and then channel mixing is performed through a 1*1 convolution layer, and a GELU activation function is applied to increase nonlinearity; thereafter, the feature map is further subjected to a 3*3 deep convolution to further extract spatial features, and is again subjected to batch normalization and GELU activation; next, a matrix multiplication operation is performed on the processed feature map and the initial input feature map after dimension transformation, and the result is processed through a 1*1 convolution layer and then added to the initial input feature map; after MobileGLU, layer normalization and a linear focus attention mechanism are applied to enhance feature representation capabilities and generate the final feature map.
Citation Information
Patent Citations
Method and apparatus for detecting dark noise artifacts
CN102119400A
Anterior segment disease multi-label classification method based on double-branch cross attention pooling
CN116797522A
Pathological image classification method and device, electronic equipment and storage medium
CN117409233A
Method for automatic detection of data in in-vivo images
US20100080460A1
Cited By
Pathological image preprocessing and feature extraction method
CN119964154A
Image bubble noise removal method and system based on dual-channel characteristics
CN120219230A
Colorectal cancer HER2 state classification method based on double-branch collaborative optimization network
CN120451639A
Colorectal cancer her2 status classification method based on double-branch cooperative optimization network
CN120451639B
Crack identification method and device based on graph convolutional network and hyperspectral imaging
CN121095258A
