Pathological image artifact fine-grained classification method based on double-branch fusion network

Abstract

A kind of pathological image artifact fine-grained classification method based on double-branch fusion network, comprising the following steps: step 1: first, obtain the whole slice WSI of tissue pathological image with artifact, and the artifact area is annotated by pathologist, to obtain the annotation data of artifact area;Step 2: the WSI is subjected to OTSU threshold segmentation operation to distinguish the foreground and background in image, and the foreground image in the annotation area is segmented into N×N pixel small graph;Step 3: the crack feature in the image block is strengthened using the edge detection module based on HSV to highlight the crack feature in artifact;Step 4: the image block is subjected to data enhancement;Step 5: preliminary feature extraction and data dimension adjustment are carried out;Step 6: the data is input into double-branch network, and the final artifact fine-grained classification result is obtained through classification processing.The present application realizes fine-grained classification of similar artifacts on pathological image, and improves artifact recognition precision.

Description

Technical Field

[0001] The present invention belongs to the fields of digital image processing, histopathological image artifact recognition and computer-aided technology, and relates to a fine-grained classification method for pathological image artifacts. Background Art

[0002] Histopathology image analysis provides precise pathological information for tumor diagnosis and treatment and is recognized by experts in the field as the gold standard for tumor diagnosis. Before digitization, histopathology images undergo complex steps, including specimen collection, cutting, and staining. This process is cumbersome, has a low degree of automation, and is subject to subjective manipulation by pathologists, making it prone to unexpected defects (referred to as "artifacts"). Artifacts are primarily categorized into three types: 1) tissue-level artifacts, such as knife marks and loose sections; 2) slice-level artifacts, such as bubbles and ink; and 3) scan-level artifacts, such as loss of focus. These artifacts can prevent pathologists from observing and assessing true lesions while simultaneously overlooking normal tissue, leading to misdiagnosis or missed diagnosis, and thus compromising patient treatment and prognosis. Furthermore, histopathology images with artifacts can negatively impact computer-aided diagnosis systems. Therefore, artifact detection in completed histopathology images is crucial.

[0003] Currently, there are three main approaches to artifact detection in pathology images: manual quality inspection, machine learning, and deep learning. Manual quality inspection is the most commonly used method in medical institutions. Due to the large number of tissue pathology slides prepared daily, clinical pathology departments can only perform manual spot checks on a certain percentage of slides, which cannot cover all newly prepared slides. Furthermore, due to the lack of unified evaluation criteria, the judgment results may vary between different doctors. In contrast, machine learning and deep learning methods provide a more efficient and objective solution. Early artifact detection technologies mostly used machine learning algorithms, which often rely on image color features or artificially designed features. Methods based on color features are effective in identifying artifacts with obvious color differences (such as tissue folds and ink stains), but their detection performance is insufficient when the color difference is subtle. Methods based on artificially designed features not only increase the complexity of algorithm design but also make it difficult to ensure the universality of the designed features.

[0004] Deep learning methods offer a more adaptive solution. Artifact recognition methods based on deep learning have significantly surpassed traditional machine learning approaches in terms of accuracy and generalization. Many researchers have applied convolutional neural networks (CNNs) to artifact recognition in histopathology images, using models such as ResNet101, YOLOV5, and DenseNet121 as feature extractors. These feature extractors excel at capturing local features such as color and texture, but they lack the ability to effectively process global information, such as macroscopic topological structures over large areas. To address this issue, some researchers have attempted to introduce attention mechanisms. However, these attention mechanisms are limited by their receptive field size, which means they are limited in processing global information and, consequently, their performance gains. In recent years, the Transformer model has made significant progress in the field of vision. Its core self-attention mechanism gives the model a wide global receptive field, significantly enhancing its ability to process image information. Guo et al. introduced a multi-head attention mechanism for artifact classification, but the local and global information were simply combined at the end of the model, resulting in very limited performance gains.

[0005] Existing research has achieved good results in identifying and distinguishing artifacts with large morphological and color differences, such as ink, tissue folding, and blur. However, there are many types of artifacts, and some artifacts have high similarity, such as Figure 1 Four artifacts are shown. Observing the local areas marked by red boxes, we can find that they have significant similarities: irregular cracks are distributed in the tissues of these areas. If the model only focuses on these details, it will be difficult to distinguish different artifacts. However, observing the overall distribution of cracks, we can find that there are obvious differences between them. For example, loose cracks are distributed more densely and in inconsistent directions; cracks in fissures mostly extend in the same direction. Therefore, in order to effectively distinguish these artifacts, the model needs to remain sensitive to the local characteristics of the defects while also understanding their overall pattern and structure. However, existing deep learning methods often find it difficult to fully pay attention to their global information when distinguishing these similar artifacts, which urgently requires us to conduct more in-depth research.

[0006] With the rapid advancement of deep learning technology in the field of computer vision, convolutional neural networks (CNNs) have made significant progress in the application of artifact recognition in pathological images. Foucart et al. used a deep residual network to extract information from coarse labels for artifact recognition and segmentation in hematoxylin-eosin (H&E) and immunohistochemistry (IHC) whole slide images (WSIs). Haghighat et al. used a multi-task deep neural network based on ResNet18 to evaluate the quality of WSI block by block at 5x magnification. However, the network models used in these studies, such as deep residual networks and ResNet18, have certain limitations in processing global information.

[0007] To overcome this challenge, Li et al. proposed a two-stage CNN method that effectively weighted the features of different channels by introducing a channel attention mechanism and an attention residual block. However, this method does not solve the problem of limited receptive field. To this end, Shakarami et al. proposed a new CNN architecture, TCNN, which enhances the model's ability to capture the global contextual information of the image by applying dilated convolutions in the last few layers of the convolutional layer. Pizarro et al. (2023) stacked multiple convolutional layers to increase its receptive field layer by layer. The above CNN-based improvements have, to a certain extent, broadened the network's receptive field, enabling it to capture global information more effectively. However, the receptive field of CNN is still limited by the size of the convolution kernel and the depth of the network architecture. In particular, as the network depth increases, the risk of losing detailed information also increases. These are issues that need to be further addressed in current research.

[0008] With the introduction of the Vision Transformer (ViT), the application of the Transformer architecture in the field of computer vision has attracted widespread attention. Thanks to the self-attention mechanism, the receptive field of the Transformer can immediately cover the entire input image. Guo et al. proposed the BIQA model. The innovation of this model lies in that it combines a saliency module based on DINet and an image block module (patch block) that uses multi-head self-attention and a pyramid position encoding generator. This module is specifically used to extract the mutual correlation and spatial information between image patches. The model combines CNN-based and Transformer-based network modules to process local artifact features and global artifact features respectively. However, the model only integrates local information and global information through simple splicing before the final classification head. This method fails to fully utilize the advantages of the combination of the two, resulting in limited performance improvement.

[0009] In response to this, Chen et al. and Han et al. successively proposed the Mobile-Former and Dyn-Perceiver. They both adopted a parallel design of CNN and Transformer, and achieved feature fusion between the two through a bidirectional bridge structure. However, the feed-forward network (FFN) used in the Transformer branch of Mobile-Former did not perform well in the parallel framework, and the input of the MobileNet sub-block depended on the output of the Transformer sub-block, which limited the independent parallel performance of the two branches. Dyn-Perceiver focuses more on dynamic early exit.

[0010] At present, all the mentioned artifact recognition technologies mainly distinguish artifacts with significant feature differences. However, for artifacts with high feature similarity, the existing recognition methods show limitations in their ability to distinguish. Among the current artifact recognition methods based on deep learning, some methods borrow strategies for color processing in the field of traditional machine learning, but compared with traditional methods, these deep learning methods have significantly reduced their dependence on color. For example, TCNN uses a color mapping layer in the initial layer, which converts the color space of the image from the RGB domain to the HSV domain, but this method only performs a simple color domain conversion. The AR-Classifier proposed by Ke et al. further utilizes complementary information from different color spaces (such as LAB, HED, and HSV), but it also leads to an exponential increase in the amount of computation. Summary of the Invention

[0011] To overcome the shortcomings of existing technologies, the present invention provides a fine-grained classification method for pathological image artifacts based on a dual-branch fusion network, which fully fuses and interacts the local and global information of artifacts to achieve fine-grained classification of similar artifacts on pathological images. This dual-branch architecture consists of two key parts: a convolutional neural network (CNN) branch and a visual transformer (Transformer) branch. The former is used to fully extract local information of texture and color, and the latter is used to capture global information such as the overall distribution of cracks. On the basis of this architecture, an HSV-based edge detection module is constructed to enhance the network's attention to crack features. In addition, in order to further enhance the interaction and fusion of local and global information, a cross-attention mechanism is designed. This mechanism can promote the effective integration of information between different regions, thereby improving the model's recognition accuracy for similar artifacts.

[0012] The technical solution adopted by the present invention to solve its technical problem is:

[0013] A fine-grained classification method for pathological image artifacts based on a dual-branch fusion network includes the following steps:

[0014] Step 1: First, obtain a full-slice WSI of a tissue pathology image with artifacts, and have a pathologist professionally annotate the artifact area to obtain the annotated data of the artifact area;

[0015] Step 2: Perform OTSU threshold segmentation on the WSI obtained in step 1 to distinguish the foreground and background in the image, and segment the foreground image in the marked area into small images of N×N pixels;

[0016] Step 3: Use the HSV-based edge detection module to enhance the crack features in the image block obtained in step 2 to highlight the crack features in the artifact;

[0017] Step 4: Perform data augmentation on the image patches obtained in step 3 to increase the generalization ability of the model and reduce overfitting;

[0018] Step 5: Perform preliminary feature extraction and adjust the data dimension of the data obtained in step 4 to ensure that it can be adapted and input into the two processing branches respectively;

[0019] Step 6: Input the data converted in step 5 into the two-branch network and obtain the final artifact fine-grained classification result through classification processing.

[0020] Furthermore, in step 2, the full slice is divided into pixel sizes suitable for mainstream convolutional neural network processing, and the process is as follows:

[0021] Step 2.1: Use the OpenSlide module to read the full pathology slide image and simultaneously obtain the relevant annotation data. This step ensures that the image data and its corresponding annotation information can be accurately loaded, laying the foundation for subsequent processing;

[0022] Step 2.2: Use the OTSU threshold segmentation algorithm to process the image to identify and extract the foreground area in the image, that is, the part containing tissue structure;

[0023] Step 2.3: Generate a mask of the annotated area based on the existing annotation information. Then, perform a logical union operation on the foreground area mask and the annotated area mask to accurately locate the tissue area within the annotated area. Finally, evenly segment the selected tissue area according to the standard size of N×N pixels.

[0024] Preferably, in step 2, N×N selects 256×256, 512×512 or 1024×1024 pixels.

[0025] Furthermore, in step 3, an HSV-based edge detection module is used to enhance the crack information in the image while ensuring that the color information of the image is not lost. The process is as follows:

[0026] Step 3.1: Use the OpenCV library to load the image file from the specified path and perform color space conversion to convert the image from BGR to HSV color space. This conversion helps to more intuitively process the hue, saturation, and brightness information of the image.

[0027] Step 3.2: Decompose the image into three independent channels in the HSV color space: hue (H), saturation (S), and brightness (V). Apply a Gaussian blur technique to the brightness channel to smooth the image and reduce noise interference, creating a clearer background for subsequent edge recognition.

[0028] Step 3.3: Apply the Canny edge detection algorithm to the brightness channel after Gaussian blur processing to accurately identify the edge lines in the image;

[0029] Step 3.4: Use the edge obtained by Canny edge recognition as a mask to extract the edge area in the original image;

[0030] Step 3.5: Perform a weighted merge of the original image and the image in the edge area to highlight the edge while retaining the color information of the original image.

[0031] Furthermore, in step 4, data augmentation is used to improve the model's adaptability to diverse image content. The process is as follows:

[0032] Step 4.1: Perform a series of transformations on the image to achieve data augmentation, including rotation, horizontal or vertical flipping, scaling, cropping, and random brightness adjustment.

[0033] Step 4.2: Convert the enhanced image into a four-dimensional tensor format suitable for deep learning model input. This conversion ensures that the image data can be effectively processed and analyzed by the model.

[0034] In step 5, image features are initially extracted and the data dimension is adjusted so that the data can be adapted to two different network branches. The process is as follows:

[0035] Step 5.1: Input the data obtained in step 4 into a stem module, which consists of several convolutional layers. The core function of this module is to perform preliminary feature extraction and dimension conversion on the input image. After this step, the data will be ready to enter the CNN branch.

[0036] Step 5.2: Perform further dimensionality conversion on the output data obtained in step 5.1, converting it into a three-dimensional data format suitable for the Transformer branch. This conversion is to meet the Transformer branch's requirements for specific dimensions of input data and ensure that the data can be correctly fed into the Transformer network for processing.

[0037] In step 6, a dual-branch network structure is used to simultaneously capture local details and global context information of the image, and the recognition ability of the model is improved through feature fusion and interaction. The process is as follows:

[0038] Step 6.1: Input the data processed in step 5 into the CNN branch and the Transformer branch respectively. These two branches are implemented using the MobileNet and MG-former (MobileGLU-Transformer) architectures respectively.

[0039] Step 6.2: Perform feature extraction in both the MobileNet branch and the MG-former branch. The MG-former module is a neural network module optimized on the basis of the Transformer, consisting of an improved feed-forward layer MobileGLU and a linear focused attention.

[0040] Step 6.3: While the MobileNet branch and the MG-former branch are learning features independently, the feature interaction between the two branches is realized through the FLCA module. The FLCA module receives two sets of input features: when FLCA is the first FLCA module from top to bottom, the current branch features correspond to the features of the MG-former branch, and the other branch features correspond to the features of the MobileNet branch. Each set of features is first batch normalized; then, the current branch features are linearly projected W Q Generate query vector Q, the other branch is linearly projected W K 、W V Generate the key vector K and the value vector V, and then apply the focus similarity (F) to Q and K to calculate the attention weight. The focus similarity calculation process is shown in formula (1). In formula (2), x **p It performs a p-th power operation on each element in x, focusing on the function f p It can help focus on similar features. The norm of the features before and after mapping is the same, that is, ||x||=||f p (x)||, indicating that the function only adjusts the feature direction; then the obtained attention weight is multiplied by V element-wise, and then added element-wise to Q after deep convolution DWC processing to generate the updated current branch feature. This process not only enhances the feature expression ability, but also promotes information interaction between the two branches;

[0041]

[0042] in,

[0043] Step 6.4: Finally, the features of the two branches are fused (concatenation) to generate a comprehensive feature vector, and the final classification result is output based on it.

[0044] In step 6.2, the MG-former module is composed of a MobileGLU module and a linear focused attention. In the MG-former module, BN is a batch normalization operation, DWC (depthwise conv) is a depthwise convolution, gelu is an activation function, LN is a layer normalization operation, and FL-Att (Focused Linear Attention) is a linear focused attention mechanism. The MobileGLU processing flow is as follows: first, the input feature map is batch normalized, and then channel mixing is performed through a 1*1 convolution layer, and a GELU activation function is applied to increase nonlinearity. After that, the feature map is further subjected to a 3*3 depthwise convolution to extract spatial features, and is again subjected to batch normalization and GELU activation. Next, the processed feature map is matrix multiplied with the initial input feature map after dimension transformation, and is processed through a 1*1 convolution layer and added to the initial input feature map. After MobileGLU, layer normalization and a linear focused attention mechanism are applied to enhance feature representation capabilities and generate the final feature map.

[0045] The technical concept of this invention is to adopt the idea of ​​a parallel framework, but optimize the interaction timing between the CNN branch and the Transformer branch. To deepen the interaction and fusion of features, a linear focused cross attention module, FocusedLinearCrossAttention (FLCA), is designed to improve the efficiency and effectiveness of the feature interaction mechanism. In addition, to enhance the feature extraction capability of the Transformer branch in the parallel framework, MobileGLU is designed based on the inverted residual structure and gating mechanism (GLU) in MobileNet2 to replace the feedforward layer in the traditional self-attention mechanism, in order to achieve better performance.

[0046] Among the artifacts to be distinguished in this paper, the significant color difference between cracks and surrounding tissues is the key point of distinction. An HSV-based edge detection module is proposed. This module first converts the image from the RGB domain to the HSV domain and then introduces the Canny edge detector to enhance the boundary information between cracks and normal tissue in the image, thereby improving the ability to distinguish artifacts.

[0047] To address the complexity of similar artifacts in pathological images, a dual-branch fusion network structure was introduced and improved. This structure optimizes the integration of local and global information, allowing the model to capture image details while also gaining a deep understanding of its overall pattern, significantly improving classification accuracy. Furthermore, a designed HSV-based edge detection module significantly enhances the model's ability to recognize subtle features such as cracks, while ensuring the integrity of image color information and maintaining the model's high sensitivity to artifacts. This effectively addresses the shortcomings of existing technologies in fine-grained classification of similar artifacts, providing a more efficient and accurate tool for artifact recognition in tissue pathology images.

[0048] The beneficial effects of the present invention are mainly manifested in: fully fusing and interacting the local and global information of artifacts to achieve fine-grained classification of similar artifacts on pathological images and improve artifact recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The following are examples of artifacts. From top to bottom, the types of artifacts in each row are loose, tissue bubbles, cracks, and knife marks.

[0050] Figure 2 It's a flow chart.

[0051] Figure 3 It is a two-branch network diagram.

[0052] Figure 4 This is a schematic diagram of the MG-former module.

[0053] Figure 5 This is the cross-attention FLCA module diagram. DETAILED DESCRIPTION

[0054] The present invention will be further described below with reference to the accompanying drawings.

[0055] Reference Figures 2 to 5 , a fine-grained classification method for pathological image artifacts based on a dual-branch fusion network, comprising the following steps:

[0056] Step 1: First, a whole slide image (WSI) of tissue pathology images with artifacts is obtained, and a pathologist professionally annotates the artifact area to obtain the annotated data of the artifact area;

[0057] Step 2: Perform OTSU's Thresholding on the WSI obtained in step 1 to distinguish the foreground and background in the image, and segment the foreground image in the marked area into small images of 256 × 256 pixels.

[0058] Optionally, the size of the small image cut in step 2 can be selected to be 512×512, 1024×1024 pixels, etc.

[0059] In step 2, the full slice is divided into pixel sizes suitable for mainstream convolutional neural network processing. The process is as follows:

[0060] Step 2.1: Use the OpenSlide module to read the full pathology slide image and simultaneously obtain the relevant annotation data. This step ensures that the image data and its corresponding annotation information can be accurately loaded, laying the foundation for subsequent processing;

[0061] Step 2.2: Use the OTSU threshold segmentation algorithm to process the image to identify and extract the foreground area in the image, that is, the part containing tissue structure;

[0062] Step 2.3: Generate a mask for the annotated region based on the existing annotation information. Then, perform a logical union between the foreground mask and the annotated mask to precisely locate the tissue region within the annotated region. Finally, evenly segment the selected tissue region according to the standard 256×256 pixel size.

[0063] Step 3: Use the HSV-based edge detection module to enhance the crack features in the image block obtained in step 2 to highlight the cracks and other features in the artifacts;

[0064] In step 3, an HSV-based edge detection module is used to enhance the crack information in the image while ensuring that the color information of the image is not lost. The process is as follows:

[0065] Step 3.1: Use the OpenCV library to load the image file from the specified path and perform color space conversion, converting the image from BGR to HSV color space. This conversion helps to more intuitively process the hue, saturation, and brightness information of the image;

[0066] Step 3.2: Decompose the image into three independent channels in the HSV color space: hue (H), saturation (S), and brightness (V). Apply a Gaussian blur technique to the brightness channel to smooth the image and reduce noise interference, creating a clearer background for subsequent edge recognition.

[0067] Step 3.3: Apply the Canny edge detection algorithm to the brightness channel after Gaussian blur processing to accurately identify the edge lines in the image;

[0068] Step 3.4: Use the edge obtained by Canny edge recognition as a mask to extract the edge area in the original image;

[0069] Step 3.5: Perform a weighted merge of the original image and the image in the edge area to highlight the edge while retaining the color information of the original image.

[0070] Step 4: Perform data augmentation on the image patches obtained in step 3 to increase the generalization ability of the model and reduce overfitting;

[0071] In step 4, the purpose is to improve the model's adaptability to diverse image content through data augmentation. The process is as follows:

[0072] Step 4.1: Perform a series of transformation operations on the image to achieve data augmentation, including but not limited to rotation, horizontal or vertical flipping, scaling, cropping, and random brightness adjustment;

[0073] Step 4.2: Convert the enhanced image into a four-dimensional tensor format suitable for deep learning model input. This conversion ensures that the image data can be effectively processed and analyzed by the model.

[0074] Step 5: Perform preliminary feature extraction and adjust the data dimension of the data obtained in step 4 to ensure that it can be adapted and input into the two processing branches respectively;

[0075] In step 5, the purpose is to preliminarily extract image features and adjust the data dimension so that the data can adapt to two different network branches. The process is as follows:

[0076] Step 5.1: Input the data from step 4 into a stem module, which consists of several convolutional layers. The core function of this module is to perform preliminary feature extraction and dimensionality conversion on the input image. After this step, the data will be ready to enter the CNN branch.

[0077] Step 5.2: Perform further dimensionality conversion on the output data obtained in step 5.1, converting it into a three-dimensional data format suitable for the Transformer branch. This conversion is to meet the Transformer branch's requirement for specific dimensionality of the input data, ensuring that the data can be correctly fed into the Transformer network for processing.

[0078] Step 6: Input the data converted in step 5 into the two-branch network and obtain the final artifact fine-grained classification result through classification processing.

[0079] In step 6, a dual-branch network structure is used to capture both local details and global context information of the image, and the recognition capability of the model is improved through feature fusion and interaction. Figure 3 As shown, the process is as follows:

[0080] Step 6.1: Input the data processed in step 5 into the CNN branch and the Transformer branch respectively. In the present invention, these two branches are implemented using the MobileNet and MG-former (MobileGLU-Transformer) architectures respectively.

[0081] Step 6.2: Perform feature extraction in both the MobileNet branch and the MG-former branch. The MG-former module is a neural network module optimized on the basis of Transformer, mainly composed of an improved feedforward layer MobileGLU and a linear focused attention layer. Figure 4 As shown in the figure, in this module, BN is batch normalization operation, DWC (depthwise conv) is depth convolution, gelu is activation function, LN is layer normalization operation, and FL-Att (Focused Linear Attention) is linear focused attention mechanism.

[0082] The MobileGLU processing flow is as follows: First, the input feature map is batch normalized, followed by channel mixing through a 1*1 convolutional layer, and the GELU activation function is applied to increase nonlinearity. The feature map then undergoes a 3*3 depthwise convolution to further extract spatial features, and is again batch normalized and activated with the GELU. Next, the processed feature map is matrix multiplied with the transformed initial input feature map, processed through a 1*1 convolutional layer, and then added to the initial input feature map. After the MobileGLU module, layer normalization and a linear focused attention mechanism are applied to enhance feature representation and generate the final feature map.

[0083] Step 6.3: While the MobileNet branch and the MG-former branch are learning features independently, the FLCA module is used to implement feature interaction between the two branches, such as Figure 5 As shown, the FLCA module receives two sets of input features: Figure 3 When the first FLCA module from top to bottom in [1], the current branch features correspond to the features of the MG-former branch, while the other branch features correspond to the features of the MobileNet branch. Each set of features is first batch normalized. Subsequently, the current branch features are linearly projected by W Q Generate query vector Q, the other branch is linearly projected W K 、W V Generate the key vector K and the value vector V, and then apply the focus similarity (F) to Q and K to calculate the attention weight. The focus similarity calculation process is shown in formula (1). In formula (2), x**p It performs a p-th power operation on each element in x, focusing on the function f p It can help focus on similar features. The norm of the features before and after mapping is the same, that is, ||x||=||f p (x)|| indicates that this function only adjusts the feature direction. The resulting attention weights are then element-wise multiplied by V and added element-wise to Q after the depthwise convolution (DWC) process to generate the updated features of the current branch. This process not only enhances the expressive power of the features but also promotes information exchange between the two branches.

[0084]

[0085] in,

[0086] Step 6.4: Finally, the features of the two branches are fused (concatenation) to generate a comprehensive feature vector, and the final classification result is output based on it.

[0087] The solution of this embodiment introduces a dual-branch architecture that fuses CNN and Transformer to address the characteristics of artifacts. To improve the feature extraction performance of the Transformer branch, the present invention designs a MobileGLU based on an inverted residual structure and a gated mechanism GLU to replace the traditional feedforward layer. This is combined with linear focus attention to form an MG-former to replace the Transformer, significantly improving the recognition accuracy of the model. To enhance the interaction and fusion of local and global information, the present invention designs a linear focus cross attention mechanism FLCA. This attention mechanism promotes the effective integration of information between different regions by introducing a linear focus function in the cross attention. To enhance the network's attention to crack features, an HSV-based edge detection module is constructed. This module performs edge recognition while converting the color domain to the HSV color space, significantly enhancing the model's ability to recognize subtle features such as cracks while preserving image color information.

[0088] The embodiments of this specification are merely examples of implementations of the invention and are provided for illustrative purposes only. The scope of protection of the present invention should not be considered limited to the specific embodiments described in these embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by a person of ordinary skill in the art based on the invention.

Claims

1. A fine-grained classification method for pathological image artifacts based on a dual-branch fusion network, characterized in that: The method comprises the following steps: Step 1: First, obtain a full-slice WSI of a tissue pathology image with artifacts, and have a pathologist professionally annotate the artifact area to obtain the annotated data of the artifact area; Step 2: Perform OTSU threshold segmentation on the WSI obtained in step 1 to distinguish the foreground and background in the image, and segment the foreground image in the marked area into small images of N×N pixels; Step 3: Use the HSV-based edge detection module to enhance the crack features in the image block obtained in step 2 to highlight the crack features in the artifact; Step 4: Perform data augmentation on the image patches obtained in step 3 to increase the generalization ability of the model and reduce overfitting; Step 5: Perform preliminary feature extraction and adjust the data dimension of the data obtained in step 4 to ensure that it can be adapted and input into the two processing branches respectively; Step 6: Input the data converted in step 5 into the two-branch network and obtain the final artifact fine-grained classification result through classification processing; In step 6, a dual-branch network structure is used to simultaneously capture local details and global context information of the image, and the recognition ability of the model is improved through feature fusion and interaction. The process is as follows: Step 6.1: Input the data processed in step 5 into the CNN branch and the Transformer branch respectively. These two branches are implemented using the MobileNet and MG-former architectures respectively. Step 6.2: Perform feature extraction in both the MobileNet branch and the MG-former branch. The MG-former module is a neural network module optimized on the basis of the Transformer, consisting of an improved feed-forward layer MobileGLU and linear focus attention. Step 6.3: While the MobileNet branch and the MG-former branch are learning features independently, the feature interaction between the two branches is realized through the FLCA module. The FLCA module receives two sets of input features: when FLCA is the first FLCA module from top to bottom, the current branch features correspond to the features of the MG-former branch, and the other branch features correspond to the features of the MobileNet branch. Each set of features is first batch normalized; then, the current branch features are linearly projected W Q Generate query vector Q, the other branch is linearly projected W K 、W V Generate the key vector K and the value vector V, and then apply the focus similarity (F) to Q and K to calculate the attention weight. The focus similarity calculation process is shown in formula (1). In formula (2), x **p Yes x Each element in p Power operation, focus function f p It can help focus on similar features, and the norm of the features before and after mapping is the same, that is, || x ||=|| f p ( x )||, indicating that the function only adjusts the feature direction; then the obtained attention weight is multiplied by V element by element, and then added element by element with Q after deep convolution DWC processing to generate the updated current branch feature; , (1) in, , (2) Step 6.4: Finally, the features of the two branches are fused to generate a comprehensive feature vector, and the final classification result is output based on it.

2. The fine-grained classification method for pathological image artifacts based on a dual-branch fusion network according to claim 1, characterized in that: In step 2, the full slice is divided into pixel sizes suitable for mainstream convolutional neural network processing. The process is as follows: Step 2.1: Use the OpenSlide module to read the pathology full slide image and simultaneously obtain the relevant annotation data; Step 2.2: Use the OTSU threshold segmentation algorithm to process the image to identify and extract the foreground area in the image, that is, the part containing tissue structure; Step 2.3: Generate a mask of the annotated area based on the existing annotation information. Then, perform a logical union operation on the foreground area mask and the annotated area mask to accurately locate the tissue area within the annotated area. Finally, evenly segment the selected tissue area according to the standard size of N×N pixels.

3. The fine-grained classification method for pathological image artifacts based on a dual-branch fusion network according to claim 1 or 2, characterized in that: In step 2, N×N selects 256×256, 512×512 or 1024×1024 pixels.

4. The fine-grained classification method for pathological image artifacts based on a dual-branch fusion network according to claim 1 or 2, characterized in that: In step 3, an HSV-based edge detection module is used to enhance the crack information in the image while ensuring that the color information of the image is not lost. The process is as follows: Step 3.1: Use the OpenCV library to load the image file from the specified path and perform color space conversion to convert the image from BGR to HSV color space; Step 3.2: In the HSV color space, decompose the image into three independent channels: hue H, saturation S, and brightness V. Apply a Gaussian blur technique to the brightness channel. Step 3.3: Apply the Canny edge detection algorithm to the brightness channel after Gaussian blur processing to accurately identify the edge lines in the image; Step 3.4: Use the edge obtained by Canny edge recognition as a mask to extract the edge area in the original image; Step 3.5: Perform a weighted merge of the original image and the image in the edge area to highlight the edge while retaining the color information of the original image.

5. The fine-grained classification method for pathological image artifacts based on a dual-branch fusion network according to claim 1 or 2, characterized in that: In step 4, data augmentation is used to improve the model's adaptability to diverse image content. The process is as follows: Step 4.1: Perform a series of transformations on the image to achieve data augmentation, including rotation, horizontal or vertical flipping, scaling, cropping, and random brightness adjustment. Step 4.2: Convert the augmented image into a 4D tensor format suitable for deep learning model input.

6. The method for fine-grained classification of pathological image artifacts based on a dual-branch fusion network according to claim 1 or 2, characterized in that: In step 5, image features are initially extracted and the data dimension is adjusted so that the data can be adapted to two different network branches. The process is as follows: Step 5.1: Input the data obtained in step 4 into a stem module, which consists of several convolutional layers. The core function of this module is to perform preliminary feature extraction and dimension conversion on the input image. After this step, the data will be ready to enter the CNN branch. Step 5.2: Perform further dimension conversion on the output data obtained in step 5.1 and convert it into a three-dimensional data format suitable for the Transformer branch.

7. The fine-grained classification method for pathological image artifacts based on a dual-branch fusion network according to claim 1, characterized in that: In step 6.2, the MG-former module is composed of a MobileGLU module and a linear focus attention module, where BN is a batch normalization operation, DWC is a depthwise convolution, gelu is an activation function, LN is a layer normalization operation, and FL-Att is a linear focus attention mechanism. The MobileGLU processing flow is as follows: first, the input feature map is batch normalized, and then channel mixing is performed through a 1*1 convolution layer, and a GELU activation function is applied to increase nonlinearity. After that, the feature map is further subjected to a 3*3 depthwise convolution to extract spatial features, and is again subjected to batch normalization and GELU activation. Next, the processed feature map is matrix multiplied with the initial input feature map after dimension transformation, and is processed through a 1*1 convolution layer and added to the initial input feature map. After MobileGLU, layer normalization and a linear focus attention mechanism are applied to enhance feature representation capabilities and generate the final feature map.

Images