Pathological image virtual restaining method and device based on global-local feature fusion

Abstract

The application provides a kind of global-local feature fusion pathological image virtual restaining method, with such characteristics, comprising the following steps: step S1, construct the pathological image virtual restaining network including global branch and local branch, then according to existing pathological image, the pathological image virtual restaining network is trained, and the trained pathological image virtual restaining network is obtained as pathological image virtual restaining model;Step S2, the staining section image is input into pathological image virtual restaining model, and the virtual restaining image is obtained.In short, the method can better solve the block effect, so that the virtual restaining image is closer to the real staining effect.

Description

Technical Field

[0001] This invention belongs to the field of intelligent analysis technology of pathological images, specifically relating to a method and apparatus for virtual restaining of pathological images through global-local feature fusion. Background Technology

[0002] Pathological examination is the gold standard for disease diagnosis, especially cancer diagnosis. By staining pathological sections, different dyes can interact with different components of tissues or cells, making it easier to distinguish different microstructures, abnormal substances, and lesions. Among various staining methods, hematoxylin-eosin (HE) staining is the most common and fundamental. However, HE staining results alone are often insufficient for accurate diagnosis. Therefore, immunohistochemical staining (IHC), based on the specific binding of antigens and antibodies, is needed to assist in diagnosis, such as helping doctors eliminate uncertainties in slide reading. Generally, cancer diagnosis requires the mutual assistance of multiple IHC staining results, making it complex, time-consuming, and expensive. Due to the limitations of IHC, virtual restaining technology aims to generate another type of virtual restaining image from one stained slide image (usually HE). This image can accurately depict the corresponding type of staining, achieving a high degree of similarity to real staining. Based on this, the actual staining process can be eliminated, reducing the consumption of materials, money, and time in pathological diagnosis.

[0003] In recent years, a considerable amount of work has been dedicated to researching high-fidelity virtual restaining methods. Virtual restaining is similar to style transfer in natural images, thus style transfer methods can be applied to virtual restaining. Since pathological images are usually unpaired, virtual restaining generally employs unsupervised methods. Current research, based on advanced style transfer models, has made some improvements according to the characteristics of pathological images, achieving good results. Its development can be roughly summarized as follows:

[0004] In the paper "Virtualization of tissue staining in digital pathology using an unsupervised deep learning approach" by Amal Lahiani, Jacob Gildenblat, Irina Klaman, Shadi Albarqouni, Nassir Navab, and Eldad Klaiman, in *European Congress on Digital Pathology*, pages 47–55, Springer, 2019," an unsupervised virtual restaining method based on CycleGAN was used to convert FAP-CK to Ki67-CD8, and an overlapping region was added during post-processing image patch stitching. Experiments showed that this method could generate relatively realistic FAP-CK images in most cases, and the reliability of the method was verified by quantitative comparison of tumor cells and FAP in real and generated images.

[0005] In the paper Xinyang Li, Guoxun Zhang, Hui Qiao, Feng Bao, Yue Deng, Jiamin Wu, Yangfan He, Jingping Yun, Xing Lin, Hao Xie, et al. Unsupervised content-preserving transformation for optical microscopy. Light: Science & Applications, 10(1):1–11, 2021., a saliency constraint loss method is proposed to locate image content, ensuring that the saliency map remains unchanged during cross-domain style transfer, thereby avoiding distortion of image content. This method can well preserve semantic information for further biomedical analysis and achieves stable and high-fidelity image transformation under different imaging conditions and modes.

[0006] In the paper "Unpaired stain transfer using pathology-consistent constrained generative adversarial networks" by Huting Liu, Baochang Zhang, Yiqing Liu, Anjia Han, Huijuan Shi, Tian Guan, and Yonghong He, IEEE Transactions on Medical Imaging, 40(8): 1977–1989, 2021," a pathological representation network was added to CycleGAN to mine the pathological representation heatmap of the input image. Pathological consistency constraints were introduced to compensate for the shortcomings of CycleGAN, ensuring that the generated image and the original image have the same pathological characteristics in different staining domains. Furthermore, structural consistency constraints and skip connections were added to further improve the model performance, achieving competitive results.

[0007] However, pathological images have their own characteristics compared to natural images. The resolution of pathological images is usually much higher than that of natural images. Due to the limited memory of GPUs, the entire image cannot be loaded into memory for comprehensive staining conversion at once. Most existing virtual restaining models address this issue by segmenting the WSI image into smaller image patches for training and inference, and then stitching these patches together into a WSI image through post-processing. This leads to differences in color and brightness between adjacent patches, a phenomenon known as the block effect. In fact, the block effect exists because block-based virtual restaining lacks global information, resulting in mismatches in hue, contrast, and brightness between adjacent patches, especially for regions with different tissue structures and boundary regions. Furthermore, since the staining of each image patch is independent, even image patches with similar tissue structures may have some color differences. Most existing research studies virtual restaining models at the image patch level, without considering global information, leading to severe block effects. However, some recent studies have attempted to address this problem, for example:

[0008] In the paper Amal Lahiani, Irina Klaman, Nassir Navab, Shadi Albarqouni, and Eldad Klaiman, "Seamless virtual wholeslide image synthesis and validation using perceptual embedding consistency," IEEE Journal of Biomedical and Health Informatics, 25(2):403–411, 2020, a perceptual embedding consistency loss was proposed to force the generator to learn features such as color, brightness, and contrast. Experiments have shown that this method can reduce the block effect to some extent. However, the perceptual embedding consistency loss is only calculated for each image patch and does not consider more global information, so its effect is still limited.

[0009] In the paper "Ultra-high-resolution unpaired stainless transformation via kernelized instance normalization" by Ming-Yang Ho, Min-Sheng Wu, and Che-Ming Wu, in European Conference on Computer Vision, pages 490–505, Springer, 2022, a method was proposed to correct the mean and standard deviation of the central image patch using the mean and standard deviation of the surrounding image patches, thereby reducing color differences between adjacent image patches. This method achieved good results on the public dataset ANHIR and private datasets. However, the parameters of this method are manually designed rather than learned, resulting in poor generalization and difficulty in directly applying it to other datasets.

[0010] In summary, since high-fidelity WSI images are required for clinical diagnosis, the block effect remains a critical challenge in virtual restaining tasks for pathological images. Existing methods all have certain limitations, either failing to consider global image information or exhibiting poor generalization. Summary of the Invention

[0011] This invention is made to solve the above-mentioned problems, and aims to provide a method and apparatus for virtual restaining of pathological images by fusing global and local features.

[0012] This invention provides a virtual restaining method for pathological images based on global-local feature fusion, used to virtually restain stained slide images to obtain corresponding virtual restaining images. It includes the following steps: Step S1, constructing a virtual restaining network for pathological images including global and local branches, and then training the virtual restaining network based on existing pathological images to obtain a trained virtual restaining network as a virtual restaining model; Step S2, inputting the stained slide image into the virtual restaining model to obtain the virtual restaining image. Specifically, the processing of the stained slide image by the virtual restaining model includes... The following steps are as follows: Step T1: Downsample the stained section image to obtain the overall image as the global feature map; Step T2: Segment the stained section image to obtain multiple image patches as local feature maps; Step T3: Perform convolution, spatial attention, and channel attention processing on the global feature map sequentially to obtain a new global feature map; Step T4: Perform convolution, spatial attention, channel attention, and image patch-level attention processing on the local feature map sequentially to obtain a new local feature map; Step T5: Segment and upsample the new global feature map according to the corresponding position of each image patch in the stained section image, and then concatenate the upsampled result with the new local feature map in the channel dimension to obtain the global feature map. - The local fusion feature map is used as the local feature map; Step T6: According to the corresponding position of each image block in the stained slice image, the new local feature map is concatenated and downsampled, and then the downsampled result is concatenated with the new global feature map in the channel dimension to obtain the local-global fusion feature map as the global feature map; Step T7: Steps T3 to T6 are repeated twice, and the global feature map and local feature map are subjected to residual processing respectively to obtain the global residual feature map and the local residual feature map respectively; Step T8: The global residual feature map is concatenated with the global feature map, and then upsampled to obtain the first global upsampled feature map; Step T9: The local residual feature map and the local feature map are concatenated... The images are stitched together and then upsampled to obtain a first local upsampled feature map; in step T10, the second local-global fusion feature map is stitched together with the first global upsampled feature map and then upsampled to obtain a second global upsampled feature map; in step T11, the second global-local fusion feature map is stitched together with the first local upsampled feature map and then upsampled to obtain a second local upsampled feature map; in step T12, the second global upsampled feature map, the second local upsampled feature map, and the first global-local fusion feature map are stitched together along the channel dimension to obtain a stitched feature map, and then the stitched feature map is output to obtain a virtual recolored image.

[0013] The virtual restaining method for pathological images with global-local feature fusion provided by the present invention may also have the following features: in step T1, the resolution of the overall image after downsampling is the same as the resolution of the image block, and the specific process of obtaining multiple image blocks in step T2 is as follows: the stained slide image with a resolution of H×W is sequentially divided into image blocks with a resolution of h×w, and there are k overlapping pixels between adjacent image blocks.

[0014] The virtual restaining method for pathological images based on global-local feature fusion provided by this invention may also have the following features: In steps T3 and T4, the specific process of spatial attention processing is as follows: For the input feature map of size (N,C,H,W), two corresponding spatial attention feature maps are obtained in the second dimension according to the average and max pooling methods, respectively. Then, the two spatial attention feature maps are concatenated in the channel dimension and then passed through a convolutional layer and a sigmoid function to obtain the final spatial attention feature map. Finally, the final spatial attention feature map is multiplied by the input feature map as the output of spatial attention processing.

[0015] The virtual restaining method for pathological images based on global-local feature fusion provided by this invention may also have the following features: In steps T3 and T4, the specific process of channel attention processing is as follows: For the input feature map of size (N,C,H,W), two corresponding channel attention feature maps are obtained in the third and fourth dimensions according to the averaging and max pooling methods, respectively. The two channel attention feature maps are then added together and then passed through the sigmoid function to obtain the final channel attention feature map. Finally, the final channel attention feature map is multiplied by the input feature map as the output of the channel attention processing.

[0016] The virtual restaining method for pathological images based on global-local feature fusion provided by this invention may also have the following features: In step T4, the specific process of image block-level attention processing is as follows: For the input feature map of size (N,C,H,W), two corresponding image block-level attention feature maps are obtained in the second, third, and fourth dimensions respectively according to the average and max pooling methods. Then, the two image block-level attention feature maps are concatenated in the channel dimension and then sequentially processed by convolution and the sigmoid function to obtain the final image block-level attention feature map. Finally, the final image block-level attention feature map is multiplied by the input feature map as the output of image block-level attention processing.

[0017] The virtual restaining method for pathological images with global-local feature fusion provided by the present invention may also have the following features: in steps T8 and T9, upsampling is to process the feature map obtained after stitching through an upsampling block, and the upsampling block includes a transposed convolutional layer, an instance normalization layer and a ReLU function.

[0018] The virtual restaining method for pathological images based on global-local feature fusion provided by this invention may also have the following feature: wherein, in step T12, the output head includes a convolutional layer and a Tanh function.

[0019] The virtual restaining method for pathological images based on global-local feature fusion provided by this invention may also have the following features: during the training process, the second global upsampled feature map and the first obtained local-global fusion feature map are sequentially passed through convolution and Tanh function to obtain a target domain generated image, which is used for calculating the loss function of the global branch; the second local upsampled feature map and the first obtained global-local fusion feature map are sequentially passed through convolution and Tanh function to obtain a target domain generated image patch, which is used for calculating the loss function of the local branch.

[0020] This invention also provides a virtual restaining device for pathological images based on global-local feature fusion, used to virtually restain stained slide images to obtain corresponding virtual restaining images. It features the following characteristics: an image input module for receiving stained slide images to be restained; a virtual restaining module for processing the stained slide images to obtain virtual restaining images; and a result display module for displaying the virtual restaining images. The virtual restaining module includes a global encoder submodule, a local encoder submodule, a global intermediate submodule, a local intermediate submodule, and a decoder submodule. The global encoder submodule performs a first convolution-attention processing on the downsampled stained slide image to obtain a first global feature map; performs local-global fusion on the first global feature map and the first local feature map, followed by a second convolution-attention processing to obtain a second global feature map; and performs local-global fusion on the second global feature map and the second local feature map, followed by a third convolution-attention processing to obtain a third global feature map. The local encoder submodule performs a first convolution-attention processing on multiple image blocks obtained after segmenting the stained slide image to obtain a third global feature map. A local feature map is obtained by performing global-local fusion on the first local feature map and the first global feature map, followed by a second convolution-attention process to obtain a second local feature map. A third local feature map is obtained by performing global-local fusion on the second local feature map and the second global feature map, followed by a third convolution-attention process. A global intermediate submodule performs local-global fusion on the third global feature map and the third local feature map, followed by residual processing to obtain a global residual feature map. A local intermediate submodule performs global-local fusion on the third global feature map and the third local feature map, followed by residual processing to obtain a local residual feature map. A decoder submodule upsamples the global residual feature map, the third global feature map, and the second global feature map to obtain a global upsampled feature map, and upsamples the local residual feature map, the third local feature map, and the second local feature map to obtain a local upsampled feature map. Finally, a virtual restaining image is obtained based on the first local feature map, the global upsampled feature map, and the local upsampled feature map. The pathological image virtual restaining model is constructed using any of the above global-local feature fusion methods for pathological images.

[0021] The role and effect of invention

[0022] The virtual restaining method and apparatus for pathological images based on global-local feature fusion according to the present invention, by constructing global and local branches, and extracting global and local features from global and local information through three convolutions and attention processing respectively, and by fusing global features and corresponding local features in the global branch to obtain new global features, and fusing local features and corresponding global features in the local branch to obtain new local features, and by fusing global and local information through multiple feature fusions, information between image patches is learned, thereby improving the correlation of color, brightness, and contrast between generated image patches. Therefore, the virtual restaining method and apparatus for pathological images based on global-local feature fusion of the present invention can better solve the block effect, thus making the virtual restaining image closer to the real staining effect. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating the structure and working principle of the virtual restaining model for pathological images in an embodiment of the present invention;

[0024] Figure 2 This is a flowchart illustrating the virtual restaining method for pathological images based on global-local feature fusion in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the process of processing stained slide images using a virtual restaining model for pathological images in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of image blocks obtained by segmenting a stained slide image in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the framework of the virtual restaining device for pathological images with global-local feature fusion in an embodiment of the present invention. Detailed Implementation

[0028] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the virtual restaining method and apparatus for pathological images based on global-local feature fusion.

[0029] The global-local feature fusion method for virtual restaining of pathological images in this embodiment uses a trained virtual restaining network for pathological images as a virtual restaining model to perform virtual restaining on pathological slide images to obtain virtual restaining images.

[0030] Figure 1 This is a schematic diagram illustrating the structure and working principle of the virtual restaining model for pathological images in an embodiment of the present invention.

[0031] like Figure 1 As shown, the virtual restaining model 200 for pathological images includes a global branch 210, a local branch 220, and an output head 230. The global branch 210 includes three convolutional + attention blocks 211, a residual block 212, and two upsampling blocks 213 connected in sequence. The local branch 220 includes three convolutional + attention blocks 221, a residual block 222, and two upsampling blocks 223 connected in sequence. Its working principle is as follows: the virtual restaining model 200 downsamples the stained slide image and uses the downsampled overall image as the input of the global branch 210. At the same time, the stained slide image is segmented, and the multiple image blocks obtained from the segmentation are used as the input of the local branch 220. Then, through feature fusion, the local information of the image blocks in the local branch 220 is integrated into the global branch 210, and the global information of the overall image in the global branch 210 is integrated into the local branch 220. Finally, the outputs of the global branch 210 and the local branch 220 are processed together through the output head 230 to obtain the virtual restaining image.

[0032] In this embodiment, the stained section images are pathological section images after hematoxylin-eosin staining, and the virtual restaining images are images of immunohistochemical staining based on the specific binding of antigens and antibodies.

[0033] Figure 2 This is a flowchart illustrating the virtual restaining method for pathological images using global-local feature fusion in an embodiment of the present invention.

[0034] like Figure 2 As shown in this embodiment, the global-local feature fusion-based virtual restaining method for pathological images is used to virtually restain stained slide images to obtain corresponding virtual restaining images, and includes the following steps:

[0035] Step S1: Construct a virtual restaining network for pathological images, including global and local branches. Then, train the virtual restaining network for pathological images based on existing pathological images to obtain the trained virtual restaining network for pathological images as a virtual restaining model 200. In this embodiment, the virtual restaining network for pathological images is constructed based on the CycleGAN framework.

[0036] Step S2: Input the stained slide image into the pathological image virtual restaining model 200 to obtain a virtual restaining image.

[0037] Figure 3 This is a schematic diagram illustrating the process of processing stained slide images using a virtual restaining model for pathological images in an embodiment of the present invention.

[0038] like Figure 3 As shown, the virtual restaining model for pathological images specifically includes the following sub-steps for processing stained slide images:

[0039] Step T1: Downsample the stained section image to obtain the overall image as a global feature map. In this embodiment, the resolution of the stained section image is 1600×1600, and the size of the global feature map is 448×448.

[0040] In this case, the resolution of the overall image after downsampling is the same as the resolution of the image patch.

[0041] Step T2 involves segmenting the stained slice image into multiple image blocks as local feature maps.

[0042] Figure 4 This is a schematic diagram of image blocks obtained by segmenting a stained slide image in an embodiment of the present invention.

[0043] like Figure 4 As shown, the left side is a stained section image with a resolution of H×W, and the right side is a 9 image block with a resolution of h×w obtained by dividing the stained section image in a three-row, three-column order. There are k overlapping pixels between adjacent image blocks. For example, for the image block in the second row and second column, which is the center, it is adjacent to four other image blocks. Then, (h-2*k)×(w-2*k) pixels in this image block do not overlap with other image blocks, and 2*h*k+2*w*k-4*k*k pixels overlap with the other four image blocks. In this embodiment, the stained section image is divided with k=64 pixels to obtain a 448×448 image block.

[0044] Step T3 involves sequentially performing convolution, spatial attention, and channel attention on the global feature map to obtain a new global feature map.

[0045] Step T4 involves sequentially performing convolution, spatial attention, channel attention, and image block-level attention on the local feature map to obtain a new local feature map.

[0046] The specific process of spatial attention processing is as follows: For the input feature map of size (N,C,H,W), two corresponding spatial attention feature maps are obtained in the second dimension, C, using the average and max pooling methods respectively. The two spatial attention feature maps are then concatenated in the channel dimension and passed through a convolutional layer and a sigmoid function to obtain the final spatial attention feature map. The final spatial attention feature map is then multiplied by the input feature map as the output of spatial attention processing. In this embodiment, N is the number of feature maps, C is the number of channels, H is the height, and W is the width.

[0047] The specific process of channel attention processing is as follows: For the input feature map of size (N,C,H,W), two corresponding channel attention feature maps are obtained in the third and fourth dimensions, namely H and W, respectively, using the average and max pooling methods. The two channel attention feature maps are then added together and passed through the sigmoid function to obtain the final channel attention feature map. Finally, the final channel attention feature map is multiplied by the input feature map to obtain the output of channel attention processing.

[0048] The specific process of image block-level attention processing is as follows: For the input feature map of size (N,C,H,W), two corresponding image block-level attention feature maps are obtained in the second, third, and fourth dimensions, namely C, H, and W, respectively, using the average and max pooling methods. Then, the two image block-level attention feature maps are concatenated in the channel dimension and then sequentially processed by convolution and the sigmoid function to obtain the final image block-level attention feature map. Finally, the final image block-level attention feature map is multiplied by the input feature map to obtain the output of image block-level attention processing.

[0049] Step T5: Based on the corresponding position of each image block in the stained slice image, the new global feature map is segmented and upsampled. Then, the upsampled result is concatenated with the new local feature map in the channel dimension to obtain a global-local fusion feature map as the local feature map.

[0050] Step T6: Based on the corresponding position of each image block in the stained slice image, the new local feature map is combined and downsampled. Then, the downsampled result is combined with the new global feature map in the channel dimension to obtain a local-global fused feature map as the global feature map.

[0051] Step T7: Repeat steps T3 to T6 twice, then perform residual processing on the global feature map and local feature map respectively to obtain global residual feature map and local residual feature map. In this embodiment, the residual processing is an intermediate layer containing 6 residual blocks. The number of channels does not change during this process. In this embodiment, steps T3 to T6 are repeated twice. Figure 2 The process determines whether it has been "executed twice", so that steps T3 to T6 in the whole process are executed a total of 3 times. During this process, the global branch and the local branch are convolutional three times. The kernel sizes of the three convolutions are 7×7, 3×3 and 3×3 respectively, and the corresponding number of convolution output channels are 32, 64 and 128 respectively.

[0052] Step T8: Concatenate the global residual feature map with the global feature map, and then perform upsampling to obtain the first global upsampled feature map.

[0053] Step T9: Concatenate the local residual feature map with the local feature map, and then perform upsampling to obtain the first local upsampled feature map.

[0054] In steps T8 and T9, upsampling involves processing the concatenated feature map using an upsampling block. The upsampling block includes a transposed convolutional layer, an instance normalization layer, and a ReLU function. In this embodiment, the transposed convolutional kernel size is 3×3.

[0055] Step T10: The second local-global fusion feature map is concatenated with the first global upsampled feature map, and then upsampled to obtain the second global upsampled feature map.

[0056] Step T11: The second global-local fusion feature map is concatenated with the first local upsampled feature map, and then upsampled to obtain the second local upsampled feature map.

[0057] Step T12: The second global upsampled feature map, the second local upsampled feature map, and the first global-local fusion feature map are concatenated in the channel dimension to obtain a concatenated feature map. The concatenated feature map is then passed through the output head to obtain a virtual recolored image.

[0058] The output head 230 includes a convolutional layer and a Tanh function. In this embodiment, the kernel size of the convolutional layer is 7×7, and the output of the output head 230 is an image block of size 448×448. After further processing, a virtual recolored image of size 1600×1600 is obtained.

[0059] In this embodiment, a training dataset is constructed using existing pathological images, and the virtual restaining network for pathological images is trained for 140 rounds, including 70 rounds using a constant learning rate and 70 rounds using a linearly decaying learning rate, thus obtaining the trained virtual restaining network for pathological images.

[0060] In the training process of the virtual restaining network for pathological images, the second global upsampled feature map and the first obtained local-global fusion feature map are sequentially passed through convolution and Tanh function to obtain the target domain generated image, which is used for the loss function calculation of the global branch. The second local upsampled feature map and the first obtained global-local fusion feature map are sequentially passed through convolution and Tanh function to obtain the target domain generated image patch, which is used for the loss function calculation of the local branch.

[0061] In this embodiment, the virtual restaining effect of the pathological image virtual restaining model is evaluated using the breast and lungesion subsets of the publicly available dataset ANHIR. The CSS index of the virtual restaining images obtained by the pathological image virtual restaining model is calculated, and the CSS index on the breast subset is 87.59% and the CSS index on the lungesion subset is 93.56%, which is a leading level compared with the models constructed by existing methods.

[0062] This embodiment also provides a virtual restaining device for pathological images based on global-local feature fusion, which is used to implement the virtual restaining method for pathological images based on global-local feature fusion in this embodiment.

[0063] Figure 5 This is a schematic diagram of the framework of the virtual restaining device for pathological images with global-local feature fusion in an embodiment of the present invention.

[0064] like Figure 5 As shown, the virtual restaining device 100 for pathological images includes: an image input module 10, a virtual restaining module 20, and a result display module 30.

[0065] Image input module 10 is used to receive images of stained sections to be restained.

[0066] The virtual restaining module 20 is used to process the stained slide image to obtain a virtual restaining image. The virtual restaining module 20 is constructed based on the pathological image virtual restaining model 200 in this embodiment.

[0067] The virtual recoloring module 20 includes a global encoder submodule 201, a local encoder submodule 202, a global intermediate submodule 203, a local intermediate submodule 204, and a decoder submodule 205.

[0068] The global encoder submodule 201 is constructed based on the three convolutional + attention blocks 211 of the global branch 210. The actions it performs include: performing a first convolution-attention process on the downsampled stained slice image to obtain a first global feature map, i.e., the action of obtaining a new global feature map in step T3; performing local-global fusion on the first global feature map and the first local feature map and then performing a second convolution-attention process to obtain a second global feature map, i.e., the action of obtaining a local-global fused feature map in step T6, and the action of obtaining a new global feature map in step T3; performing local-global fusion on the second global feature map and the second local feature map and then performing a third convolution-attention process to obtain a third global feature map, i.e., the action of obtaining a local-global fused feature map in step T6, and the action of obtaining a new global feature map in step T3.

[0069] The local encoder submodule 202 is constructed based on the three convolutional + attention blocks 221 of the local branch 220. The actions it performs include: performing a first convolution-attention process on multiple image blocks obtained after segmenting the stained slice image to obtain a first local feature map, which is the action of obtaining a new local feature map in step T4; performing global-local fusion on the first local feature map and the first global feature map and then performing a second convolution-attention process to obtain a second local feature map, which is the action of obtaining a global-local fused feature map in step T5, and the action of obtaining a new local feature map in step T4; performing global-local fusion on the second local feature map and the second global feature map and then performing a third convolution-attention process to obtain a third local feature map, which is the action of obtaining a global-local fused feature map in step T5, and the action of obtaining a new local feature map in step T4.

[0070] The global intermediate submodule 203 is constructed based on the residual block 212 of the global branch 210. Its action is to perform local-global fusion on the third global feature map and the third local feature map and then perform residual processing to obtain the global residual feature map, which is the action of obtaining the global residual feature map in step T7.

[0071] The local intermediate submodule 204 is constructed based on the residual block 222 of the local branch 220. Its operation is to perform global-local fusion on the third global feature map and the third local feature map and then perform residual processing to obtain the local residual feature map, which is the operation of obtaining the local residual feature map in step T7.

[0072] The decoder submodule 205 is constructed based on two upsampling blocks 213, two upsampling blocks 223, and an output head 230. The actions it performs include: upsampling based on the global residual feature map, the third global feature map, and the second global feature map to obtain a global upsampled feature map, i.e., the action of obtaining the first global upsampled feature map in step T8, and the action of obtaining the second global upsampled feature map in step T10; upsampling based on the local residual feature map, the third local feature map, and the second local feature map to obtain a local upsampled feature map, i.e., the action of obtaining the first local upsampled feature map in step T9, and the action of obtaining the second local upsampled feature map in step T11; and obtaining a virtual recolored image based on the first local feature map, the global upsampled feature map, and the local upsampled feature map, i.e., the action of obtaining the virtual recolored image in step T12.

[0073] The results display module 30 is used to display the virtual restaining image.

[0074] The role and effect of the embodiments

[0075] According to the virtual restaining method and apparatus for pathological images based on global-local feature fusion involved in this embodiment, global and local branches are constructed. Global and local features are extracted from global and local information through triple convolution and attention processing, respectively. During this process, feature fusion is performed in the global branch to fuse global features with corresponding local features to obtain new global features, and in the local branch to fuse local features with corresponding global features to obtain new local features. Through multiple feature fusions, global and local information are fused to learn information between image patches, thereby improving the correlation in color, brightness, and contrast between generated image patches. In summary, this method can effectively solve the block effect, making the virtual restaining image closer to the real staining effect.

[0076] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for virtual restaining of pathological images using global-local feature fusion, used to virtually restain stained slide images to obtain corresponding virtual restaining images, characterized in that, Includes the following steps: Step S1: Construct a virtual restaining network for pathological images, including global and local branches, and then train the virtual restaining network for pathological images based on existing pathological images to obtain the trained virtual restaining network for pathological images as a virtual restaining model for pathological images. Step S2: Input the stained slide image into the pathological image virtual restaining model to obtain the virtual restaining image. The virtual restaining model for pathological images specifically includes the following sub-steps in processing the stained slide images: Step T1: Downsample the stained slice image to obtain the overall image as a global feature map; Step T2: The stained slice image is segmented to obtain multiple image blocks as local feature maps; Step T3: Perform convolution, spatial attention, and channel attention processing on the global feature map in sequence to obtain a new global feature map; Step T4: Perform convolution, spatial attention, channel attention, and image block-level attention processing on the local feature map in sequence to obtain a new local feature map; Step T5: According to the corresponding position of each image block in the stained slice image, the new global feature map is segmented and upsampled, and then the upsampling result is concatenated with the new local feature map in the channel dimension to obtain a global-local fusion feature map as the local feature map; Step T6: Based on the corresponding position of each image block in the stained slice image, the new local feature map is combined and downsampled, and then the downsampling result is combined with the new global feature map in the channel dimension to obtain a local-global fusion feature map as the global feature map; Step T7: Repeat steps T3 to T6 twice, and then perform residual processing on the global feature map and the local feature map to obtain the global residual feature map and the local residual feature map respectively. Step T8: Concatenate the global residual feature map with the global feature map, and then perform upsampling to obtain the first global upsampled feature map; Step T9: The local residual feature map is concatenated with the local feature map, and then upsampling is performed to obtain the first local upsampled feature map; Step T10: The second obtained local-global fusion feature map is concatenated with the first global upsampled feature map, and then upsampled to obtain the second global upsampled feature map; Step T11: The second obtained global-local fusion feature map is concatenated with the first local upsampled feature map, and then upsampled to obtain the second local upsampled feature map; Step T12: The second global upsampled feature map, the second local upsampled feature map, and the first obtained global-local fusion feature map are concatenated in the channel dimension to obtain a concatenated feature map. The concatenated feature map is then output to obtain the virtual recolored image.

2. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, In step T1, after downsampling, the resolution of the overall image is the same as the resolution of the image patch. The specific process of obtaining multiple image patches in step T2 is as follows: The stained slice image with a resolution of H×W is sequentially divided into image blocks with a resolution of h×w, and there are k overlapping pixels between adjacent image blocks.

3. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, In steps T3 and T4, the specific process of spatial attention processing is as follows: For an input feature map of size (N,C,H,W), two corresponding spatial attention feature maps are obtained in the second dimension using average and max pooling methods respectively. The two spatial attention feature maps are then concatenated in the channel dimension and passed through a convolutional layer and a sigmoid function to obtain the final spatial attention feature map. The final spatial attention feature map is then multiplied by the input feature map to obtain the output of the spatial attention processing.

4. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, In steps T3 and T4, the specific process of channel attention processing is as follows: For an input feature map of size (N,C,H,W), two corresponding channel attention feature maps are obtained in the third and fourth dimensions using average and max pooling methods, respectively. The two channel attention feature maps are then added together and passed through the sigmoid function to obtain the final channel attention feature map. The final channel attention feature map is then multiplied by the input feature map to obtain the output of the channel attention processing.

5. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, In step T4, the specific process of the image block-level attention processing is as follows: For an input feature map of size (N,C,H,W), two corresponding image block-level attention feature maps are obtained in the second, third, and fourth dimensions using average and max pooling methods, respectively. The two image block-level attention feature maps are then concatenated in the channel dimension and sequentially processed by convolution and the sigmoid function to obtain the final image block-level attention feature map. The final image block-level attention feature map is then multiplied by the input feature map to serve as the output of the image block-level attention processing.

6. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, In steps T8 and T9, the upsampling involves processing the concatenated feature map using an upsampling block, which includes a transposed convolutional layer, an instance normalization layer, and a ReLU function.

7. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, In step T12, the output head includes a convolutional layer and a Tanh function.

8. The virtual restaining method for pathological images based on global-local feature fusion according to claim 1, characterized in that: in, During training, the second global upsampled feature map and the first obtained local-global fusion feature map are sequentially processed by convolution and the Tanh function to obtain the target domain generated image, which is used for calculating the loss function of the global branch. The second local upsampled feature map and the first obtained global-local fusion feature map are sequentially processed by convolution and Tanh function to generate image patches in the target domain, which are used for calculating the loss function of the local branch.

9. A virtual restaining apparatus for pathological images based on global-local feature fusion, used for performing the virtual restaining method for pathological images based on global-local feature fusion as described in any one of claims 1-8, characterized in that, include: An image input module is used to receive the image of the stained section to be restained; A virtual restaining module is used to process the stained slice image to obtain the virtual restaining image; The results display module is used to display the virtual restained image. The virtual recoloring module includes a global encoder submodule, a local encoder submodule, a global intermediate submodule, a local intermediate submodule, and a decoder submodule. The global encoder submodule and the local encoder submodule perform convolution-attention processing on the downsampled and segmented stained slice images to obtain global feature maps and local feature maps, respectively. Then, the global encoder submodule and the local encoder submodule perform two local-global fusions and two global-local fusions on the global feature maps and local feature maps, respectively. After each local-global fusion and global-local fusion, a convolution-attention processing is performed again to obtain the latest global feature maps and local feature maps. The global intermediate submodule is used to perform local-global fusion on the latest global feature map and the local feature map, and then perform residual processing to obtain a global residual feature map. The local intermediate submodule is used to perform global-local fusion on the latest global feature map and the local feature map, and then perform residual processing to obtain a local residual feature map. The decoder submodule is used to concatenate and upsample the global residual feature map and the local residual feature map with the corresponding global feature map and local feature map twice, and then concatenate and output the virtual recolored image.

Images