Image dyeing style conversion method based on feature decoupling and staged training
By employing feature decoupling and phased training methods, the problems of structural distortion and inconsistent staining in pathological image staining conversion are solved, achieving efficient pathological image staining style conversion, which is applicable to pathological diagnosis and computer-aided diagnostic systems.
Patent Information
- Application Number
- CN202510977529.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-16
- Publication Date
- 2025-10-31
AI Technical Summary
Existing technologies for pathological image staining conversion suffer from problems such as distortion of tissue morphology features, texture degradation, inconsistent staining styles, and low training efficiency. In particular, it is difficult to achieve high-quality cross-staining domain mapping when paired data is lacking.
A feature-based decoupling and phased training approach is adopted. The Base module extracts structural features, the Style module extracts style features, and the Transformer Encoder and Decoder modules are combined to generate images. The training process is optimized by using a color-aware total loss function to achieve structural fidelity and color consistency.
It achieves high-quality cross-domain mapping in pathological image staining conversion, maintains tissue structure integrity and staining consistency, improves training efficiency and visual consistency of generated images, and is suitable for clinical applications.
Smart Images
Figure CN120876207A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of staining style conversion technology in digital pathology, and specifically relates to an image staining style conversion method based on feature decoupling and staged training. Background Technology
[0002] In digital pathology, staining is a crucial process for visualizing tissue components, helping to identify disease markers, cell morphology, and structure. However, variations in staining protocols, reagents, and scanners lead to staining variability, which can significantly impact diagnostic consistency and accuracy, especially when comparing samples from different medical centers or batches. Staining variability (i.e., “staining gaps”) not only affects the accuracy of pathologists' manual diagnoses but also interferes with the performance of computer-aided diagnostic (CAD) systems. Furthermore, traditional staining procedures, such as Hematoxylin and Eosin (H&E) and Masson's Trichrome (MAS) staining, require multiple repetitions, adding significant time and cost.
[0003] Current methods for addressing this problem mainly fall into two categories: traditional statistical methods, such as Reinhard et al.'s color normalization and Macenko's staining normalization; and deep learning-based methods, particularly those utilizing Generative Adversarial Networks (GANs) for cross-staining domain transformation. While these methods have shown promise, they still face two major problems: first, tissue morphology gradually distorts during staining transformation, leading to blurred tissue boundaries and texture degradation; second, nonlinear domain differences between stained images are difficult to model, especially when paired data is lacking, which is common in pathological images. Traditional GAN-based methods, while generating relatively similar images for cross-staining domain transformation, suffer from issues such as distorted cell structures and inconsistent staining styles. Furthermore, the training process of GANs is prone to mode collapse, resulting in uncontrollable quality control of the generated images. Diffusion models, as an emerging generative model, have demonstrated unique advantages in image generation quality control by generating images through progressive denoising. Compared to GANs, diffusion models have significant advantages in training stability and generation quality. However, several problems remain when applying diffusion models to pathological image staining transformation:
[0004] Pathological images have strict spatial structure requirements, while the generation process of diffusion models has a certain degree of randomness. This makes it possible that spatial correspondences may not be accurately preserved when processing pathological images, leading to distortion of structural details. Furthermore, in staining conversion tasks, the lack of pixel-level aligned ground truth data makes it difficult for diffusion models to establish reliable cross-modal mapping relationships in unsupervised learning, thus affecting the quality of the generated results. Moreover, diffusion models typically require long training times and consume significant computational resources, especially in pathological image applications where image sizes are large and multiple iterations are required. This results in low training efficiency for diffusion models, making them difficult to meet clinical needs in practical applications. Summary of the Invention
[0005] To address the above problems, this invention proposes an image coloring style transfer method based on feature decoupling and staged training, comprising the following steps:
[0006] S1. Obtain a medical image dataset, which includes multiple image pairs, each image pair including HE-stained and MAS-stained images of the same tissue slice;
[0007] S2. Construct a pathological image staining style transfer model, which includes:
[0008] The Base module is used to process HE-stained images to obtain structural features;
[0009] The Style module is used to process MAS-stained images to obtain style features;
[0010] The Transformer Encoder module is used to weightedly fuse structural features and style features to obtain fused encoded features.
[0011] The Decoder module is used to process fused encoded images and generate style-transferred images.
[0012] S3. A three-stage method was used to train the pathological image staining style transfer model, wherein:
[0013] In the first training phase, the parameters of the Style module are frozen, and only the parameters of the Base module are trained and optimized.
[0014] In the second training phase, the Base module parameters obtained in the first training phase are used, the Base module parameters are frozen, and only the Style module parameters are trained and optimized.
[0015] In the third training phase, the Base module parameters obtained in the first training phase and the Style module parameters obtained in the second training phase are loaded as initialization parameters, and the Base module parameters and Style module parameters are trained and optimized together.
[0016] The total training loss is calculated using the color perception total loss function;
[0017] S4. Implement pathological image staining style transfer using a trained pathological image staining style transfer model.
[0018] The beneficial effects of this invention are:
[0019] In the forward training phase, this invention decouples the tissue structure features and staining style features of pathological images through independent Base and Detail modules, avoiding interference between staining style and structural information in traditional style conversion. By dynamically adjusting the weights of each module, this invention can efficiently achieve alignment between different staining styles, ensuring that the converted image maintains structural integrity while achieving the desired staining effect.
[0020] This invention employs a phased training method. First, the style extraction module is frozen, with a focus on training the structure preservation module to ensure that the structural features of the image are not interfered with by staining styles. Subsequently, the style module is gradually unfrozen to learn staining styles, ultimately achieving efficient staining style alignment. This phased training strategy ensures a good balance between structural fidelity and staining consistency during the staining conversion of pathological images.
[0021] To further improve the accuracy and consistency of color style transfer, this invention introduces histogram loss and perceptual loss. By optimizing the similarity of color distribution and the alignment of high-level semantic features, the visual consistency and naturalness of the generated image across different colorings are improved.
[0022] The method of this invention effectively solves the problem of lack of pixel-level alignment ground truth constraints between stained images. Through an innovative combination of feature decoupling and staged training, this invention not only achieves high-quality cross-domain mapping in staining style transfer, but also greatly improves the practicality and adaptability of pathological images in clinical applications. The pathological image staining style transfer method of this invention, through efficient feature decoupling and staged training strategies, provides a powerful tool for staining standardization of pathological images, which has significant clinical implications and broad application prospects. Attached Figure Description
[0023] Figure 1 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] This invention provides an image coloring style transfer method based on feature decoupling and staged training, such as... Figure 1 As shown, it includes the following steps:
[0026] S1. Obtain a medical image dataset, which includes multiple image pairs, each image pair including HE-stained and MAS-stained images of the same tissue slice.
[0027] Specifically, the acquisition of medical image datasets employs a systematic quality control scheme, and the process includes:
[0028] S11. Perform HE staining and MAS staining on the same tissue section to obtain HE-stained sections and MAS-stained sections;
[0029] S12. A spatial alignment algorithm is used to accurately register the HE-stained sections and MAS-stained sections to ensure pixel-level correspondence of histological structures, resulting in HE-stained registered sections.
[0030] S13. The blank area at the edge of the slide is removed by adaptive threshold segmentation technology (Otsu algorithm combined with morphological operation), the tissue area of the HE stained and registered section is extracted and the whole slide is imaged to obtain the first full field of view digital image, and the tissue area of the MAS stained section is extracted and the whole slide is imaged to obtain the second full field of view digital image.
[0031] S14. Using a sliding window, the first full-view digital image and the second full-view digital image are standardized and cut into 256×256 pixels respectively to obtain HE staining images and MAS staining images of the same tissue section, forming a set of image pairs.
[0032] S15. Repeat steps S11-S15 to obtain multiple sets of image pairs to form a medical image dataset.
[0033] S2. Construct a pathological image staining style transfer model, which includes:
[0034] The Base module is used to process HE-stained images to obtain structural features;
[0035] The Style module is used to process MAS-stained images to obtain style features;
[0036] The Transformer Encoder module is used to weightedly fuse structural features and style features to obtain fused encoded features.
[0037] The Decoder module is used to process fused encoded images and generate style-transferred images.
[0038] Specifically, to ensure structural fidelity and accurate transfer of staining style, this invention designs two complementary sub-modules: a Transformer Encoder module and a feature extraction module. These are the core components of the pathological image staining style transfer model. The feature extraction module is responsible for extracting separable structural and style features from the input image. The Transformer Encoder module uses an Efficient Transformer Encoder (ETE) to encode the extracted structural and style features into combinable sequences, which are then used to generate the final style-transferred image.
[0039] This invention divides the feature extraction module into two parallel branches based on the principle of structural decoupling: the Base module and the Style module, which extract structural features and style features respectively. This feature decoupling method ensures that style and structural features are aligned, providing more precise control for subsequent staining style conversion. In this embodiment, the HE-stained image is used as the content image to provide tissue structure information. The Base module extracts the structural features of the HE-stained image, primarily focusing on the morphological features of the tissue (such as cells and tissue boundaries). The MAS-stained image is used as the target style image to provide target staining style information. The Style module extracts the staining features (i.e., style features) from the MAS-stained image, focusing on staining distribution and color variation characteristics.
[0040] Specifically, the Base module includes three invertible blocks, which preserve the topological information and boundary morphology of the organizational structure in the content image. Each invertible block includes a channel separation mapping layer and an affine transformation layer for information splitting and exchange. The processing flow for the k=1, 2, and 3 invertible blocks includes:
[0041] Enter L k Feature L is obtained through a channel separation mapping layer. k [:,0:C] and feature L k [:,C:], where C represents the input L k Half the number of channel dimensions;
[0042] Feature L k [:,0:C] and feature L k [:,C:] is transformed into feature L through an affine transformation layer. k+1[:,0:C] and feature L k+1 [:,C:], represents
[0043] L k+1 [:,0:C]=L k [:,0:C]+T1(Y k [:,0:C])
[0044] L k+1 [:,C:]=L k [:,C:]·exp(T2(L k+1 [:,0:C]))+T3(L k+1 [:,0:C])
[0045] Among them, Y k [:,0:C] represents the tensor Y. k The first C columns of features, typically structural features, play a role in distinguishing content and style during feature decoupling. Here, the tensor Y... k That is, L k T1, T2, and T3 represent three affine transformation networks, primarily responsible for modeling the nonlinear relationships between channels.
[0046] Feature L k+1 [:,0:C] and feature L k+1 The output L is obtained by concatenating [:,C:]. k+1 .
[0047] The processing of the entire Base module can be represented as follows:
[0048] F structure =Base(I content )
[0049] Among them, I content Represents a content image, F structure Indicates structural characteristics; Base() represents the Base module.
[0050] Specifically, the Style module includes two lightweight Transformer (LT) structures, designed to improve efficiency while ensuring model performance by reducing computational cost and the number of parameters. Compared to the standard Transformer (ViT) structure, the LT structure has the following characteristics:
[0051] 1) Streamlined self-attention mechanism: Using small-sized multi-head self-attention (MHSA) layers reduces the number of heads and dimensions per layer compared to the ViT structure, thereby reducing computational cost.
[0052] 2) Embedded Reduction Layer: To further reduce computational complexity, the LT structure uses an embedded reduction layer on the input features of each Transformer layer. The input features are reduced in dimensionality through a 1×1 convolution operation, thereby reducing the consumption of computational resources.
[0053] 3) Simplified Feedforward Network (FFN): Compared to the ViT structure, the LT structure feedforward network uses fewer fully connected layers and smaller hidden layer dimensions, thereby reducing computational overhead.
[0054] 4) Layer normalization and residual connection: The LT structure retains the residual connection and layer normalization operations of the ViT structure, but further optimizes the computational cost through lightweight normalization strategies (such as lightweight layer normalization).
[0055] 5) Highly efficient style transfer capability: The LT structure enhances the transfer capability of chromatic style through an optimized self-attention mechanism and feedforward network, ensuring high-quality conversion between the target style and the original image while preserving the organizational structure.
[0056] The LT structure captures long-range color-dependent features through a deep attention mechanism while preserving the statistical distribution of coloring. The entire processing of the Base module can be represented as follows:
[0057] F style =Detail(I style )
[0058] Among them, I style F represents the target style image. style This indicates style characteristics, and Detail() indicates the Detail module.
[0059] Specifically, the Transformer Encoder module includes three Transformer Blocks, and the specific processing steps include:
[0060] Assuming the content image size is H×W, and the patch size is set to P×P;
[0061] The structural features are converted into M = H / P × W / P patches and then flattened to obtain the structural sequence; the style features are converted into M patches and then flattened to obtain the style sequence.
[0062] Perform a linear transformation on each patch in the structure sequence and each patch in the style sequence to map each patch to a d-dimensional feature space, thus obtaining the structure embedding sequence. and style embedding sequence
[0063] The structural embedding sequence and the style embedding sequence are fused and the dependency relationship between structure and style is modeled through three Transformer Blocks to obtain fused encoded features.
[0064] Specifically, the processing flow for each Transformer Block includes:
[0065] The first feature vector is passed through the first linear layer to obtain the Q feature, and the second feature vector is passed through the second linear layer and the third linear layer to obtain the K feature and the V feature, respectively.
[0066] The attention score is calculated based on the Q feature, K feature, and V feature. The attention score is then added to the Q feature to obtain the third feature vector.
[0067] The third eigenvector is passed through FFN and then added to the Q eigenvector to obtain the fourth eigenvector;
[0068] In the first Transformer Block, the first feature vector and the second feature vector are the structure embedding sequence and the style embedding sequence, respectively; in the second Transformer Block, the first feature vector and the second feature vector are the Q feature and the fourth feature vector obtained in the first Transformer Block, respectively; in the third Transformer Block, the first feature vector and the second feature vector are the Q feature and the fourth feature vector obtained in the second Transformer Block, respectively; the fourth feature vector obtained in the third Transformer Block is used as the fused encoding feature.
[0069] Learnable identifiers are embedded in the structure embedding sequence and the style embedding sequence. These learnable identifiers are updated and optimized through multiple Transformer Blocks in the Transformer Encoder module, and finally used as global features to assist in the reconstruction process of style transfer.
[0070] Specifically, the Decoder module uses a convolutional neural network to process the fused encoded image, recovering the image information layer by layer, which can be represented as follows:
[0071]
[0072] in, Indicates fused coding features, I output This represents a style-transformed image.
[0073] S3. A three-stage method was used to train the pathological image staining style transfer model, wherein:
[0074] In the first training phase, the parameters of the Style module are frozen, and only the parameters of the Base module are trained and optimized to ensure that the structural information of the pathological image is preserved. At the same time, both the Transformer Encoder and Decoder modules are trained to learn the encoding and decoding of structural features.
[0075] In the second training phase, the Base module parameters obtained in the first training phase are used and frozen. Only the Style module parameters are trained and optimized to ensure that the style information of the pathological images is preserved. At the same time, the Transformer Encoder module continues to participate in training, while the Decoder module parameters are frozen.
[0076] In the third training phase, the Base module parameters obtained in the first training phase and the Style module parameters obtained in the second training phase are loaded as initialization parameters. The Base module parameters and Style module parameters are jointly trained and optimized to ensure that the final generated image achieves the best balance between structural fidelity and color consistency. At the same time, the Transformer Encoder module and Decoder module also participate in joint optimization.
[0077] The total training loss is calculated using the color perception total loss function.
[0078] Specifically, this invention designs different loss functions to achieve structural fidelity and staining accuracy.
[0079] To preserve organizational structure features, a perceptual loss L was designed. perceptual and content loss L content , represented as
[0080]
[0081] Where N represents the number of VGG network layers, φ i () represents the i-th layer of the VGG network.
[0082] By calculating the high-level feature differences between the generated image (i.e., the generated style-transferred image) and the target style image, as well as the detailed differences between the generated image and the content image, we ensure the high-level semantic consistency of the generated image and the accurate preservation of organizational structure features.
[0083] To ensure consistency in coloring style, this invention designs a color distribution loss L. histogram and style loss L style These are used to measure the differences in color distribution and style between the target image and the generated image, respectively, to optimize the color consistency of the generated image.
[0084] L histogram =||H(I output)-H(I style )||2
[0085]
[0086] Where H() represents the calculation of the color histogram of the image, which is used to measure the distribution characteristics of the image in the color channels to ensure that the generated image is close to the target style image in terms of color consistency; μ() represents the mean operation, σ() represents the variance operation, and ||·||2 represents the L2 norm.
[0087] The ultimate optimization objective is the weighted sum of all loss functions:
[0088] L total =λ c L content +λ s L style +λ perceptual L perceptual +λ histogram L histogram
[0089] L total Let λ represent the total loss function for color perception. c , λ s , λ perceptual , λ histogram This represents the weighting coefficient.
[0090] S4. Implement pathological image staining style transfer using a trained pathological image staining style transfer model.
[0091] In one embodiment, the method of the present invention is compared with traditional staining normalization methods (such as MUNIT, StainCUT, UGATIT), cross-domain transfer methods based on generative adversarial networks (such as StainGAN, StarGANv2, GramGAN), and transformation methods based on diffusion models (UMDST, ArtFusion, OTCS) through simulation experiments, and the structure is shown in Table 1.
[0092] Table 1
[0093]
[0094]
[0095] In terms of structural fidelity, the average SSIM index reached 0.8563, PSNR was 17.62 dB, and LPIPS was 0.1421, effectively solving the problems of blurred tissue boundaries and loss of texture structure. Regarding color consistency, the staining vector similarity (SVS) reached 0.9982, and the histogram distance (HD) decreased to 0.8044, achieving efficient conversion between H&E and MAS staining styles. This invention has significant advantages in terms of structural fidelity and staining consistency.
[0096] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0097] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. An image coloring style transfer method based on feature decoupling and staged training, characterized in that, Includes the following steps: S1. Obtain a medical image dataset, which includes multiple image pairs, each image pair including HE-stained and MAS-stained images of the same tissue slice; S2. Construct a pathological image staining style transfer model, which includes: The Base module is used to process HE-stained images to obtain structural features; The Style module is used to process MAS-stained images to obtain style features; The Transformer Encoder module is used to weightedly fuse structural features and style features to obtain fused encoded features. The Decoder module is used to process fused encoded images and generate style-transferred images. S3. A three-stage method was used to train the pathological image staining style transfer model, wherein: In the first training phase, the parameters of the Style module are frozen, and only the parameters of the Base module are trained and optimized. In the second training phase, the Base module parameters obtained in the first training phase are used, the Base module parameters are frozen, and only the Style module parameters are trained and optimized. In the third training phase, the Base module parameters obtained in the first training phase and the Style module parameters obtained in the second training phase are loaded as initialization parameters, and the Base module parameters and Style module parameters are trained and optimized together. The total training loss is calculated using the color perception total loss function; S4. Implement pathological image staining style transfer using a trained pathological image staining style transfer model.
2. The image coloring style transfer method based on feature decoupling and staged training according to claim 1, characterized in that, The Base module consists of three reversible blocks, each including a channel separation mapping layer and an affine transformation layer; the processing flow for the k=1, 2, and 3 reversible blocks includes: Enter L k Feature L is obtained through a channel separation mapping layer. k [:,0:C] and feature L k [:,C:], where C represents the input L k Half the number of channel dimensions; Feature L k [:,0:C] and feature L k [:,C:] is transformed into feature L through an affine transformation layer. k+1 [:,0:C] and feature L k+1 [:,C:], represents 50 k+1 [:,0:C]=L k [:,0:C]+T1(Y k [:,0:C]) L k+1 [:,C:]=L k [:,C:]exp(T2(L k+1 [:,0:C]))+T3(L k+1 [:,0:C]) Among them, Y k [:,0:C] represents the tensor Y. k The features in the first C columns are T1, T2, and T3, which represent three affine transformation networks. Feature L k+1 [:,0:C] and feature L k+1 [:,C:] concatenation yields the output L k+1 .
3. The image coloring style transfer method based on feature decoupling and staged training according to claim 1, characterized in that, The Style module includes two lightweight Transformer structures.
4. The image coloring style transfer method based on feature decoupling and staged training according to claim 1, characterized in that, The Transformer Encoder module includes three Transformer Blocks, and the specific processing steps include: Assuming the content image size is H×W, and the patch size is set to P×P; The structural features are converted into M = H / P × W / P patches and then flattened to obtain the structural sequence; the style features are converted into M patches and then flattened to obtain the style sequence. Linear embedding is performed on each patch in the structure sequence and each patch in the style sequence to obtain the structure embedding sequence and the style embedding sequence; By passing the structural embedding sequence and the style embedding sequence through three Transformer Blocks, the dependency relationship between structure and style is modeled to obtain fused encoded features.
5. The image coloring style transfer method based on feature decoupling and staged training according to claim 4, characterized in that, The processing flow for each Transformer Block includes: The first feature vector is passed through the first linear layer to obtain the Q feature, and the second feature vector is passed through the second linear layer and the third linear layer to obtain the K feature and the V feature, respectively. The attention score is calculated based on the Q feature, K feature, and V feature. The attention score is then added to the Q feature to obtain the third feature vector. The third eigenvector is passed through FFN and then added to the Q eigenvector to obtain the fourth eigenvector; In the first Transformer Block, the first feature vector and the second feature vector are the structure embedding sequence and the style embedding sequence, respectively; in the second Transformer Block, the first feature vector and the second feature vector are the Q feature and the fourth feature vector obtained in the first Transformer Block, respectively; in the third Transformer Block, the first feature vector and the second feature vector are the Q feature and the fourth feature vector obtained in the second Transformer Block, respectively; the fourth feature vector obtained in the third Transformer Block is used as the fused encoding feature.
6. The image coloring style transfer method based on feature decoupling and staged training according to claim 1, characterized in that, The total loss function for color perception is expressed as follows: L total =λ c L content +λ s L style +λ perceptual L perceptual +λ histogram L histogram L total Let λ represent the total loss function for color perception. c , λ s , λ perceptual , λ histogram Indicates the weighting coefficient; Perceived loss L perceptual Represented as Content loss L content Represented as Color distribution loss L histogram Represented as Style loss L style Represented as Where H() represents calculating the color histogram of the image, μ() represents the mean operation, σ() represents the variance operation, ||·||2 represents the L2 norm, N represents the number of layers in the VGG network, and φ i () represents the i-th layer of the VGG network, where I output Indicates a style-transferred image, I content Represents content image, I style This represents the target style image.