H&E image enhancement method and system based on biological mechanism and frequency attention

Abstract

The application discloses an H&E image enhancement method and system based on a biological mechanism and frequency attention, and the method first acquires a virtual staining data set, pre-processes images contained in the data set, generates input image data, and synchronously divides the data set.Secondly, an H&E image virtual staining model based on biological tone perception and frequency separation attention is constructed, feature extraction is performed on the input image data, H&E images after virtual staining are obtained, and enhancement is completed.Finally, a composite loss function composed of an adversarial loss, a cycle consistency loss and an identity loss is constructed, parameter optimization training is performed on the H&E image virtual staining model, and test evaluation is performed.The application effectively improves the color consistency and stability of the virtual staining result while maintaining the tissue structure information, effectively maintains the detail features of the pathological tissue while suppressing the interference of the background artifact.

Description

Technical Field

[0001] This invention belongs to the fields of medical image processing and computer vision, and particularly relates to a method and system for enhancing hematoxylin-eosin staining (H&E) images based on biological tone perception and frequency separation attention. Background Technology

[0002] Pathological slide images play a crucial role in clinical diagnosis and disease research. Hematoxylin-eosin staining, in particular, clearly reveals tissue structure and cell morphology, making it one of the most commonly used staining methods in routine pathological examinations. Immunohistochemical staining (IHC) is used to display the expression information of specific molecules or proteins, and it is highly complementary to hematoxylin and eosin (H&E) staining in pathological analysis. To improve the efficiency of pathological analysis and reduce the time and cost associated with actual staining procedures, virtual staining techniques that use computational methods to switch between different staining modalities have emerged in recent years, becoming an important research direction in the field of medical image processing.

[0003] With the development of deep learning and generative adversarial networks, virtual staining methods for pathological images based on generative models have attracted widespread attention, enabling mapping between IHC and H&E images without requiring strictly paired samples. However, existing virtual staining methods often focus on overall style transfer or pixel distribution matching, paying insufficient attention to the differences in frequency information in pathological images. This can easily damage key structural features such as cell nuclear boundaries and glandular contours during the generation process. Furthermore, during block training and reconstruction, color distribution differences between different image blocks may be amplified, leading to color drift or tonal discontinuities in adjacent regions of the virtual staining results, thus affecting overall visual consistency and practical application effectiveness.

[0004] Furthermore, some existing methods do not fully incorporate the biological characteristics of pathological images during virtual staining, and do not adequately consider the biological tone perception mechanism during the staining process. This makes it difficult to simultaneously ensure the accuracy of tissue structure preservation and color expression under complex tissue structures and diverse staining conditions. Therefore, how to effectively perceive and regulate biological tone information during virtual staining, and how to perform differentiated modeling for different frequency components to improve the structural stability and color consistency of virtual staining results, remains a pressing technical problem to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for H&E image enhancement based on biological mechanisms and frequency attention. By introducing a perceptual mechanism for the biological hue information of pathological staining during the generative modeling process and performing differential modeling of different frequency components in the image, this invention effectively maintains tissue structural features and improves the consistency and stability of color expression while achieving cross-staining modality mapping. This solves the problems of structural blurring, color drift, and insufficient overall visual consistency that are prone to occur in existing virtual staining methods, and provides a more stable and reliable technical means for pathological image analysis and assisted diagnosis.

[0006] To achieve the above objectives, this invention proposes a method for H&E image enhancement based on biological mechanisms and frequency attention, comprising the following steps:

[0007] Step 1: Obtain the IHC and H&E datasets for virtual staining, preprocess the images contained in the datasets to generate input image data, and simultaneously partition the datasets;

[0008] Preferably, a virtual staining dataset of H&E images is obtained, including a set of proprietary pathological image datasets and a set of publicly available pathological image datasets (BCI). The IHC and H&E images contained in the dataset are preprocessed to generate image data that meets the model input requirements.

[0009] Step 2: Construct a virtual coloring model for H&E images based on biological tone perception and frequency separation attention, extract features from the input image data, and obtain the virtual colored H&E image to complete the enhancement;

[0010] Preferably, the H&E image virtual staining model is constructed using a generator and discriminator collaborative adversarial training method, wherein the generator is used to realize the virtual staining mapping between IHC images and H&E images, and the discriminator is used to distinguish the authenticity of the generated image and the real staining image, so as to jointly optimize the virtual staining effect.

[0011] Preferably, the generator adopts an overall network architecture of encoder-residual connection-decoder, wherein the encoder is used to perform feature encoding and representation learning on the preprocessed IHC and H&E images, the decoder is used to reconstruct the image representation of the target stained domain in the feature space, and the encoder and decoder transmit feature information through residual connection to enhance the ability to preserve structural information.

[0012] Preferably, a biological tone perception mechanism and a frequency separation attention mechanism are introduced into the generator to perform targeted modeling and regulation of color and structural information in pathological images, so as to reduce color drift during virtual staining and improve staining consistency and structural fidelity.

[0013] Step 3: Use a biological tone-aware encoder to encode the features of the input pathological image. In the feature extraction stage, introduce a color-aware modulation structure that simulates the inhibition-facilitation receptive field mechanism to enhance color discrimination ability and maintain tissue structure consistency.

[0014] Preferably, after preliminary feature extraction of the pathological image at the input layer, the biological tone-aware encoder differs from conventional encoder structures that rely solely on convolution and downsampling operations. Instead, it introduces an inhibition-facilitation receptive field modulation module in the feature domain, simulating the color perception characteristics of the biological visual system. This module adaptively adjusts the initially extracted features to enhance the model's ability to perceive staining-related color changes. The inhibition-facilitation receptive field modulation module enhances regions with significant color aggregation by differentially modeling the feature responses at different spatial locations and in different channels, while suppressing regions with color redundancy or structural irrelevantity. This maintains the continuity of tissue texture while strengthening the expression of color distribution features related to the target staining. The features modulated by the inhibition-facilitation receptive field are used as intermediate outputs of the encoder. Further downsampling operations are then used for feature compression and abstraction, progressively constructing high-level semantic feature representations. This provides coded feature inputs that balance color stability and structural consistency for the subsequent virtual staining decoding process.

[0015] Step 4: The encoded features are reconstructed step by step using a decoder based on a frequency separation attention mechanism. A virtual staining image is generated by combining high-frequency information refinement with low-frequency information stability preservation.

[0016] Preferably, the decoder differs from conventional decoding structures that directly reconstruct the output image through upsampling. Instead, it constructs a dual-channel decoding path based on the concept of frequency separation, differentially processing high-frequency structural information and low-frequency color information in the encoded features to improve the structural clarity and color consistency of the virtual staining results. In the high-frequency decoding path, residual differential processing is performed on the original input features and the results of progressive upsampling to continuously extract and refine high-frequency structural information such as tissue boundaries and cell contours, thereby suppressing structural blurring and artifact interference introduced during upsampling and achieving layer-by-layer enhancement of high-frequency information. In the low-frequency decoding path, stable transmission and channel-level response adjustment of the input features maintain the continuity and consistency of the overall color distribution of the pathological image, reducing the impact of color drift on overall visual perception during virtual staining.

[0017] Preferably, in the feature fusion stage of the high-frequency decoding path and the low-frequency decoding path, a channel-space joint attention mechanism is introduced to adaptively weight the response intensity of different frequency features. This guides the decoder to focus on high-frequency information reconstruction in structurally significant regions and on preserving low-frequency information in color-smooth regions, achieving information mixing and synergistic enhancement in both frequency and spatial dimensions. After multi-level frequency separation decoding and attention-guided fusion, the fused features are mapped to the target staining domain through an output layer composed of reflection filling, convolution, and hyperbolic tangent activation, generating a virtual stained pathological image with clear structure and stable color.

[0018] Step 5: Construct a composite loss function consisting of adversarial loss, cycle consistency loss, and identity loss, and perform parameter optimization training on the virtual coloring model;

[0019] Preferably, the adversarial loss is used to constrain the virtual staining image output by the generator to closely approximate the real staining image in terms of overall distribution, thereby improving the realism and naturalness of the generated result; the cycle consistency loss is used to constrain the bidirectional mapping relationship between different staining domains, maintaining the consistency of tissue structure information in the virtual staining process; and the identity loss is used to suppress unnecessary color mapping changes, enhancing the stability of the model in terms of color preservation.

[0020] Step 6: Quantitatively evaluate the virtual staining performance of the model on the test set and generate visualized H&E image staining results;

[0021] In another aspect, the present invention provides a H&E image enhancement system based on biological mechanisms and frequency attention, comprising the following modules:

[0022] The image data processing module is used to acquire a virtual coloring dataset, preprocess the images contained in the dataset, generate input image data, and simultaneously divide the dataset.

[0023] The H&E image virtual coloring module is used to construct an H&E image virtual coloring model based on biological tone perception and frequency separation attention. It extracts features from the input image data to obtain the virtual-colored H&E image, thus completing the enhancement.

[0024] The loss training module is used to construct a composite loss function composed of adversarial loss, cycle consistency loss, and identity loss, to optimize the parameters of the H&E image virtual coloring model and to perform testing and evaluation.

[0025] The beneficial effects of this invention are:

[0026] (1) In view of the problem that existing virtual staining methods are not sensitive enough to color changes in the feature encoding stage and are prone to color drift, this invention introduces a tone perception coding structure based on biological vision mechanism in the encoder part of the generator. By simulating the inhibition-facilitation receptive field characteristics in the biological visual cortex, the feature distribution changes in pathological images are differentiated and modeled, thereby enhancing the model's ability to express staining-related color features. While maintaining tissue structure information, it effectively improves the color consistency and stability of virtual staining results.

[0027] (2) In view of the problem that upsampling operation in the existing virtual staining decoding process can easily lead to blurred tissue structure and loss of high-frequency details, this invention proposes a novel decoding strategy of joint frequency separation attention. By performing channel processing on high-frequency feature information and low-frequency feature information, and combining multi-level difference to enhance high-frequency features such as hue information and cell outline layer by layer, the detailed features of pathological tissue can be effectively maintained while suppressing background artifact interference.

[0028] (3) In view of the problem that color information and structural information are difficult to optimize in a coordinated manner during the virtual staining process, the present invention further introduces a channel-space joint attention mechanism in the decoding stage, which adaptively weights and guides the fusion of response intensity of different frequency features, so that the model can strengthen the reconstruction of high-frequency information in structurally significant regions and maintain the stability of low-frequency information in color smooth regions, thereby achieving a coordinated improvement of color accuracy and structural fidelity in the virtual staining results as a whole. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the implementation process provided in the embodiments of the present invention;

[0030] Figure 2 This is a schematic diagram of the overall structure provided for an embodiment of the method;

[0031] Figure 3 A schematic diagram of the generator structure provided in this embodiment of the method;

[0032] Figure 4 The bio-tone-sensing encoder structure provided in this embodiment of the method;

[0033] Figure 5 This is a schematic diagram of the corresponding visual cortex receptive field provided in this embodiment of the method;

[0034] Figure 6 This is a schematic diagram of the frequency separation attention decoder structure provided in this embodiment of the method;

[0035] Figure 7 This is a schematic diagram illustrating the color inconsistencies in the CycleGAN model results provided in this embodiment at high resolution.

[0036] Figure 8 This is a schematic diagram showing the image results compared with the CycleGAN model provided in this embodiment;

[0037] Figure 9 This is a schematic diagram of the FFT results of the image compared with the CycleGAN model provided in this embodiment;

[0038] Figure 10 This is a schematic diagram of full-view pathological images from a proprietary dataset. Detailed Implementation

[0039] The invention will now be described in detail with reference to the accompanying drawings. One aspect of the invention provides a method for H&E image enhancement based on biological tone perception and frequency separation attention, such as… Figure 1 As shown, the process includes the following steps:

[0040] Step 1: Constructing image datasets for virtual staining IHC and H&E

[0041] In this embodiment, the dataset includes a proprietary pathology image dataset and a publicly available pathology image dataset. The proprietary pathology image dataset is provided by the French Pathology Center Cypath-RB (Centre de Pathologie, Villeurbanne, France) and contains pathology slide images derived from normal appendix tissue, as shown in the image below. Figure 10 As shown. The proprietary dataset comprises 44 pairs of full-field pathological images, corresponding to immunohistochemical staining images and hematoxylin-eosin staining images, with an image scanning resolution of 0.25. Pixels, corresponding to optical magnification of All images were derived from continuous tissue sections, exhibiting highly consistent tissue structures, but showing significant differences in staining patterns and color distribution. In the data preprocessing stage, tissue region extraction was first performed on the full-field pathological images to remove invalid background areas; subsequently, the images were cropped within the valid tissue regions to obtain images of size [size missing]. Image patches are used as model input.

[0042] In addition, this embodiment also uses the BCI (Breast Cancer Immunohistochemical Image Generation) public dataset as a supplementary dataset to verify the applicability of the virtual staining method on publicly available pathological data. The BCI dataset contains 4783 pairs of structurally aligned IHC and H&E image samples. All images are derived from continuous tissue sections, and the image size is [missing information]. After cutting, the size is obtained as follows: Image patches are used as model input.

[0043] For example, after completing the above processing, the constructed image data is randomly sorted and then arranged according to... The model is divided into a training set and a test set. The training set is used to train the virtual coloring model, and the test set is used to evaluate the model's performance.

[0044] Step 2: Construct a virtual staining network structure based on biological tone perception and frequency separation attention to model the mapping relationship between IHC images and H&E images;

[0045] In this embodiment, the virtual staining model adopts a generator-discriminator co-adversarial training network structure and is built based on the CycleGAN framework under the condition of no registration samples, such as... Figure 2 As shown, the network implements bidirectional mapping between the IHC image domain and the H&E image domain through two generators, and constrains the authenticity of the generated results through corresponding discriminators, thereby ensuring the style consistency and structural stability of the virtual staining results at the overall level.

[0046] Within the aforementioned overall adversarial learning framework, this invention focuses on improving the generator structure design, such as... Figure 3 As shown, a biological tone perception mechanism and a frequency separation attention mechanism are introduced into the generator to enhance the modeling ability of color information and tissue structure information of pathological images, reduce color drift during virtual staining, and provide a more stable feature expression basis for subsequent encoding and decoding processes.

[0047] Step 3: Biological mechanism-based encoder;

[0048] In this embodiment, the encoder incorporates the tone perception mechanism of a biological visual system to simulate the dynamic interaction between the classical receptive field and the inhibition-facilitation receptive field in the visual cortex, thereby enhancing the ability to jointly model tissue structure and color information in pathological images. In a biological visual system, the central part of the visual cortex's receptive field is called the classical receptive field, and the periphery is called the non-classical receptive field. The inhibition-facilitation receptive field belongs to the non-classical receptive field. When a large number of stimuli are generated within the classical receptive field, the main region of the inhibition-facilitation receptive field exhibits inhibitory characteristics towards the classical receptive field, but the terminal region with the optimal orientation exhibits concentrated facilitation characteristics towards the classical receptive field. The scale of the receptive field varies at different locations in the visual cortex, and the feedback effect from higher to lower visual cortexes also needs to be considered. Therefore, a multi-scale modulation model is used to simulate various states. Specifically, as... Figure 4 and Figure 5 As shown, the input image is processed by reflection padding, convolution, instance normalization, and nonlinear activation to obtain the depth feature map. .

[0049] This invention employs a Gaussian difference function model. To represent the inhibition-facilitation receptive field response, butterfly-shaped receptive fields with different orientations are used to represent the lateral inhibition and end-region facilitation structures of the inhibition-facilitation receptive field, using a Gaussian difference function model. The expression is as follows:

[0050]

[0051] in Indicates coordinate position, For a classic experience of the wild, The non-classical receptive field is 3 to 5 times the size of the classical receptive field, representing the size of the inhibition-facilitation receptive field at different scales. Studies have shown that the non-classical receptive field is 3 to 5 times the size of the classical receptive field. The values ​​are 3, 4, and 5 respectively. Calculate the inhibition amount of the inhibition-facilitating receptive field. The expression is as follows:

[0052]

[0053]

[0054]

[0055] in To suppress the weight, To determine the inhibition range of the inhibition-facilitating receptive field, This is a numerical stability constant used to prevent the denominator from being zero. It is used to calculate the facilitation factor of the suppression-facilitation receptive field. The expression is as follows:

[0056]

[0057]

[0058]

[0059] in for Four-directional butterfly-shaped sensory fields, To suppress the facilitating range of the facilitating receptive field, Representation of deep feature map The median coordinate is The value of the point, It is a numerical stability constant used to prevent the denominator from being zero.

[0060] At this point, we obtain the parameters of the effect of the suppression-facilitation receptive field on the classical receptive field. The expression is:

[0061]

[0062] in This indicates the proportion of cells in the visual cortex that play both inhibitory and facilitative roles; based on research, it is set to 0.2.

[0063] Considering the diversity of cells in the visual cortex and the feedback effect from higher to lower visual cortexes, this invention further superimposes influence parameters at multiple scales, resulting in a set of influence parameters of the inhibition-facilitation receptive field on the classical receptive field at different scales. The expression is:

[0064]

[0065] in To suppress the size of the facilitated receptive field relative to the classical receptive field, Take respectively Modulated characteristic response The expression is:

[0066]

[0067] in This represents the connection strength between the suppression-facilitated receptive field and the classical receptive field. The effect of the suppression-facilitated receptive field on the classical receptive field decreases with increasing distance; based on research, a value of 0.4 is used. Finally, the modulated characteristic response is... The signal is fed into a feature fusion layer, where it undergoes convolution, instance normalization, and a non-linear activation function to obtain the encoder output. :

[0068]

[0069] Step 4: Construct a virtual staining decoder based on frequency separation attention. During the decoding stage, high-frequency structural features are continuously extracted through the differential information between the original input and the stepwise reconstruction results. Combined with the stable transmission of low-frequency color features and attention-guided fusion, high-fidelity reconstruction of the virtual staining image is achieved.

[0070] In this embodiment, inspired by the idea that high-frequency details can be extracted step by step through residual information and low-frequency information needs to be stably maintained during image reconstruction, this invention proposes a virtual coloring decoder structure based on frequency separation attention, such as... Figure 6 As shown, this decoder no longer relies solely on upsampling operations to directly recover the output image during the decoding stage. Instead, it divides the encoded features into high-frequency structural information and low-frequency color information, and performs differential processing and attention-guided fusion on the two types of features. This allows for the simultaneous high-fidelity reconstruction of tissue structure details and stable maintenance of the overall color distribution during the virtual staining process.

[0071] In the high-frequency structure decoding path, the decoder upsamples features through step-by-step reconstruction and generates high-frequency residual information using the difference between the step-by-step reconstruction results and the original input features. This difference information mainly reflects local details such as tissue boundaries and cell contours. By continuously subtracting the reconstruction results after step-by-step upsampling and downsampling from the original input, the high-frequency structure information is extracted and refined layer by layer. Furthermore, a channel-space joint attention module (CSAM) is used to calculate the weights of different channels in the high-frequency feature map using channel attention and the weights of feature maps at different spatial locations using spatial attention. The calculated weight parameters are then adaptively used to weight and fuse the difference information to generate high-frequency information. This process can be described as generating high-frequency structural features by utilizing the difference between input features and reconstructed features, thereby suppressing structural blurring and artifact interference introduced during upsampling and enhancing the clarity and coherence of tissue structures. High-frequency information The extraction process can be represented as:

[0072]

[0073] in and These represent upsampling operations including deconvolution and activation layers, and downsampling operations including convolution and activation layers, respectively. This represents the feature map composed of convolution and nonlinear activation after upsampling and downsampling, used to transform differential residual information into high-frequency structural features. The output of the high-frequency channel... The generation process can be represented as:

[0074]

[0075] in, This indicates splicing, used to fuse high-frequency features from different levels.

[0076] In the low-frequency color decoding pathway, the decoder focuses on the stable transmission of low-frequency features related to the overall staining distribution. It utilizes a channel attention mechanism (CAM) module to generate channel weights and adaptively recalibrate color-related features. This enhances color channels consistent with the target staining style while suppressing channels with gradual color changes or those unrelated to staining. This maintains the spatial color continuity of the pathological image and reduces the risk of color drift or cross-regional color inconsistencies during virtual staining. (Low-frequency information) The extraction process can be represented as:

[0077]

[0078] in This indicates an upsampling operation including deconvolution and activation layers. The output of the low-frequency channel... The generation process can be represented as:

[0079]

[0080] After completing the superposition of features from the high-frequency and low-frequency paths, reflection filling and... Convolutional and hyperbolic tangent activation layers adjust the size and number of channels of the output features to obtain an image of the same size as the input image. The specific superposition process is as follows:

[0081]

[0082] in For reflective filling layer, It is a hyperbolic tangent activation layer to prevent unbounded growth of pixel values.

[0083] Step 5: Construct a composite loss function consisting of adversarial loss, cycle consistency loss, and identity loss, and perform parameter optimization training on the virtual coloring model;

[0084] In a preferred embodiment, adversarial loss (LossAdv) based on least squares error is employed to improve network training stability and enhance the realism of the generated images. This applies to the IHC→H&E generator. Its resistance loss can be expressed as:

[0085]

[0086] in For the generated virtual H&E image, For a discriminator used to distinguish between real and virtual images, For a real H&E image, similarly, a generator for H&E→IHC can be used. Define the corresponding adversarial loss.

[0087] Preferably, to ensure the generator maintains consistency in its organizational structure during image style transfer, the following is used: The cycle consistency loss defined by the norm can be expressed as:

[0088]

[0089] in For IHC images, For H&E images, and These are generators for the IHC→H&E and H&E→IHC directions, respectively. and To reconstruct the image.

[0090] Preferably, to further maintain the consistency of color mapping and reduce irrelevant style shifts, an identity loss is added, which can be expressed as:

[0091]

[0092] in This indicates that a real H&E image is input into the generator that processes IHC to H&E. The output should remain almost unchanged; if changes occur, they should be compensated for by identity loss. Identity loss is used to suppress additional changes produced by the generator when no transformation is needed, making the colors more stable.

[0093] Based on the above loss terms, the total loss function of the CycleGAN model can be further expressed as:

[0094]

[0095] in The sum of losses from two-way confrontation. For cycle consistency loss weights, Weight is lost due to identity loss.

[0096] This composite loss function can ensure the realism of the generated image style while preserving the tissue structure information of the original IHC image to the greatest extent, thus obtaining a virtual H&E image that is more consistent with pathological characteristics.

[0097] Step 6: Train the model proposed in this method on the training set obtained in Step 1, optimize the model parameters using the loss function in Step 5, evaluate the final training effect of the model using the test set, and output the visualized virtual coloring results of the model on the test set of the proprietary dataset and the BCI dataset.

[0098] In another aspect, the present invention provides a H&E image enhancement system based on biological mechanisms and frequency attention, comprising the following modules:

[0099] The image data processing module is used to acquire a virtual coloring dataset, preprocess the images contained in the dataset, generate input image data, and simultaneously divide the dataset.

[0100] The H&E image virtual coloring module is used to construct an H&E image virtual coloring model based on biological tone perception and frequency separation attention. It extracts features from the input image data to obtain the virtual-colored H&E image, thus completing the enhancement.

[0101] The loss training module is used to construct a composite loss function composed of adversarial loss, cycle consistency loss, and identity loss, to optimize the parameters of the H&E image virtual coloring model and to perform testing and evaluation.

[0102] To verify the effectiveness of the virtual staining method proposed in this invention in real pathological image scenarios, this embodiment experimentally verifies the method on both a proprietary pathological image dataset and a publicly available pathological image dataset. The proprietary pathological image dataset was provided by the Cypath-RB (Centre de Pathologie, Villeurbanne, France) and contains pathological slide images derived from normal appendix tissue, as shown in the image below. Figure 10 As shown in the figure, the proprietary dataset comprises 44 pairs of whole-slide images (WSIs), corresponding to immunohistochemical staining images and hematoxylin-eosin staining images, respectively. All images are derived from the same tissue section or adjacent consecutive sections, exhibiting highly consistent tissue structures, but showing significant differences in staining methods and color distribution.

[0103] Furthermore, to further verify the generalization ability of the method of the present invention under different data source conditions, this embodiment also conducted experimental verification on the BCI pathological image dataset. The BCI dataset is a publicly available pathological image dataset, which includes registered immunohistochemical images and corresponding hematoxylin-eosin staining images, and is suitable for virtual staining and cross-modal mapping tasks of pathological images.

[0104] The scanning resolution of the full-field pathological images is 0.25. Pixels, corresponding to optical magnification of Images are stored in .svs format and support multi-resolution hierarchical access. During the experiment, 256×256 image patches were extracted and cropped from the original full-view pathological images as model input. These patches were then divided into training and testing sets on both our own dataset and the BCI dataset to ensure the objectivity and reliability of the experimental results.

[0105] In this embodiment, the virtual coloring model proposed in this invention was trained and tested on an NVIDIA GeForce RTX 4090 graphics processor. The experimental environment consisted of Python 3.9.13 and PyTorch 2.5.0. The model was trained using the Adam optimizer with an initial learning rate of 0.0002. There were 200 training epochs, with the learning rate remaining constant for the first 100 epochs and a linear decay strategy used for the last 100 epochs. The batch size was set to 4, and the input image size was 256×256.

[0106] To quantitatively evaluate the performance of the virtual staining method proposed in this invention, this embodiment employs multiple evaluation metrics for comprehensive analysis of the generated results. These include: Structural Similarity Index (SSIM) and Multiscale Structural Similarity Index (MS-SSIM) to measure structural consistency between the generated and real images; Peak Signal-to-Noise Ratio (PSNR) and Mean Absolute Error (MAE) to evaluate pixel-level reconstruction quality; and Perceptual Similarity Index (LPIPS) to reflect the consistency of overall visual perception and color distribution. Tables 1 and 2 present the quantitative evaluation results of different virtual staining methods on the proprietary pathological image dataset and the BCI dataset, respectively. It can be seen that the virtual staining method proposed in this invention achieves superior structural consistency, pixel reconstruction quality, and visual perceptual similarity on both datasets.

[0107] Table 1: Performance comparison of existing image virtual coloring algorithms on our own dataset

[0108]

[0109] Table 2: Performance comparison of existing image virtual coloring algorithms on the BCI dataset

[0110]

[0111] To further verify the visual effect of the method of the present invention in virtual staining tasks, combined with Figure 7 , Figure 8 and Figure 9 Qualitative analysis was performed on the visualization results of the virtual H&E image shown. Experimental results show that the method of the present invention can effectively improve the overall color consistency and reduce color drift between adjacent areas while maintaining the continuity and detail integrity of the tissue structure. Furthermore, from a frequency domain perspective, the image generated by this method better preserves the high-frequency signals of the original image in the spectrogram. The high-frequency signals exhibit a dispersed and uniform distribution, which means that morphological information such as cell nucleus boundaries and glandular contours are better preserved. Simultaneously, observations were made... Figure 8 The image obtained after Fast Fourier Transform (FFT) shows that the image generated by this method does not exhibit the checkerboard-like abnormal clustering of spectral signals, indicating that generator artifacts are effectively suppressed. Combined with the above quantitative and qualitative analysis, it can be demonstrated that even in cases with complex tissue structures and significant staining differences, the virtual staining image generated by this method maintains a stable tonal distribution and clear tissue contours, exhibiting strong robustness.

Claims

1. A method for H&E image enhancement based on biological mechanisms and frequency attention, characterized in that, Includes the following steps: Step 1: Obtain the virtual staining dataset, preprocess the images contained in the dataset to generate input image data, and simultaneously divide the dataset. Specifically, this is done by: obtaining the virtual staining dataset and preprocessing the IHC and H&E images contained in the dataset; performing tissue region extraction on the full-view image and removing the background region. Subsequently, the images were cropped within the effective organization area to obtain image patches of the same size, and the dataset was divided into training and test sets proportionally. Step 2: Construct a virtual coloring model for H&E images based on biological tone perception and frequency separation attention. Extract features from the input image data to obtain the virtually colored H&E image, thus completing the enhancement. Specifically, the virtual coloring model for H&E images is constructed using a generator and discriminator collaborative adversarial training method. The generator is used to realize the virtual coloring mapping between IHC images and H&E images, and the discriminator is used to distinguish the authenticity of the generated image and the real coloring image, jointly optimizing the virtual coloring effect. The generator adopts an overall network architecture of encoder-residual connection-decoder, wherein the encoder performs feature encoding and representation learning on the preprocessed IHC and H&E images, the decoder reconstructs the image representation of the target stained domain in the feature space, and the encoder and decoder transfer feature information through residual connection; The generator incorporates a biological tone perception mechanism and a frequency separation attention mechanism to perform targeted modeling and control of color and structural information in pathological images. The biological tone perception mechanism uses an encoder that incorporates biological tone perception to encode the input image to obtain encoded features. In the feature extraction stage, a color perception modulation structure that simulates the inhibition-facilitation receptive field mechanism is introduced to enhance color discrimination ability and maintain tissue structure consistency. The frequency separation attention mechanism utilizes a decoder based on the frequency separation attention mechanism to reconstruct the encoded features step by step, generating a virtual stained image; Step 3: Construct a composite loss function consisting of adversarial loss, cycle consistency loss, and identity loss, perform parameter optimization training on the H&E image virtual coloring model, and conduct testing and evaluation.

2. The H&E image enhancement method based on biological mechanisms and frequency attention according to claim 1, characterized in that, The aforementioned bio-tone perception mechanism introduces a bio-tone perception encoder that performs preliminary feature extraction on the preprocessed IHC and H&E images at the input layer. Then, it introduces an inhibition-facilitation receptive field modulation module, simulating the perceptual characteristics of the biological visual system, into the preliminary features to adaptively adjust them, as detailed below: First, the input image is sequentially processed through reflection filling, convolution, instance normalization, and nonlinear activation to obtain the depth feature map. ; Gaussian difference function model The inhibition-facilitation receptive field response is represented by butterfly-shaped receptive fields with different orientations, which represent the lateral inhibition and terminal facilitation structures of the inhibition-facilitation receptive field. Indicates coordinate position, The size of the inhibition-facilitation receptive field relative to the classical receptive field at different scales; The inhibition range of the inhibition-facilitation receptive field, where the response When it is a non-negative number Place Otherwise, set to zero. and The ratio of the positions is used as the position. Suppression weights ; depth feature map With the inhibition weight Convolution results in the amount of suppression by the facilitated receptive field. ; To suppress the facilitating range of the facilitating receptive field, where the response are non-negative and When located in the butterfly-shaped sensory field Place Otherwise, set to zero, where Indicates the first The butterfly-shaped sensory field area faces at various angles, and The ratio as position Facilitated weight , Representation of deep feature map The median coordinate is The value of the point will be used to define the depth feature map. Weight of facilitated quantity Convolution yields the facilitation value for one orientation. The maximum value among the facilitation values ​​for all orientations is then calculated to obtain the facilitation value of the inhibition-facilitation receptive field. ; At this point, the inhibition amount will be... and By subtracting the values, we obtain the parameters representing the effect of the inhibition-facilitation receptive field on the classical receptive field. , This indicates the ratio of cells in the visual cortex that play inhibitory and facilitative roles. Different scales The influence parameters of the suppression-facilitation receptive field on the classical receptive field are obtained by superimposing the following parameters. Using depth feature maps and The characteristic response after suppression-facilitated receptive field modulation is obtained by subtraction. , This indicates the connection strength between the inhibition-facilitation receptive field and the classical receptive field; Modulated characteristic response The signal is fed into a feature fusion layer, where it undergoes convolution, instance normalization, and a non-linear activation function to obtain the encoder output. .

3. The H&E image enhancement method based on biological mechanisms and frequency attention according to claim 2, characterized in that, The decoder based on the frequency separation attention mechanism constructs a dual-channel decoding path based on the frequency separation idea, including a high-frequency decoding path and a low-frequency decoding path. In the high-frequency decoding path, structural information is continuously extracted by performing residual differential processing on the input features and the progressively upsampled results. In the low-frequency decoding path, the continuity and consistency of the overall color distribution of the pathological image are maintained by stabilizing the input features and adjusting the channel-level response, as detailed below: In the high-frequency decoding path, the decoder upsamples features through step-by-step reconstruction and generates high-frequency residual information using the difference between the step-by-step reconstruction results and the input features. This difference information is obtained by continuously subtracting the reconstruction results after step-by-step upsampling and downsampling from the original input, thereby extracting and refining high-frequency structural information layer by layer. Furthermore, the Channel-Spatial Joint Attention (CSAM) module calculates the weights of different channels in the high-frequency feature map using channel attention and the weights of feature maps at different spatial locations using spatial attention. The calculated weights are then adaptively weighted and fused to generate high-frequency information. The process involves extracting high-frequency information at each level, concatenating the high-frequency information from each level, performing a deconvolution operation, and then applying an activation function to obtain the output of the high-frequency channel. ; In the low-frequency color decoding path, the decoder stably transmits features related to the overall color distribution and uses the Channel Attention (CAM) module to calculate the feature responses of different channels in the feature map for recalibration and reweighting to generate low-frequency information. Finally, the low-frequency information from each level is concatenated and then deconvolutioned. After passing through an activation function, the output of the low-frequency channel is obtained. ; Will and After element-wise addition, reflection filling is applied, followed by convolution. Finally, hyperbolic tangent activation is used to adjust the size and number of output features, resulting in an image of the same size as the input image. Complete H&E image enhancement.

4. The H&E image enhancement method based on biological mechanisms and frequency attention according to claim 3, characterized in that, The adversarial loss is used to constrain the virtual staining image output by the generator to closely approximate the real staining image in terms of overall distribution, thereby improving the realism and naturalness of the generated results; the cycle consistency loss is used to constrain the bidirectional mapping relationship between different staining domains, maintaining the consistency of tissue structure information in the virtual staining process; the identity loss is used to suppress color mapping changes and enhance the stability of the model in terms of color preservation.

5. A H&E image enhancement system based on biological mechanisms and frequency attention, used to implement the H&E image enhancement method according to any one of claims 1 to 4, characterized in that, Includes the following modules: The image data processing module is used to acquire a virtual staining dataset, preprocess the images contained in the dataset, generate input image data, and simultaneously divide the dataset. The H&E image virtual staining module is used to construct an H&E image virtual staining model based on biological tone perception and frequency separation attention. It extracts features from the input image data to obtain the virtual-stained H&E image, thus completing the enhancement. The loss training module is used to construct a composite loss function composed of adversarial loss, cycle consistency loss, and identity loss, to optimize the parameters of the H&E image virtual coloring model and to perform testing and evaluation.

Images