A rapid intraoperative diagnostic method using digital frozen section without HE staining

Abstract

This invention provides a rapid intraoperative diagnostic method for digital frozen sections without HE staining, belonging to the field of intraoperative real-time auxiliary diagnostic technology. The method includes: scanning frozen sections, image preprocessing, constructing a virtual staining neural network model, constructing a deep learning diagnostic model, displaying results and facilitating human interaction, and generating an intraoperative report. This invention provides a rapid intraoperative diagnostic method for digital frozen sections without HE staining by directly scanning unstained frozen section images and inputting them into a virtual staining model, omitting the traditional HE staining step and improving intraoperative diagnostic efficiency. This invention uses virtual stained images directly input into a pre-trained deep learning model for case diagnosis for real-time diagnosis and lesion identification. This invention solves the diagnostic consistency problem between virtual and real images, realizing cross-modal AI diagnosis.

Description

Technical Field

[0001] This invention relates to the field of intraoperative real-time auxiliary diagnostic technology, and in particular to a rapid intraoperative auxiliary diagnostic method for digital frozen sections without HE staining. Background Technology

[0002] Existing frozen section pathology diagnostic process:

[0003] Tissue sample → frozen section preparation → HE staining → microscopic examination → pathological diagnosis;

[0004] a. Tissue sample collection: Surgeons obtain tissue samples during surgery;

[0005] b. Frozen section preparation: Rapidly prepare tissue sections using a cryostat;

[0006] c. HE staining: Technicians perform routine HE staining on the slides, which usually takes 6-10 minutes;

[0007] d. Manual examination under a microscope: The pathologist examines the stained slides under a microscope to determine whether they are malignant tumors and whether extensive resection is necessary;

[0008] e. Provide diagnostic opinions: Feedback the results to the surgical team.

[0009] Existing methods for diagnosing frozen pathological sections have the following problems:

[0010] 1. The time required for HE staining cannot be reduced, which limits the efficiency of intraoperative diagnosis;

[0011] 2. The staining process relies on manual operation, which is subject to fluctuations, and the image quality is easily affected by contamination or uneven staining;

[0012] 3. Pathologists' interpretation of slides is based on subjective human judgment, which carries the risk of fatigue and misinterpretation;

[0013] 4. It cannot provide real-time, automated intraoperative diagnostic assistance.

[0014] Therefore, there is an urgent need in this field for a technical solution that can improve the efficiency of intraoperative diagnosis, eliminate the need for manual staining, assist doctors in reading images through models, and provide automated intraoperative diagnostic assistance.

[0015] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0016] The purpose of this invention is to provide a technical solution that can improve intraoperative diagnostic efficiency, eliminate the need for manual staining, assist doctors in reading images through models, and provide automated intraoperative diagnostic assistance.

[0017] To achieve the above objectives, the present invention provides the following solution:

[0018] A rapid intraoperative diagnostic method for digital frozen sections without HE staining, characterized by comprising:

[0019] Step 1: Scan the frozen section;

[0020] Step 2: Image preprocessing;

[0021] Step 3: Construct a virtual coloring neural network model;

[0022] A virtual staining neural network model is constructed using the Pix2Pix model architecture, with an unstained image as input and a simulated HE staining image as output.

[0023] The virtual coloring neural network model was trained using a labeled dataset;

[0024] Step 4: Construct a deep learning diagnostic model; based on the ResNet / ViT structure, pre-trained using real HE slice data, with virtual stained images as input and category labels, lesion segmentation maps, and malignancy probability scores as outputs; supports tumor detection, boundary recognition, and human interaction;

[0025] Step 5: Results Display and Human Interaction; The original image, virtual HE image, and model diagnostic suggestions are displayed simultaneously through an interactive software interface; Doctors can choose to accept, modify, or add annotations.

[0026] Step 6: Generate intraoperative report.

[0027] Optionally, the scanned frozen sections specifically include:

[0028] High-resolution images of unstained frozen sections were acquired using a digital section scanner.

[0029] Optionally, the image preprocessing includes:

[0030] Image normalization and noise suppression.

[0031] Optionally, the generation of the intraoperative report includes:

[0032] Outputs standard format PDF reports, enabling information synchronization with hospital LES or PACS systems.

[0033] Optionally, the Pix2Pix model architecture includes: a generator and a discriminator;

[0034] The generator uses a U-Net structure, taking an autofluorescence image of an unstained frozen section as input and outputting a simulated HE-stained image. The U-Net network includes an encoder and a decoder structure. The encoder uses a combination of multiple convolutional layers, BatchNorm, and LeakyReLU to progressively downsample image features. The decoder uses transposed convolutional layers and skips connections with the corresponding encoder layers to preserve spatial information.

[0035] The discriminator uses the PatchGAN structure, with the input being a combination of the original image and the generated image, and the output being a true probability map representing the truthfulness of each image patch.

[0036] Optionally, the loss function adopts a combination of Pix2Pix standard loss functions, including:

[0037] Adversarial loss function: ;

[0038] Reconstruction loss function: used to maintain structural consistency between the generated image and the real HE image, defined as:

[0039] ;

[0040] The total loss function is:

[0041] .

[0042] G: Generator, used to generate fake images.

[0043] D: Discriminator, which determines whether an image is real or generated.

[0044] x: Input image (e.g., low-quality image or some kind of conditional input).

[0045] y: The ground truth image (e.g., a high-quality image).

[0046] λ: A hyperparameter used to weigh adversarial loss against reconstruction loss.

[0047] Optionally, the multi-task output branch of the deep learning diagnostic model is designed with three types of output modules based on the feature tensor output by the backbone network:

[0048] Classification branch: The fully connected layer outputs class labels, using Softmax activation;

[0049] Segmentation branch: Lightweight U-Net decoder, outputting a mask image of the same size as the original image;

[0050] Malicious probability branch: The fully connected layer outputs a regression probability value between 0 and 1, activated by Sigmoid;

[0051] The loss function of the deep learning diagnostic model adopts a multi-task joint training objective, and the overall loss function is as follows:

[0052]

[0053] Classification loss:

[0054] Segmentation loss:

[0055] Malicious prediction loss: ;

[0056] in, a,b,c Weighting coefficients are used to adjust the importance of each part of the loss.

[0057] y : Real tags;

[0058] : The predicted probability distribution;

[0059] P : Predicted segmentation region;

[0060] V： The actual segmented region;

[0061] N : Sample size;

[0062] p i : The true probability of malignancy for the i-th sample;

[0063] i : The probability predicted by the model.

[0064] Optionally, the deep learning diagnostic model outputs the following three types of data, which can be fully displayed in the system interface:

[0065] Category labels, output as text + confidence percentage;

[0066] Lesion segmentation map, with data format being a binary mask at the same resolution as the image;

[0067] The probability score of malignancy is a floating-point value ranging from 0 to 1.

[0068] Optionally, the image normalization includes:

[0069] Image size standardization: All images were acquired by scanning at 40x magnification and divided into uniformly sized image blocks of 256×256 pixels using a sliding window method to ensure consistent model input dimensions;

[0070] Pixel intensity normalization: Pixels are normalized using the ImageNet mean;

[0071] Background removal: The Otsu adaptive thresholding method is used to segment the background of the image, and the background region is excluded during training;

[0072] Non-organic background areas can be eliminated using threshold or color clustering methods;

[0073] The noise suppression includes:

[0074] Gaussian filtering: Applying a 5x5 Gaussian kernel to smooth the image;

[0075] Bilateral filtering: It considers both pixel distance and color distance, preserving edge structure while smoothing texture areas;

[0076] Median filtering: Removes interference from cell nucleus edges.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] This invention provides a rapid intraoperative diagnostic method for digital frozen sections without HE staining. It directly scans unstained frozen section images and inputs them into a virtual staining model, eliminating the traditional HE staining step and improving intraoperative diagnostic efficiency. Unlike traditional procedures that require HE staining, this invention uses a deep neural network to generate simulated HE-stained images, achieving a "zero-staining" digital alternative.

[0079] This invention uses virtual stained images as direct input to a pre-trained deep learning model for case diagnosis, enabling real-time diagnosis and lesion identification. Traditional auxiliary diagnostic models are trained based on real stained images; this invention solves the diagnostic consistency problem between virtual and real images, achieving cross-modal AI diagnosis.

[0080] This invention establishes an automated workflow system that integrates scanning, staining, and diagnosis, which is used to assist in intraoperative decision-making and saves more than 5-10 minutes of total diagnostic time. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a schematic diagram of the process for a rapid intraoperative auxiliary diagnosis method using digital frozen sections without HE staining, as provided in an embodiment of the present invention. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] The purpose of this invention is to provide a method that can improve intraoperative diagnostic efficiency, eliminate the need for manual staining, assist doctors in reading images through models, and provide automated intraoperative diagnostic assistance.

[0085] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0086] Example 1:

[0087] This invention provides a rapid intraoperative diagnostic method for digital frozen sections without HE staining, such as... Figure 1 As shown, it includes:

[0088] Step 1: Scan the frozen section;

[0089] Step 2: Image preprocessing;

[0090] Step 3: Construct a virtual staining neural network model; use the Pix2Pix model architecture to construct a virtual staining neural network model, with the input being an unstained image and the output being a simulated HE-stained image;

[0091] A. Model Architecture The virtual coloring neural network model described above adopts the Pix2Pix Generative Adversarial Network (GAN) architecture, which includes the following two main parts:

[0092] The generator uses a U-Net architecture. The input is an autofluorescence image of an unstained frozen section (segmented into 256×256 patches), and the output is a simulated HE-stained image. The U-Net network consists of an encoder and a decoder. The encoder uses a combination of multiple convolutional layers (Conv2D) + BatchNorm + LeakyReLU to progressively downsample image features. The decoder uses transposed convolutional layers (Deconv) for upsampling and skips connections with the corresponding encoder layers to preserve spatial information.

[0093] Discriminator: Using the PatchGAN structure, the input is a combination of the original image and the generated image (or a combination of the original image and the real HE image), and the output is a true probability map, representing the realism of each image patch.

[0094] B. Loss Function Design:

[0095] Using a combination of Pix2Pix standard loss functions, including:

[0096] Adversarial Loss:

[0097]

[0098] L1 reconstruction loss (L1 Loss): Used to maintain structural consistency between the generated image and the real HE image, defined as:

[0099]

[0100] The total loss function is:

[0101] ;

[0102] in, G : Generator, used to generate fake images;

[0103] D Discriminator: Used to determine whether an image is real or generated;

[0104] x Input image;

[0105] y : A realistic target image;

[0106] l : Hyperparameters used to balance adversarial loss and reconstruction loss.

[0107] C. Training settings;

[0108] Optimizer: Adam optimizer;

[0109] Learning rate: 0.0002;

[0110] Batch size: 16;

[0111] Epochs: 100 rounds;

[0112] Framework: Implemented based on PyTorch;

[0113] D. Evaluation indicators;

[0114] After the model training is complete, the following metrics are used to evaluate its performance:

[0115] SSIM (Structural Similarity Index);

[0116] PSNR (Peak Signal-to-Noise Ratio);

[0117] Color Distance (Lab color space);

[0118] Step 4: Construct a deep learning diagnostic model; based on the ViT-B / 16 structure, pre-trained on real HE slice data; input virtual stained images, output as category labels, lesion segmentation maps, and malignancy probability scores; supports tumor detection, boundary recognition, and human interaction;

[0119] A. Backbone network ViT-B / 16 (Vision Transformer): Suitable for extracting non-local semantic features from large-scale data, with stronger global perception capabilities.

[0120] B. Multi-task Heads: Based on the feature tensors output by the backbone network, three types of output modules are designed:

[0121] Classification branch: The fully connected layer (FC) outputs class labels and uses Softmax activation;

[0122] Segmentation branch: Lightweight U-Net decoder, outputting a mask image of the same size as the original image;

[0123] Malicious probability branch: The fully connected layer outputs a regression probability value between 0 and 1, which is activated by Sigmoid.

[0124] C. Loss Function Design: A multi-task joint training objective is adopted, and the overall loss function is as follows:

[0125]

[0126] Classification loss:

[0127]

[0128] Segmentation loss:

[0129] Malicious prediction loss:

[0130] in, a,b,c Weighting coefficients are used to adjust the importance of each part of the loss.

[0131] y： Authentic labels;

[0132] ： The predicted probability distribution;

[0133] P： Predicted segmentation region;

[0134] V： The actual segmented region;

[0135] N： Sample size;

[0136] p i ： The true malignancy probability of the i-th sample;

[0137] i ： The probability predicted by the model;

[0138] D. Training settings:

[0139] Optimizer: Adam optimizer;

[0140] Learning rate: 1e-4;

[0141] Batch Size: 16;

[0142] Epochs: 80;

[0143] Framework: Implemented based on PyTorch;

[0144] E. Evaluation Indicators:

[0145] Classification accuracy, AUC curve;

[0146] Divide the Dice coefficient and IoU score;

[0147] Mean Absolute Error (MAE) is the probability of a malignant error.

[0148] Diagnostic consistency index (expert consensus, kappa);

[0149] Step 5: Results Display and Human Interaction; The original image, virtual HE image, and model diagnostic suggestions are displayed simultaneously through an interactive software interface; Doctors can choose to accept, modify, or add annotations.

[0150] The model outputs the following three types of data, which can be fully displayed in the system interface:

[0151] A. Classification Output

[0152] Output format: Text + confidence percentage

[0153] Example:

[0154] AI Prediction: Papillary Thyroid Carcinoma

[0155] Confidence level: 92.4%;

[0156] B. Segmentation Output;

[0157] Data format: Binary mask (0 / 1) with the same resolution as the image;

[0158] Color visualization: The predicted area is covered with an opaque color;

[0159] C. Malignancy probability score (Regression Output);

[0160] Data type: Floating-point number, ranging from 0 to 1.

[0161] Step 6: Generate intraoperative report.

[0162] In one embodiment, the frozen section scanning specifically includes:

[0163] High-resolution images of unstained frozen sections were acquired using a digital section scanner.

[0164] In one embodiment, the image preprocessing includes:

[0165] Image standardization and noise suppression.

[0166] A. Image normalization:

[0167] Image size standardization: All images were acquired using a 40x magnification scan and divided into uniformly sized image patches of 256×256 pixels using a sliding window method to ensure consistent model input dimensions;

[0168] Pixel intensity normalization: Pixels are normalized using the ImageNet mean;

[0169] Background removal: Otsu's adaptive thresholding method is used for background segmentation of the image. Background regions (such as slide edges and areas without tissue) are excluded during training.

[0170] Non-organic background areas (typically pure white or light gray areas) are removed using threshold or color clustering methods (such as Otsu or K-means).

[0171] B. Noise suppression:

[0172] Gaussian Blur: Smooths an image using a 5x5 Gaussian kernel;

[0173] Bilateral Filter: It considers both pixel distance and color distance, preserving edge structure while smoothing textured areas;

[0174] Median Filter: Removes interference from cell nucleus edges.

[0175] In one embodiment, generating the intraoperative report includes:

[0176] Outputs standard format PDF reports, enabling information synchronization with hospital LES or PACS systems.

[0177] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0178] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for rapid aided diagnosis in digital frozen section without HE staining, characterized in that, Comprise: Step one: scan the frozen section; Step two: image preprocessing; Step three: build a virtual staining neural network model; Use the Pix2Pix model architecture to build a virtual staining neural network model, the input is an unstained image, and the output is an HE-like staining image; Train the virtual staining neural network model through the labeled dataset; Step four: build a deep learning diagnosis model; based on the ResNet / ViT structure, pre-train with real HE slice data, input virtual staining image, output class label, lesion segmentation map and malignant probability score; support tumor detection, boundary recognition and artificial interaction; Step five: result display and artificial interaction; display the original image, virtual HE image and model diagnosis suggestion through the interactive software interface; doctors can choose to accept, modify or add annotations; Step six: generate intraoperative report; The loss function adopts the standard loss function combination of Pix2Pix, including: Adversarial loss function: ; Reconstruction loss function: used to keep the generated image consistent with the structure of the real HE image, defined as: ； The total loss function is: ； wherein G : a generator for generating fake images; D : discriminator, for determining whether an image is real or generated; x : input image; y : real target image; Lambda : hyperparameters for balancing the adversarial loss and the reconstruction loss; The multi-task output branch of the deep learning diagnosis model is based on the feature tensor output by the backbone network, and three types of output modules are designed: Classification branch: full connection layer outputs class label, adopts Softmax activation; Segmentation branch: lightweight U-Net decoder, outputs a mask map with the same size as the original image; Malignant probability branch: full connection layer outputs a regression probability value between 0 and 1, adopts Sigmoid activation; The loss function of the deep learning diagnosis model adopts a multi-task joint training target, and the overall loss function is as follows: Classification loss: Segmentation loss: Malignant prediction loss: ; wherein, Alpha, beta, gamma : weight coefficient, used to adjust the importance of each part loss; y： Authentic labels; ： Predicted probability distribution; P： Predicted segmentation region; V： Real segmentation region; N： Sample size; p i ： true malignancy probability of the i-th sample; i ： Model predicted probabilities.

2. The method for rapid ancillary diagnosis in digital frozen section without HE staining according to claim 1, characterized in that, The scanning of the frozen section specifically includes: Use a digital slice scanner to collect high-definition images of the unstained frozen section.

3. The method for rapid ancillary diagnosis in digital frozen section without HE staining according to claim 1, characterized in that, The image preprocessing includes: Image standardization and noise suppression.

4. The method for rapid ancillary diagnosis in digital frozen section without HE staining according to claim 1, characterized in that, The generation of the intraoperative report includes: Output standard format PDF report, interface with hospital LES or PACS system to realize information synchronization.

5. The method for rapid ancillary diagnosis in digital frozen section without HE staining according to claim 1, characterized in that, The Pix2Pix model architecture includes: generator and discriminator; The generator uses U-Net structure, the input is the autofluorescence image of the unstained frozen section, and the output is the HE-like staining image; U-Net network contains encoder and decoder structure, the encoder uses multiple convolutional layers + BatchNorm + LeakyReLU combination, and gradually down-samples image features; the decoder uses transpose convolutional layer, which is connected with the corresponding encoder layer through skip connection to preserve spatial information; The discriminator uses PatchGAN structure, the input is the combination of the original image and the generated image, and the output is a real probability map representing the authenticity of each image patch.

6. The method for rapid ancillary diagnosis in digital frozen section without HE staining according to claim 1, characterized in that, The deep learning diagnosis model outputs the following three types of data, which can be displayed completely in the system interface: Class label, output form is text + confidence percentage; Lesion segmentation map, data format is binary mask with the same resolution as the image; Malignant probability score, data type is floating point value, range is 0~1.

7. The method for rapid ancillary diagnosis in digital frozen section without HE staining according to claim 3, characterized in that, The image standardization includes: Image size normalization: All images were scanned using 40x objective and divided into image patches with the size of 256x256 pixels by sliding window method to ensure the consistency of model input dimension; Pixel intensity normalization: Pixel normalization was performed using ImageNet mean; Background removal: Otsu adaptive threshold method was used to segment the background of images, and the background region was excluded during training; Non-tissue region background was removed by threshold or color clustering method; The noise suppression includes: Gaussian filtering: 5x5 Gaussian kernel was applied to smooth the image; Bilateral filtering: Both pixel distance and color distance were considered to preserve the edge structure while smoothing the texture region; Median filtering: The interference of the nucleus edge was removed.

Images