A virtual staining verification method for autofluorescence and HE staining image registration

Abstract

The application discloses a kind of virtual dyeing verification methods for autofluorescence and HE staining image registration, including marking paper module, slice buckle module, image acquisition module, image registration module, virtual dyeing module and verification output module.The virtual dyeing verification method for autofluorescence and HE staining image registration of the application introduces high-precision marking paper as registration reference, combines deep learning model with image processing algorithm, realizes the accurate registration of two modal images and virtual dyeing verification.The scheme aims to solve the registration error problem caused by tissue deformation or dyeing difference in traditional dyeing technology, ensure that virtual dyeing result and real dyeing image are highly consistent in space, and provide reliable basis for pathological diagnosis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence image processing technology, especially the image processing of biological HE staining, and particularly to a virtual staining verification method for autofluorescence and HE staining image registration. BACKGROUND

[0002] Pathological examination is the "gold standard" of tissue examination in clinical practice, which makes pathological sections of visualized tissue and cell structure through dye or fluorescent labeling to conduct microscopic evaluation of tissues. Pathological examination provides core basis in key links such as diagnosis, treatment plan selection, efficacy evaluation and surgical margin judgment. HE staining is one of the most commonly used staining techniques in pathology, which specifically stains cell nuclei and cytoplasm by alkaline dye hematoxylin and acid dye eosin, making cell structure clear and visible. Its principle is based on the combination of acid substances in the cell nucleus with hematoxylin to form blue, and the combination of alkaline components in the cytoplasm and matrix with eosin to form red, which form a sharp contrast, making various tissue structures more intuitive. HE staining is widely used in disease diagnosis, mechanism research and efficacy evaluation.

[0003] However, the current preparation process of pathological sections is relatively cumbersome, involving many steps such as sample fixation, dehydration, transparentization treatment, wax immersion, embedding, sectioning, baking, dewaxing, staining, retransparency and final mounting. This process not only consumes time and effort, but also causes certain pollution to human health and environment during operation with substances such as benzene, aldehyde and paraffin, which has attracted widespread attention from all walks of life.

[0004] With the large-scale digitization of pathological sections and the development of artificial intelligence technology, the cooperation between artificial intelligence and pathology is closer, and virtual staining technology is a method of digital staining and enhancement of biological samples using computer algorithms and image processing technology. Virtual staining technology simulates the traditional staining effect through artificial intelligence algorithms, analyzes the unstained or partially stained sample images, and generates images similar to the traditional staining effect, thereby reducing the dependence on physical dyes, with the advantages of no dye, high efficiency, multi-modal, etc. Among them, the digital staining of biological autofluorescence images of unlabeled sample tissues to achieve the same staining image of the sample is a virtual staining scheme that is currently used more in research.

[0005] While virtual staining technology has shown great potential in the current field of medical image processing, it faces numerous challenges in verifying its accuracy. First, verifying the reliability of virtual staining results is urgent. For virtual stained images to prove their accuracy, they must be precisely compared and verified with pathological stained images. However, image registration between the two is a critical obstacle. During the actual staining process, pathological tissues are affected by various factors, such as batch differences in staining reagents, slight variations in staining time, and uneven thickness of tissue sections, resulting in variations in color, texture, and tissue structure in each pathological stained image. Achieving precise registration requires overcoming a series of complex problems, including differences in image position, grayscale, and geometric deformation. Currently, there is no mature method that can guarantee a perfectly matched image for verification. Furthermore, the lack of a unified standard for comparing and verifying virtual and pathological stained images leads to different research teams and medical institutions using different methods and indicators. This significantly increases the difficulty of judging the reliability of virtual staining results, making it difficult to reach a widely accepted accuracy verification conclusion and hindering the further promotion and clinical application of virtual staining technology. Therefore, how to verify the reliability of the obtained virtual staining results is a problem that urgently needs to be solved. The image matching criteria between the two is a key issue that needs to be addressed. How to obtain matching images for verification is a problem that needs to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a virtual staining verification method for registering autofluorescence and HE staining images. This addresses the issue raised in the background section where the lack of a unified standard for comparing and verifying virtual staining images with pathological staining images leads to inconsistencies in methods and indicators used by different research teams and medical institutions. This significantly increases the difficulty of judging the reliability of virtual staining results, hindering the formation of widely accepted accuracy verification conclusions and impeding the further promotion and clinical application of virtual staining technology. Therefore, how to verify the reliability of obtained virtual staining results is a pressing problem that needs to be solved. The image matching criteria between the two are a key issue, and obtaining matching images for verification is a problem that needs to be addressed.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a virtual staining verification method for registering autofluorescence and HE staining images, comprising a labeling paper module, a slide clipping module, an image acquisition module, an image registration module, a virtual staining module, and a verification output module, wherein each module includes;

[0008] The marking paper module includes a high-contrast marking dot array and a design to prevent mismatching.

[0009] The slice buckle module rigidly connects the pathological slice and the edge of the marker paper through buckling.

[0010] The image acquisition module acquires the marker paper image of the autofluorescence image and the HE staining image as a registration reference.

[0011] The image registration module matches the corresponding points in the autofluorescence image and the HE staining image based on the marker point features of the marker paper image, maps the autofluorescence image to the HE staining image coordinate system through a transformation matrix calculation method, and generates a registered autofluorescence image.

[0012] The virtual staining module inputs the registered autofluorescence image into a virtual staining model and generates a virtual HE staining image.

[0013] The verification output module calculates the structural similarity index and the histogram overlap degree of the virtual HE staining image and the HE staining image, and outputs the virtual staining result and a registration error report.

[0014] The marker paper module adopts an asymmetric coding design, including: a transparent polyester film substrate (75mm×25mm); 7×2 grid-distributed black cross-shaped marker points (diameter 1mm, spacing 10mm);

[0015] The upper left corner marker point (M11) is provided with a triangular notch (bottom length 0.8mm, height 0.5mm); the lower right corner marker point (M72) is provided with a circular notch (diameter 1.0mm); and each marker paper center is additionally provided with an asymmetric red cross identification.

[0016] Preferably, the slice buckle is provided with an elastic locking mechanism, including: four groups of hidden plug-in buckles, each group being composed of a male buckle (T-shaped hook, 8mm×0.3mm×0.2mm) and a female buckle (inverted T-shaped groove, 8mm×0.3mm×0.2mm); the male buckle has a 45° guide bevel, and the female buckle has a 30° guide bevel; the thickness of the elastic arm root is 0.15-0.2mm, and the buckling stroke is 0.2-0.25mm.

[0017] Preferably, the image acquisition module is implemented by a multispectral microscope, and the acquisition parameters include: the autofluorescence image acquisition wavelength range is 380-700nm; the HE staining image is obtained by standard bright field imaging; and the image resolution is set to 0.25μm / pixel.

[0018] Preferably, the image registration module performs the following processing procedures: extracting marker point features based on the SIFT algorithm; excluding mirror image flip mismatching by using notch features; calculating a rigid transformation matrix by using the least squares method; and controlling the registration accuracy to be ≤0.05pixel.

[0019] Preferably, the virtual staining module adopts a deep learning model with a U-Net structure, and comprises: an input registered autofluorescence image; and an output of a virtual HE staining image (SSIM>0.95).

[0020] Preferably, the evaluation index of the verification output module comprises: a structural similarity index (SSIM) threshold value >0.95; and a histogram overlap degree >90%.

[0021] Compared with the prior art, the virtual staining verification method has the following beneficial effects:

[0022] The virtual staining verification method for registration of autofluorescence and HE staining images provided by the present application realizes accurate registration and virtual staining verification of two modal images by introducing high-precision marking paper as a registration reference, combining a deep learning model and an image processing algorithm. The scheme aims to solve the registration error problem caused by tissue deformation or staining differences in traditional virtual staining technology, and ensures that the virtual staining result is highly consistent with the real staining image in space, thereby providing a reliable basis for pathological diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 FIG. 1 is a schematic diagram of a marking paper structure according to the present application;

[0024] Figure 2 FIG. 2 is a schematic diagram of a buckle structure of a male buckle and a female buckle according to the present application;

[0025] Figure 3 FIG. 3 is a schematic diagram of a buckle structure of a male buckle and a female buckle according to the present application;

[0026] Figure 4 FIG. 4 is a schematic diagram of a buckle structure parameter according to the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0028] Please refer to Figures 1-4 The present application provides a technical solution: a virtual staining verification method for registration of autofluorescence and HE staining images, comprising the following steps,

[0029] The marking paper module comprises a high-contrast marking point array and an anti-mismatch design.

[0030] The marking paper module adopts an asymmetric coding design, comprising:

[0031] Transparent polyester film substrate (75mm x 25mm); 7 x 2 grid distribution of black cross-shaped marker points (diameter 1mm, spacing 10mm); the upper left corner marker point (M11) is provided with a triangular notch (base length 0.8mm, height 0.5mm); the lower right corner marker point (M72) is provided with a circular notch (diameter 1.0mm); an asymmetric red cross mark is additionally attached to the center of each marker paper; a slice clasp module rigidly connects the pathological section and the edge of the marker paper through a clasp; the slice clasp is provided with an elastic locking mechanism, including:

[0032] Four groups of hidden plug-in clasps, each group is composed of a male buckle (T-shaped hook, 8mm x 0.3mm x 0.2mm) and a female buckle (inverted T-shaped slot, 8mm x 0.3mm x 0.2mm); the male buckle has a 45° guide bevel, and the female buckle has a 30° guide bevel; the thickness of the elastic arm root is 0.15-0.2mm, and the clamping stroke is 0.2-0.25mm; the hidden plug-in clasp uses a male buckle (convex) and a female buckle (concave) to cooperate, and the male buckle is locked by elastic deformation after being embedded in the female buckle, and can be separated by tilting at a specific angle, balancing stability and ease of operation. Adapt to the 1mm non-marking area at the edge of the marker paper to avoid affecting the optical marking function of the center area. One clasp (total of 4) is arranged at each corner of the marker paper, with a spacing of 15mm, to ensure even force and that the clasp is located at the edge of the non-marking area, with a spacing of ≥0.5mm from the marking area boundary to prevent interference.

[0033] An image acquisition module acquires the marker paper images of autofluorescence images and HE staining images as registration reference; the image acquisition module is realized by a multispectral microscope, and the acquisition parameters include: the autofluorescence image acquisition wavelength range is 380-700nm; the HE staining image is a standard bright field image; the image resolution is set to 0.25μm / pixel.

[0034] An image registration module matches corresponding points in the autofluorescence image and the HE staining image based on the marker point features of the marker paper image, maps the autofluorescence image to the HE staining image coordinate system through a transformation matrix calculation method, and generates a registered autofluorescence image; the image registration module performs the following processing procedures: extracting marker point features based on the SIFT algorithm; using notch features to exclude mirror image flip mismatches; using a least squares method to calculate a rigid transformation matrix; the registration accuracy is controlled to be ≤0.05pixel.

[0035] The virtual staining module inputs the registered autofluorescence image into the virtual staining model and generates a virtual HE staining image; the virtual staining module adopts a deep learning model with a U-Net structure, including: an encoder-decoder architecture, a total of 5 down-sampling and up-sampling layers; a loss function combination: a pixel-level L1 loss, a perception loss and an adversarial loss; an input of a registered autofluorescence image with a pixel size of 512x512; and an output of a virtual HE staining image (SSIM>0.95). The verification output module calculates the structural similarity index and the histogram overlap degree of the virtual HE staining image and the HE staining image, and outputs the virtual staining result and the registration error report; the evaluation indexes of the verification output module include: a structural similarity index (SSIM) threshold value>0.95; a histogram overlap degree>90%; and a registration error report including a Hausdorff distance (<5μm) and a root mean square error.

[0036] The implementation includes rigidly connecting a transparent polyester film marker paper (75mmx25mm) to a pathological section through four sets of hidden plug-in buckles. Among the 7x2 grid distributed black cross-shaped marker points (diameter 1mm, spacing 10mm) on the surface of the marker paper, a triangular notch (base length 0.8mm, height 0.5mm) is arranged at the upper left corner M11 point, a circular notch (diameter 1.0mm) is arranged at the lower right corner M72 point, and an asymmetric red cross mark is additionally attached to the center of each marker paper. The buckles are paired with T-shaped hook male buckles (8x0.3x0.2mm, 45° bevel) and inverted T-shaped slot female buckles (8x0.3x0.2mm, 30° bevel), with a root thickness of 0.15-0.2mm of the elastic arm to ensure that there is no displacement in the marker area after the section is fixed.

[0037] Synchronous acquisition using a multispectral microscope: 1) autofluorescence image (excitation wavelength 380-700nm, resolution 0.25μm / pixel); 2) HE staining brightfield image; 3) marker paper positioning image. The microscope stage temperature is kept constant (25±1℃) during acquisition, and the acquisition time for each field is controlled within 500ms to ensure that the image is free of motion blur. The identification success rate of all marker points in the marker paper image must be ≥99%, and automatic alarm is triggered when the number of un-identified marker points exceeds 1.

[0038] Based on the SIFT algorithm, the features of the marker points are extracted, and the mirror flip mis-matching is excluded by using the triangular / circular notch features. When calculating the rigid transformation matrix using the least squares method, the four corner points (M11, M71, M12, M72) are preferentially selected as the reference control points. During the registration process, the error is monitored in real time, and when the single-point registration deviation is >0.05pixel, the transformation matrix is automatically recalculated, and finally the registered autofluorescence image (512x512 pixels) is generated.

[0039] The registered image is input into the U-Net model to generate a virtual HE staining image using a combined loss function (L1+ perceptual+ adversarial loss). In the verification phase, the following are calculated: 1) structural similarity index SSIM (>0.95); 2) histogram overlap degree (>90%); 3) Hausdorff distance (<5 μm). Unqualified results are automatically marked and prompted for review, and qualified data generates an analysis report containing a registration error heat map.

[0040] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications, changes, variations, substitutions, and equivalents will occur to those of ordinary skill in the art without departing from the spirit and scope of the application as defined by the following claims and their equivalents.

Claims

1. A virtual staining verification method for registering autofluorescence and HE staining images, characterized in that, It includes a marking paper module, a slice clipping module, an image acquisition module, an image registration module, a virtual staining module, and a verification output module, each of which includes: The marking paper module includes a high-contrast marking dot array and a design to prevent mismatching. The slide clip module rigidly connects the pathological slide to the edge of the marking paper via clips. The image acquisition module acquires the marker paper images of autofluorescence images and HE staining images as registration references; The image registration module, based on the features of the marker points in the marker paper image, matches the corresponding points in the autofluorescence image and the HE staining image. Through transformation matrix calculation, the autofluorescence image is mapped to the coordinate system of the HE staining image to generate the registered autofluorescence image. The virtual staining module inputs the registered autofluorescence image into the virtual staining model and generates a virtual HE staining image; The verification output module calculates the structural similarity index and histogram overlap between the virtual HE staining image and the HE staining image, and outputs the virtual staining results and registration error report. The marking paper module adopts an asymmetric coding design, including: Transparent polyester film substrate, with dimensions of 75mm × 25mm; Black cross-shaped markers are distributed in a 7×2 grid, with a diameter of 1 mm and a spacing of 10 mm; The upper left corner marker (M11) has a triangular notch with a base length of 0.8mm and a height of 0.5mm; The lower right corner marker (M72) has a circular notch with a diameter of 1.0 mm; Each marking sheet has an asymmetrical red cross symbol added to its center; The slice buckle is equipped with an elastic locking mechanism, including: Four sets of concealed insert buckles, each set consisting of a male buckle and a female buckle paired together. The male buckle is a T-shaped hook with dimensions of 8mm × 0.3mm × 0.2mm, and the female buckle is an inverted T-shaped groove with dimensions of 8mm × 0.3mm × 0.2mm. The male buckle has a 45° guide angle, and the female buckle has a 30° guide angle. The thickness at the root of the flexible arm is 0.15-0.2mm, and the engagement stroke is 0.2-0.25mm; The evaluation metrics for the verification output module include: Structural similarity index > 0.95; Histogram overlap > 90%.

2. The virtual staining verification method for registering autofluorescence and HE staining images according to claim 1, characterized in that, The image acquisition module is implemented using a multispectral microscope, and the acquisition parameters include: Autofluorescence image acquisition wavelength range: 380-700 nm; HE staining images were obtained using standard bright-field imaging. The image resolution was set to 0.25 μm / pixel.

3. The virtual staining verification method for registering autofluorescence and HE staining images according to claim 1, characterized in that, The image registration module performs the following processing flow: Extracting features of marker points based on the SIFT algorithm; Use gap features to eliminate mirror-flipped mismatches; The rigid transformation matrix is ​​calculated using the least squares method; Registration accuracy is controlled within ≤0.05 pixels.

4. The virtual staining verification method for registering autofluorescence and HE staining images according to claim 1, characterized in that, The virtual coloring module employs a deep learning model with a U-Net structure and includes: Input the registered autofluorescence image; The output is a virtual HE staining image.

Images