A method for three-dimensional reconstruction and visualization of nasal polyp tissue based on artificial intelligence recognition of continuous pathological sections

Abstract

The application discloses a method for stereoscopic reconstruction and visualization of nasal polyp tissue based on artificial intelligence recognition of continuous pathological sections, which comprises the following steps: strict continuous section of a reconstruction object; HE staining of the continuous sections; affine registration of the continuous pathological sections; one-by-one analysis of the registered continuous pathological sections; stacking according to the registration sequence, then processing, to obtain three-dimensional model data of the nasal polyp tissue structure; and visual rendering of the three-dimensional model data of the nasal polyp tissue structure, combined with a preliminary stereoscopic model, to obtain a final stereoscopic model of the nasal polyp tissue structure. The method can reconstruct and visualize the nasal polyp tissue, and can show the three-dimensional structure of the distribution characteristics of inflammatory cells and tissue structure of the nasal polyp in the spatial structure, so that the morphological structure content of the nasal polyp tissue is enriched, and morphological basis is provided for further exploring the occurrence and development mechanism of chronic rhinosinusitis with nasal polyps.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction and visualization technology of nasal polyp tissue, and in particular to a method for three-dimensional reconstruction and visualization of nasal polyp tissue based on artificial intelligence to identify continuous pathological sections. Background Technology

[0002] Chronic sinusitis with nasal polyps is a refractory nasal disease that seriously affects human health, affecting over 120 million people in China. It has a high recurrence rate and imposes a medical burden of over 100 billion yuan annually on society. Nasal polyp tissue specimens are important research materials for biomedical research on this disease and are key specimens for disease diagnosis and classification. Nasal polyp tissue imaging is an important tool for the diagnosis and research of nasal polyp tissue specimens.

[0003] Nasal polyp tissue imaging mainly includes two parts: staining and detection. (1) Staining methods include dye staining and antibody-based staining (such as immunohistochemical staining (IHC)). Among them, hematoxylin and eosin (H&E) staining, which belongs to the former, can show the characteristics of cytoplasm, nucleus and extracellular matrix. Compared with other staining methods, the staining effect is more stable and can more clearly identify various tissue types and morphological changes. It is the most basic and widely used technical method in histology and scientific research. (2) The detection part is still mainly based on the observation and reading of single tissue sections. Nasal polyps are a highly heterogeneous pathological tissue with uneven internal tissue distribution. The limitation of traditional tissue sections is that they can only reflect the situation of one section of the tissue, which is far from the three-dimensional morphology in living tissue. The information loss is huge and it is difficult to identify the whole picture.

[0004] To study the three-dimensional model of nasal polyp tissue, the current three-dimensional imaging technology has the following shortcomings: (1) Anatomical reconstruction based on CT or MRI is a relatively mature technology, but it can only observe the anatomical structure of organs and tissues and cannot effectively reflect the situation at the cellular level inside the tissue; moreover, nasal polyp tissue is small in volume and has a simple anatomical structure, so the application value of this method is limited. (2) In 2009, Zheng Zhongxi et al. performed multi-layer image acquisition within the slice thickness range and used image fusion to make the acquired area have higher clarity (authorization publication number: CN 101615289 B), which is essentially still within the scope of two-dimensional image acquisition and presentation. (3) In 2017, Yan Huanhuan et al. used continuous cutting IHC staining and layer-by-layer calibration to perform three-dimensional reconstruction of tissue (application publication number: CN 106918484A), but its main disadvantage is that the workload of layer-by-layer calibration of scanned images is large, resulting in high labor costs and relatively low practical value in clinical or scientific research. (4) In 2020, Liang Tingbo et al. used continuous sections, multicolor fluorescence staining, and alignment based on fluorescence channel signals to perform three-dimensional reconstruction and STL modeling and analysis of the region of interest (authorized publication number: CN 112113937A). Its main drawback is that it is difficult to present the morphological structure of blood vessels, glands, epithelium and other tissues in nasal polyp tissue, and it cannot efficiently and quantitatively analyze histopathological information including inflammatory cells and tissue regions. (5) In addition, although the tissue transparency staining microscope stereoscopic scanning imaging technology has the advantage of good tissue integrity, it also has the disadvantages of high hardware requirements, high cost and difficulty in popularization.

[0005] As our understanding of the two-dimensional pathological features of nasal polyp tissue deepens, exploring the spatial distribution characteristics of inflammatory cells and tissue structures within nasal polyp tissue, and revealing the three-dimensional structure of nasal polyps, not only enriches the morphological and structural content of nasal polyp tissue but also provides morphological evidence for further investigation into the mechanisms of occurrence and development of nasal polyps in chronic sinusitis. This invention addresses the two major needs of efficiently reconstructing and visualizing the three-dimensional structure of nasal polyp tissue by proposing a method for three-dimensional reconstruction and visualization of nasal polyp tissue based on artificial intelligence to identify continuous pathological sections. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a method for three-dimensional reconstruction and visualization of nasal polyp tissue based on artificial intelligence to identify continuous pathological sections, thereby solving the problems of the aforementioned traditional technologies. It presents a three-dimensional structure that shows the distribution characteristics of inflammatory cells and tissue structures of nasal polyps in space, which not only enriches the morphological structure of nasal polyp tissue, but also provides morphological evidence for further exploring the occurrence and development mechanism of nasal polyps in chronic sinusitis.

[0007] This invention is achieved using the following technical solution:

[0008] A method for three-dimensional reconstruction and visualization of nasal polyp tissue based on artificial intelligence to identify continuous pathological sections includes the following steps:

[0009] S1: Perform strict continuous slicing on the reconstructed object;

[0010] S2: Perform HE staining on serial sections according to standard procedures to obtain pathological sections;

[0011] S3: Using the SimpleITK library in Pyon software, affine registration is performed on consecutive pathological sections, followed by stereoscopic reconstruction to obtain a preliminary stereoscopic model.

[0012] S4: Using the whole slide subtype auxiliary interpretation system for chronic sinusitis with nasal polyps, the registered continuous pathological slides were analyzed one by one to obtain continuous two-dimensional digital pathological images.

[0013] S5: Stack the continuous two-dimensional digital pathological images according to their registration order to obtain the three-dimensional raw data of cells and tissue regions;

[0014] S6: Process the raw three-dimensional data of cells and tissue regions to obtain three-dimensional model data of nasal polyp tissue structure;

[0015] S7: Visualize and render the 3D model data of the nasal polyp tissue structure obtained in step S6, and combine it with the preliminary 3D model obtained in step S3 to obtain the final 3D model of the nasal polyp tissue structure.

[0016] Furthermore, in step S1, the reconstruction object is the excised nasal polyp tissue completely removed during functional endoscopic sinus surgery in a patient with chronic sinusitis and nasal polyps who has been embedded in paraffin.

[0017] Furthermore, in step S1, the specific operation steps for strictly continuous slicing are as follows: the wax block of the reconstructed object is continuously sliced ​​into conventional paraffin-embedded tissue slices using a microtome, and numbered sequentially from the root to the bottom.

[0018] Furthermore, the paraffin-embedded tissue sections were tissue sections with a thickness of 5 μm.

[0019] Furthermore, in step S3, the specific steps for affine registration are as follows:

[0020] By performing transformation operations such as translation, rotation, scaling, and shearing, and by optimizing the objective function (maximizing mutual information), the spatial alignment between adjacent slices is gradually adjusted to ensure the consistency and coherence of the geometry of each slice in three-dimensional space.

[0021] Furthermore, in step S4, the two-dimensional digital pathological image contains cell types and tissue structures.

[0022] Furthermore, the cell types are one or more of eosinophils, lymphocytes, and neutrophils, and the tissue structures are one or more of epithelium, microvessels, and glands.

[0023] Furthermore, in step S5, the original three-dimensional data of cells and tissue regions are point cloud data of inflammatory cells and inflammatory regions encapsulated in PCD format.

[0024] Furthermore, point cloud data is a digital 3D representation of a physical object or space, consisting of millions of individual measurement points, each with x, y, and z coordinates.

[0025] Furthermore, in step S6, the three-dimensional model data of the nasal polyp tissue structure includes the outer contour of the specimen epithelium and the STL polygon data of glands and blood vessels, as well as the Niagara data of each cell location.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0027] 1. The method of the present invention performs three-dimensional reconstruction and visualization of nasal polyp tissue through steps such as continuous slicing, HE staining, three-dimensional reconstruction, and visualization rendering. It presents the three-dimensional structure of nasal polyp tissue, which not only enriches the morphological structure of nasal polyp tissue, but also provides morphological evidence for further exploration of the occurrence and development mechanism of nasal polyps in chronic sinusitis.

[0028] 2. The method of the present invention also has the following advantages:

[0029] (2) Microscopic 3D Reconstruction: A standard 5μm thick slice represents only 1 / 2000th of the complete tissue of a 1cm nasal polyp. 3D nasal polyp reconstruction technology can achieve precise 3D reconstruction of the microscopic tissue structures of nasal polyps at the single-cell level based on pathological slide data. This method provides a 3D view of the tissue structure at the microscopic level, further extracting information on the heterogeneity of nasal polyp tissue, and helping doctors and researchers to understand the histological characteristics of nasal polyps more deeply.

[0030] (2) High efficiency and accuracy in image processing: The Chronic Sinusitis with Nasal Polyps Subtype Assisted Interpretation System (NPSS-WSI) is used to identify continuous pathological slides. It can identify a large number of slides in a high throughput and generate corresponding two-dimensional pathological image data. It can also accurately calculate the information of inflammatory cells and tissue regions in the whole slide.

[0031] (3) Enhanced visualization capabilities: Through advanced graphics rendering technology, more intuitive visualization of tissue structures can be achieved, supporting more complex data analysis, such as volume measurement, morphological analysis, and guidance of clinical sampling.

[0032] (4) Improved diagnostic and research tools: Information on the heterogeneity of nasal polyp tissue extracted by 3D reconstruction can improve diagnostic accuracy, facilitate precise pathological subtyping, and guide individualized clinical treatment. In addition, it provides new insights into disease progression and treatment response. Attached Figure Description

[0033] Figure 1 Two-dimensional pathological images identified by a whole-slide subtype-assisted interpretation system for chronic sinusitis with nasal polyps;

[0034] Figure 2 The original cell data distribution map of the specimen, where neutrophils are used as an example: each point in the space represents an identified neutrophil;

[0035] Figure 3 To convert the original cell point cloud data into a cell example map by point substitution, the left side is a local point cloud and the right side is a sphere of the cell example;

[0036] Figure 4 To convert the original gland point cloud data into a polygon-constructed gland structure map, the left side shows the original gland point cloud, and the right side shows the final generated polygon-constructed gland structure;

[0037] Figure 5 This is a polygonal outline generated from the outer skin point cloud;

[0038] Figure 6 The cell rendering image is replaced with a cell example, where the color of the neutrophil example is added manually.

[0039] Figure 7 This is the final rendered image. Detailed Implementation

[0040] The present invention will now be further described in conjunction with specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0041] Unless otherwise specified, the experimental methods used in the following embodiments are conventional methods, performed according to the techniques or conditions described in the literature in this field or according to the product instructions. Unless otherwise specified, the materials, reagents, equipment, etc., used in the following embodiments are commercially available.

[0042] Example 1: Obtaining Nasal Polyp Tissue

[0043] In surgery for patients with chronic sinusitis and nasal polyps, after the nasal polyps (NPs) originating from the middle meatus are completely removed during functional endoscopic sinus surgery, they are immediately placed in 10% neutral formalin solution. Since a single nasal polyp is about 1-2 cm long, the middle meatus polyp tissue, along with structures such as the uncinate process and ethmoid bulla, is removed under nasal endoscopy to ensure the integrity of the nasal polyp tissue as much as possible. The nasal polyp tissue is placed in a sterile culture dish with PBS, and the uncinate process and ethmoid bulla are carefully removed from the proximal root of the nasal polyp using ophthalmic scissors. The surface of the nasal polyp tissue is cleaned of blood, fluid and secretions, and then immersed in 10% formalin solution to ensure that the nasal polyp tissue is completely soaked in formalin.

[0044] Example 2: Paraffin embedding of nasal polyp tissue

[0045] The fixed nasal polyp tissue was placed in an embedding cassette and operated in a fume hood to dehydrate and clear the tissue: 70% ethanol I, 1 h; 70% ethanol II, 1 h; 95% ethanol I, 1 h; 95% ethanol I, 1 h; anhydrous ethanol, 1 h; anhydrous ethanol, 1 h; xylene I, 30 min; xylene II, 30 min; xylene III, 30 min.

[0046] Preheat the paraffin wax and immerse the embedding cassettes sequentially: paraffin I, 1 hour; paraffin II, 1 hour; paraffin III, 1 hour. Then place the tissue into the paraffin-immersed mold, adjusting the tissue position to the center of the tissue cassette, and place the embedding cassette on top of the mold. Cool on ice until the paraffin wax is completely solidified, then slowly demold. Trim the paraffin wax and store at 4°C.

[0047] Example 3: Serial sections and HE staining

[0048] Using a microtome, the paraffin block was continuously cut into conventional paraffin-embedded tissue sections, each 5 μm thick, and numbered sequentially from the base to the bottom. After baking at 65°C for 1 hour, the tissue sections were dewaxed in the following sequence: xylene I, 5 min; xylene II, 5 min; xylene III, 5 min; anhydrous ethanol, 5 min; anhydrous ethanol, 5 min; 90% ethanol I, 5 min; 90% ethanol II, 5 min; 70% ethanol I, 5 min; 70% ethanol II, 5 min; rinsed with purified water, 5 min; rinsed twice with PBS, 5 min each time.

[0049] Hematoxylin staining of cell nuclei: stain with hematoxylin for 5 min; rinse with purified water for 1 min. Apply 1% hydrochloric acid-alcohol differentiation solution for 10 seconds; rinse with running water for 1 min; apply 0.2% ammonia solution for 10 min; rinse with running water for 1 min; observe the hematoxylin staining under a microscope. Eosin staining of cytoplasm: rinse with 95% ethanol for 5 min; stain the cytoplasm with eosin for 1 min; rinse with purified water for 5 min; apply anhydrous ethanol I for 3 min; anhydrous ethanol II for 3 min; anhydrous ethanol III for 3 min; apply xylene I for 5 min; xylene II for 5 min; xylene III for 5 min; observe the eosin staining of the cytoplasm under a microscope. Finally, after the paraffin sections have dried completely, mount them with neutral resin.

[0050] Example 4: Three-dimensional reconstruction

[0051] 1. Registration

[0052] Affine registration was performed on consecutive pathological sections using the SimpleITK library in Pyon software.

[0053] Specifically, the SimpleITK library in Python is used to perform affine registration on consecutive pathological slices. SimpleITK's affine registration algorithm includes geometric transformations such as translation, rotation, scaling, and shearing. By optimizing the objective function to maximize mutual information (MI), it progressively adjusts the spatial alignment between adjacent slices to ensure geometric consistency. Mutual information is used to measure the similarity between two images, and the objective function is defined as follows:

[0054]

[0055] Where A and B represent two adjacent slice images respectively, I(A,B) is the objective function of mutual information between two adjacent images, p(a) is the probability distribution of a, p(b) is the probability distribution of b, and p(a,b) is their joint probability distribution. By maximizing mutual information, the optimal geometric transformation parameters are found to maximize the information overlap between adjacent slices.

[0056] "Stepwise adjustment" refers to continuously optimizing the affine transformation matrix T using iterative optimization algorithms such as gradient descent, with the matrix in the form of Equation 2 below:

[0057]

[0058] Where T is the affine transformation matrix, t x For the x-axis translation of the slice, t y The y-axis translation of the slice is defined by a and b, which are scaling controls, and b and c are shearing controls.

[0059] During optimization, the affine transformation matrix T is adjusted to update the slice translation (t). x , t y The system employs control operations (scaling, rotation, and shearing, scaling, rotation, and scaling, scaling, rotation, scaling, scaling, rotation, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling, scaling) to gradually achieve optimal registration between adjacent slices. This process ensures the geometric consistency and progressively refined spatial alignment between slices.

[0060] 2. Artificial intelligence image reading

[0061] The registered serial slides were analyzed one by one using the Chronic Sinusitis with Nasal Polyps Subtype Auxiliary Interpretation System (NPSS-WSI).

[0062] As attached Figure 1 As shown, the system automatically identifies cell types and tissue structures in each two-dimensional pathological slide, including different cell types such as eosinophils, lymphocytes, and neutrophils, as well as tissue structures such as epithelium, microvessels, and glands. Each type of cell and tissue is distinguished by a unique color, generating continuous pathological slide images.

[0063] 3. Image fusion

[0064] These consecutive two-dimensional digital pathological images are stacked according to their registration order to obtain the raw point cloud data of inflammatory cells and inflammatory regions (microvessels, glands, epithelium, etc.) in the two-dimensional digital pathological images (where point cloud is a digital 3D representation of a physical object or space, consisting of millions of individual measurement points, each with x, y, and z coordinates).

[0065] As attached Figure 2 As shown, the original data is encapsulated into a PCD format point cloud, which consists of countless points in a virtual 3D space. Each point is labeled with two 3D vectors: one labeling its spatial coordinates (x, y, z) and the other labeling its assigned color information (RGB).

[0066] 4. Data Processing

[0067] First, a Python script is used to repackage the original PCD format into a PLY format point cloud, making it easier for Houdini software to process.

[0068] Then import the data into Houdini software for the following processing:

[0069] A. Cell distribution data: as attached Figure 3As shown, for data such as plasma cells, lymphocytes, and eosinophils, each point represents a cell. Using Houdini's point replacement function, each point is replaced with a cell example (a sphere used for a simplified representation of the cell model). Using Houdini's Niagara tool, the data for a single cell class is exported as a Niagara-formatted point cloud.

[0070] B. Vascular and glandular data: as attached Figure 4 As shown, a tool suite was built in Houdini. First, the points that make up the outline are connected to form lines. Then, the lines that are close enough are closed to form loops. Finally, the exploration distance tool attempts to connect the loops that are close to each other in the axial distance of the nearest slice, thereby generating the final continuous polygons and finally constructing the shape of blood vessels and glands.

[0071] C. Epithelial data: as attached Figure 5 As shown, the epithelial point cloud is connected to form polygons, generating an outer contour model. Data integration yields a three-dimensional model of the nasal polyp tissue structure, including: the outer contour of the specimen epithelium, glands, STL polygon data of blood vessels, and Niagara data for each cell location.

[0072] 5. Visual rendering

[0073] The 3D model data of the nasal polyp tissue structure was imported into the Unreal Engine (UE, or Unreal Engine, is currently the most powerful real-time visual rendering engine, widely used in film, games, and scientific research, and possesses powerful polygon and special effects rendering capabilities). STL format polygon data was imported into the UE pipeline, and lighting and materials were set. Cell point data was imported into the UE pipeline in Niagara format. The Niagara data generated in Houdini provides attributes such as cell size, color, position, and rotation orientation.

[0074] As attached Figure 6 As shown, the niagara system in UE integrates this data and uses sprite rendering to replace each point with a more detailed cell rendering map.

[0075] Finally, as attached. Figure 7 As shown, the rendering results of each data model are combined to obtain a three-dimensional model of the nasal polyp tissue structure. It can be seen from the figure that the three-dimensional model of the nasal polyp tissue structure of this invention can demonstrate the spatial distribution characteristics of inflammatory cells and tissue structures in the nasal polyp tissue.

[0076] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for stereoscopic reconstruction and visualization of nasal polyp tissue based on artificial intelligence recognition of consecutive pathological sections, characterized in that, Comprise the following steps: S1: Strictly continuous section of the reconstructed object; S2: According to the standard procedure, HE staining is performed on the continuous sections to obtain pathological sections; S3: Through the SimpleITK library in Pyon software, affine registration is performed on the continuous pathological sections, and then stereoscopic reconstruction is performed to obtain a preliminary stereoscopic model; S4: Using the chronic rhinosinusitis with nasal polyps whole slide subtype auxiliary interpretation system, the continuous pathological sections after registration are analyzed one by one to obtain continuous two-dimensional digital pathology images; The specific steps are: automatically identifying the cell types and tissue structures in each two-dimensional pathology section, including different cell types of eosinophils, lymphocytes, and neutrophils, as well as epithelial, microvessel, and glandular tissue structures, each type of cell and tissue is distinguished by a unique color, and a continuous pathology section image is generated; S5: Stack the continuous two-dimensional digital pathology images according to their registration order to obtain three-dimensional raw data of cell and tissue regions; S6: Process the three-dimensional raw data of cell and tissue regions to obtain three-dimensional model data of nasal polyp tissue structure; The specific steps are: first, use a python script to repackage the original PCD format into a PLY format point cloud, and then import the data into Houdini software for processing as follows: A. Cell distribution data: for plasma cells, lymphocytes, and eosinophils, use the point replacement function in Houdini to replace each point with a cell example, and use the Niagara tool in Houdini to export the data in the form of Niagara special effect point cloud; B. Vessel and gland data: a tool combination is built in Houdini, which first connects the points that make up the contour line into a line, then closes the line into a ring by connecting the lines that are close enough, and finally, the exploration distance tool is used to try to connect the rings that are close to each other in the axial direction of the nearest slice, thereby generating the final continuous polygon, and finally the shape of the blood vessels and glands is constructed; C. Epithelial data: connect the epithelial point cloud into a polygon to generate an outer contour model; Integrate the data to obtain three-dimensional model data of nasal polyp tissue structure, which includes: specimen epithelial contour, gland, and blood vessel STL polygon data, as well as Niagara data for each cell point; S7: Visualize and render the three-dimensional model data of nasal polyp tissue structure obtained in step S6, and combine the preliminary stereoscopic model obtained in step S3 to obtain the final three-dimensional model of nasal polyp tissue structure.

2. The method for recognizing continuous pathological sections based on artificial intelligence to perform three-dimensional reconstruction and visualization of nasal polyp tissue according to claim 1, characterized in that, In step S1, the reconstructed object is the completely removed nasal polyp tissue of a patient with chronic rhinosinusitis with nasal polyps after paraffin embedding during functional endoscopic sinus surgery.

3. The method for recognizing continuous pathological sections based on artificial intelligence to perform three-dimensional reconstruction and visualization of nasal polyp tissue according to claim 1, characterized in that, In step S1, the specific operation steps of strict continuous section are: use a microtome to continuously cut the wax block of the reconstructed object into conventional paraffin-embedded tissue sections, and number them sequentially from the root to the bottom.

4. The method for recognizing continuous pathological sections to reconstruct and visualize nasal polyp tissues in three dimensions based on artificial intelligence according to claim 3, characterized in that, The paraffin-embedded tissue sections are 5 μm thick tissue sections.

5. The method for recognizing continuous pathological sections based on artificial intelligence to perform three-dimensional reconstruction and visualization of nasal polyp tissue according to claim 1, characterized in that, In step S3, the specific operation steps of affine registration are: Through the transformation operation of translation, rotation, scaling and shearing, the spatial alignment between adjacent slices is gradually adjusted by optimizing the objective function, and the consistency and continuity of the geometric structure of each slice in the three-dimensional space are ensured.

6. The method for recognizing continuous pathological sections to perform three-dimensional reconstruction and visualization of nasal polyp tissue based on artificial intelligence according to claim 1, characterized in that, In step S5, the three-dimensional original data of the cell and the tissue region is point cloud data of the inflammatory cells and the inflammatory region packaged in PCD format.

7. The method for recognizing continuous pathological sections to reconstruct and visualize nasal polyp tissues in three dimensions according to claim 6, wherein, Point cloud data is a digital 3D representation of a physical object or space, consisting of millions of individual measurement points, each with x, y, and z coordinates.

Images