A method and system for mapping lesion information from ESD pathological sections to gross specimen images.
By extracting lesion information and vertical line sequences from pathological slides and gross specimen images, generating and correcting one-dimensional heat bars, the problem of inaccurate lesion information mapping in existing technologies is solved, realizing automated and accurate comparative diagnosis of pathological slides and gross specimens, and improving diagnostic efficiency and accuracy.
Patent Information
- Application Number
- CN202611103401.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-23
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies lack automated means to accurately map lesion information from ESD pathological slides to gross specimen images, resulting in time-consuming manual operations, low matching accuracy, and difficulty in achieving efficient and accurate comparative diagnosis between microscopic lesions and gross specimens.
By acquiring digital images of whole pathological slides and gross specimen images, the lesion area and vertical line sequence are extracted, one-dimensional heat bars are generated and corrected, and the vertical line sequence is matched according to the slide number to achieve automated and accurate mapping of lesion information.
It achieves efficient and accurate correspondence between pathological slide lesion information and gross specimen images, reduces manual operation costs, improves diagnostic efficiency and accuracy, and is compatible with routine pathological diagnosis workflows.
Smart Images

Figure CN123118459A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, and in particular to a method and system for mapping lesion information from ESD pathological sections to gross specimen images. Background Technology
[0002] Endoscopic submucosal dissection (ESD) is a mainstream minimally invasive treatment for early gastrointestinal mucosal and submucosal lesions. After the ESD specimen is removed from the body, a standardized process must be completed sequentially, including specimen fixation, segmented sampling, tissue sectioning, digital whole-slice scanning, and pathological diagnosis. In the pathological diagnosis stage, pathologists must, on the one hand, complete the core interpretation of lesion type, lesion grade, depth of invasion, and cutting margin status on the digital images of the whole-slice; on the other hand, they must establish a spatial correspondence between the lesion location identified in the whole-slice and the tissue bands on the gross specimen photographs, clarifying the slice number corresponding to each tissue band and the distribution range of the lesion along the tissue band, thereby completing the joint comparative diagnosis of microscopic images and gross specimens.
[0003] In routine clinical practice, ESD mucosal tissue needs to be cut into multiple strips and laid flat on a fixation plate with vertical grid lines to capture gross specimen images. Each tissue strip corresponds to a vertical grid line intersecting the tissue on the fixation plate, and the laboratory uniformly agrees to match the slide number with the vertical line number one by one. When the specimen tissue length is large and the number of segments exceeds the total number of effective vertical grid lines on the fixation plate, it is also necessary to rely on reverse backfilling rules to match the corresponding vertical lines. However, the existing digital pathology workflow lacks automated mapping methods. All strip, slide, and lesion location comparison work relies on physicians manually comparing the vertical line numbers. In scenarios with a large number of specimen segments and complex grid interference, the operation is time-consuming, highly dependent on the physician's personal experience, and prone to problems such as tissue strip matching errors and omission of high-risk lesion areas.
[0004] Currently, in the field of digital pathology, lesion areas and classification information of ESD pathological slides can be obtained through various conventional methods. The obtained lesion information can completely characterize the location and grade of lesions within the slide. However, the visualization of lesion information in existing technologies is limited to the digital image of the whole pathological slide, lacking automated processing logic to adapt and display two-dimensional lesion information in conjunction with the gross morphology of the specimen. Specifically, existing technologies cannot compress and project the two-dimensional lesion area of the slide along the tissue extension direction, making it difficult to generate one-dimensional thermal visualization results that fit the morphology of the gross tissue strips and carry lesion grading indicators, and failing to intuitively reflect the extension and distribution characteristics of the lesion along the tissue strips.
[0005] Meanwhile, existing technologies lack orientation correction mechanisms for gross specimen images and pathological slide images. The tissue extension directions of the two images are prone to misalignment, and direct superposition of lesion information can easily lead to spatial matching errors. Furthermore, existing technologies have not established an automatic matching and superposition system based on slide numbers and vertical line sequences in gross images. This makes it impossible to accurately map the corrected lesion thermal information to the corresponding tissue strip regions in the gross specimen images. Matching and annotation can only be completed manually by referring to the numbers, resulting in low matching accuracy, poor repeatability, and high labor costs. This severely restricts the efficiency of comparative diagnosis between microscopic lesions under the ESD specimen and macroscopic tissues.
[0006] In summary, the industry currently lacks a complete mapping method that is compatible with ESD specimens and can automatically complete lesion information extraction, heat bar generation, orientation correction, number matching, and gross image overlay. This makes it difficult to efficiently and accurately achieve automatic spatial correspondence between full-slice lesion information and gross specimen images, greatly increasing the diagnostic workload of pathologists. Summary of the Invention
[0007] To address the problems existing in the prior art, the present invention provides a method for mapping lesion information from ESD pathological sections to gross specimen images, comprising: Step S1: Obtain the digital image of the whole pathological section corresponding to the ESD specimen and the gross specimen image obtained by fixing the ESD specimen on the fixation plate. Step S2: Obtain the lesion area and lesion classification information from the digital image of the whole pathological section, and extract the vertical line sequence that intersects with the tissue area corresponding to the ESD specimen from the gross specimen image; Step S3: Compress and project the lesion area along the tissue extension direction to generate a one-dimensional heat bar carrying a lesion grading label representing the lesion classification information; Step S4: Perform positive and negative correction on the one-dimensional heat strip and output the corrected one-dimensional heat strip that is consistent with the tissue extension direction in the gross specimen image. Step S5: Match the slice number of the digital image of the whole pathological slice corresponding to the corrected one-dimensional heat bar to obtain the vertical line number in the vertical line sequence, and superimpose the corrected one-dimensional heat bar onto the tissue area where the vertical line number is located in the gross specimen image to obtain a gross control image mapped with lesion information.
[0008] Preferably, obtaining lesion area and lesion classification information from the digital image of the whole pathological section includes: Check if a manually annotated file exists corresponding to the digital image of the pathological whole slide; If the manually labeled file is found, the lesion area and lesion classification information are obtained from the manually labeled file, and invalid labeled data are removed; If the manually annotated file is not found, the digital image of the whole pathological slide is processed based on an artificial intelligence model to obtain the lesion area and the lesion classification information.
[0009] Preferably, the step of extracting the sequence of vertical lines intersecting the tissue region corresponding to the ESD specimen from the gross specimen image includes: Boundary detection is performed on the fixation plate in the gross specimen image to crop out the effective image area of the fixation plate; Identify the vertical grid lines within the effective image area of the fixed plate; The tissue region mask corresponding to the ESD specimen is generated by segmenting the effective image area of the fixed plate, and the vertical grid lines are intersected with the tissue region mask to obtain all vertical lines that intersect with the tissue region mask. The vertical lines are then sorted in spatial order from left to right to form the vertical line sequence.
[0010] Preferably, identifying the vertical grid lines within the effective image area of the fixed plate includes: A vertical dark line mask is generated within the effective image area of the fixed plate, and a horizontal morphological opening operation is performed on the vertical dark line mask to remove wide patch-like interference. The vertical dark line mask is divided into multiple independent vertical columns along the horizontal direction. The column support and the longest continuous support ratio of each vertical column are calculated respectively. The vertical column whose column support and the longest continuous support ratio both meet the preset conditions are marked as valid line peaks. Within the effective image area of the fixed plate, a sampling area is defined at the top of the fixed plate, and invalid peaks corresponding to the printed red reference line are removed within the sampling area at the top of the fixed plate according to the color hue and saturation parameters. The median standard spacing of the lines is calculated based on all valid line peaks. A minimum spacing threshold is set based on the median standard spacing of the lines, and clustered pseudo-peaks with spacing smaller than the minimum spacing threshold are filtered out. When the distance between two adjacent vertical lines is greater than a set multiple of the median standard distance between the lines, missing vertical lines are generated at the equal division positions of the adjacent vertical lines, and the effectiveness of the supplemented vertical lines is verified by combining the column support of the local column.
[0011] Preferably, step S3 includes: At a preset reading magnification, each of the lesion classification information is rendered as an independent two-dimensional lesion mask; For each vertical column of the two-dimensional lesion mask, the maximum value of all pixels in that column is calculated along the row direction to complete the compressed projection of the two-dimensional image to one dimension. When multiple lesion classification information exist in the same vertical column, the one-dimensional heat bar is generated by rendering in layers according to the preset coloring priority of the lesion severity. Different colors obtained by layer rendering on the one-dimensional heat bar correspond to different lesion classification identifiers.
[0012] Preferably, step S4 includes: Marking detection areas are delineated at the four corners of the digital image of the whole pathological slide and the gross specimen image, and positioning marks are identified in each of the marking detection areas, and the end orientation corresponding to each positioning mark is identified. By comparing the end orientations of the same type of positioning marker regions in the digital image of the whole pathological slide and the gross specimen image, if the end orientations are consistent, the one-dimensional thermal strip is used as the corrected one-dimensional thermal strip; if the end orientations are inconsistent, the one-dimensional thermal strip is flipped to obtain the corrected one-dimensional thermal strip.
[0013] Preferably, identifying the end position corresponding to the positioning mark includes: For each of the marker detection regions, the number of pixels in the region that conform to the color range of the first type of positioning marker and the color range of the second type of positioning marker are counted, and the pixel ratio of the number of pixels of the two types of colors in the corresponding marker detection region is calculated. The marker type corresponding to the color with a higher pixel ratio is determined as the marker type of the positioning marker carried by the corresponding marker detection area, and the end orientation is determined based on the marker type.
[0014] Preferably, in step S5, the process of matching the vertical line numbers includes: When the slice number is less than or equal to the total number of vertical lines in the vertical line sequence, the vertical line number in the vertical line sequence that matches the slice number is used as the matched vertical line number. When the slice number is greater than the total number of vertical lines, the vertical line number is calculated using a reverse backfill method. The calculation formula for the reverse backfill method is as follows:
[0015] in, is the vertical line number, n is the total number of vertical lines, and N is the slice number.
[0016] Preferably, in step S5, when the corrected one-dimensional heat strip is superimposed onto the tissue region where the vertical line number is located, a rounded rectangular fusion mask is used to superimpose it onto the tissue region, and when the same vertical line number corresponds to multiple corrected one-dimensional heat strips, each corrected one-dimensional heat strip is horizontally offset and parallel.
[0017] This invention also provides a system for mapping lesion information from ESD pathological sections to gross specimen images, using the above-described method. The system includes: The image acquisition module is used to acquire digital images of the pathological whole slides corresponding to the ESD specimen and gross specimen images taken by fixing the ESD specimen on the fixation plate. The image analysis module, connected to the image acquisition module, is used to acquire lesion area and lesion classification information from the digital image of the whole pathological slide, and to extract the vertical line sequence intersecting with the tissue area corresponding to the ESD specimen from the gross specimen image; A heat bar generation module, connected to the image analysis module, is used to compress and project the lesion area along the tissue extension direction to generate a one-dimensional heat bar carrying a lesion grading label that represents the lesion classification information. The orientation alignment module is connected to the image analysis module and the heat bar generation module respectively. It performs positive and negative correction on the one-dimensional heat bar and outputs a corrected one-dimensional heat bar that is consistent with the tissue extension direction in the gross specimen image. The mapping and fusion module, which is connected to the image analysis module and the orientation alignment module respectively, is used to match the slice number of the pathological whole slice digital image corresponding to the corrected one-dimensional heat bar to obtain the corresponding vertical line number in the vertical line sequence, and to superimpose the corrected one-dimensional heat bar onto the tissue area where the vertical line number is located in the gross specimen image to obtain a gross control image mapped with lesion information.
[0018] The above technical solution has the following advantages or beneficial effects: 1) Through a complete and closed-loop automated process of image acquisition, lesion information extraction, gross image analysis, heat bar generation, orientation correction and precise overlay mapping, it effectively makes up for the technical deficiency of relying on manual operation in the spatial correspondence between microscopic lesions and gross specimens in the existing ESD pathological diagnosis. 2) This invention simultaneously acquires digital images of whole pathological slides and images of the macroscopic specimen on the fixation plate, and extracts the lesion area and lesion classification information on the slide side and the vertical line sequence intersecting with the tissue on the macroscopic image side, respectively. This establishes a two-way data association foundation between the microscopic pathological images and the macroscopic images, which overcomes the limitation of traditional technology where lesion information is only stuck in the coordinate system of the whole slide image and cannot be connected to the macroscopic specimen space, and provides reliable data support for subsequent automated and accurate mapping. 3) This invention compresses and projects the two-dimensional lesion area along the tissue extension direction, and generates a one-dimensional heat bar with lesion grading label by combining lesion classification information. It transforms the two-dimensional lesion distribution characteristics of the slice plane into a one-dimensional visualization carrier that adapts to the long strip-shaped tissue morphology of the gross specimen. This conforms to the clinical diagnostic habit of interpreting the extension distribution of lesions along tissue strips, and realizes the adaptive visualization of lesion information. It solves the problems of mismatch between the traditional two-dimensional lesion display form and the gross tissue morphology and poor intuitiveness. 4) This invention performs positive and negative correction on the generated one-dimensional heat strips to ensure that the extension direction of the heat strips is completely consistent with the extension direction of the tissue in the gross specimen image. This effectively avoids the problem of tissue misalignment caused by differences in slicing and image shooting angles, completely eliminates the direction matching deviation in the process of superimposing lesion information, and greatly improves the accuracy of lesion spatial mapping. 5) This invention relies on the correspondence between slice numbers and vertical line sequences to automatically match precise vertical line number positions and complete heat bar overlay. This achieves automated and precise spatial mapping of pathological slice lesion information to gross specimen images. It eliminates the need for pathologists to manually compare and label grid vertical line numbers, significantly reducing manual operation costs and experience dependence. It effectively avoids matching errors and lesion omissions caused by manual comparison, significantly improving the efficiency and accuracy of combined comparison diagnosis of ESD specimen microscopic lesions and gross specimens. It can be directly used for clinical specimen quality control and diagnostic image output, and is compatible with routine pathological diagnosis workflows. Attached Figure Description
[0019] Figure 1 A flowchart illustrating a method for mapping lesion information from ESD pathological sections to gross specimen images, as a preferred embodiment of the present invention. Figure 2 In a preferred embodiment of the present invention, an attention heatmap obtained by processing an artificial intelligence model is used; Figure 3 A schematic diagram of polygonal annotation of the lesion area in a preferred embodiment of the present invention; Figure 4 This is a schematic diagram showing the location of the marker detection area in a preferred embodiment of the present invention; Figure 5 In a preferred embodiment of the present invention, a schematic diagram of a general comparative image mapping all lesion information is shown. Figure 6 In a preferred embodiment of the present invention, a schematic diagram of a system for mapping lesion information from ESD pathological sections to gross specimen images is provided. Detailed Implementation
[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The present invention is not limited to this embodiment; other embodiments that conform to the spirit of the present invention may also fall within the scope of the present invention.
[0021] In a preferred embodiment of the present invention, existing ESD pathological diagnosis processes suffer from numerous problems, such as the inability to automatically map microscopic lesion information from whole pathological slides to macroscopic specimen images on fixation plates, reliance on manual comparison of band numbering leading to errors, mismatch between the two-dimensional lesion visualization format and the requirements for interpreting long strip-shaped tissues, and the potential for tissue misalignment due to differences in image shooting angles. The present invention provides a method for mapping lesion information from ESD pathological slides to macroscopic specimen images, such as... Figure 1 As shown, it includes: Step S1: Obtain the digital image of the whole pathological section corresponding to the ESD specimen and the gross specimen image obtained by fixing the ESD specimen on the fixation plate. Step S2: Obtain the lesion area and lesion classification information from the digital image of the whole pathological section, and extract the vertical line sequence that intersects with the tissue area corresponding to the ESD specimen from the gross specimen image; Step S3: Compress and project the lesion area along the tissue extension direction to generate a one-dimensional heat bar carrying a lesion classification label representing lesion classification information; Step S4: Perform positive and negative correction on the one-dimensional heat strip and output the corrected one-dimensional heat strip that is consistent with the tissue extension direction in the gross specimen image. Step S5: Match the slice number of the pathological whole slice digital image corresponding to the corrected one-dimensional heat bar to obtain the corresponding vertical line number in the vertical line sequence, and superimpose the corrected one-dimensional heat bar onto the tissue area where the vertical line number is located in the gross specimen image to obtain a gross control image mapped with lesion information.
[0022] Specifically, in this embodiment, the entire method consists of five sequentially executed steps. First, step S1 is executed, simultaneously acquiring two types of core images: reading the digital image of the whole pathological section corresponding to the ESD specimen to be tested, and simultaneously reading the gross specimen image obtained by photographing the ESD specimen after it has been fixed in vitro and laid flat on a special fixation plate. The whole pathological section digital image provides a detailed spatial framework of the lesion under the microscope, while the gross specimen image provides band numbers and a spatial reference frame. These two types of images are the basic input materials for subsequent lesion analysis, mesh line extraction, heat bar generation, and spatial mapping operations. In clinical scenarios involving batch processing of multiple tissue bands corresponding to multiple sections, the same gross specimen image only needs to be read once to match multiple whole pathological section digital images for cyclical use, effectively reducing the computational cost of repeated readings.
[0023] After image acquisition, the process proceeds to step S2, which performs two core data parsing tasks in parallel. Firstly, it extracts the complete lesion region outline and corresponding lesion classification information from the digital image of the whole pathological slide. Secondly, it processes the gross specimen image, selecting vertical grid lines that intersect with the ESD specimen tissue region and arranging them in an orderly sequence. These two sub-processes are independent of each other, but subsequently serve the heatstrip generation and spatial matching stages.
[0024] In a preferred embodiment of the present invention, obtaining lesion area and lesion classification information from digital images of whole pathological sections includes: Check if there is a manually annotated file for the corresponding whole pathology slide digital image; If a manually annotated file is found, the lesion area and lesion classification information are parsed from the manually annotated file, and invalid annotated data is removed; If no manually annotated files are found, the digital images of whole pathological slides are processed based on an artificial intelligence model to obtain information on lesion areas and lesion classification.
[0025] Specifically, when extracting lesion regions and lesion classification information from digital images of whole pathology slides, the system first checks whether a manually annotated structured file exists for the current whole pathology slide image. These manually annotated files are exported by pathologists after manually drawing lesion polygons and entering lesion hierarchical classifications in the whole slide browsing interface. The format is mostly JSON-like structured data.
[0026] If a valid manually annotated file is found, the polygon vertices and lesion grading labels in the manually annotated file are read directly. At the same time, invalid annotated data marked as deleted or questionable are automatically removed. There is no need to call the AI model for inference, saving hardware computing power.
[0027] If no corresponding manually annotated file is found, the artificial intelligence model will be automatically activated to extract lesion information. The processing steps of the artificial intelligence model include: First, the foreground region of the tissue is screened from the digital image of the whole pathological slide. Then, the tissue region is divided into fixed-size image blocks. Features of these image blocks are extracted using the UNI pre-trained visual model, which is specifically designed for pathology. The feature sequences are then input into the TransMIL attention-based multi-instance learning network, which uses a Transformer architecture. The output is shown below. Figure 2 The overall lesion classification results of the slices shown are compared with the image block-level attention heatmap. Then, binarization, morphological filtering, and contour extraction are performed on the heatmap to transform it into the following: Figure 3 The lesion polygon regions shown are uniformly readable, allowing pathologists to verify whether the AI-generated lesion regions are reasonable. Figure 2 and Figure 3 It explains where the lesion is under the microscope, but it has not yet entered the gross coordinate system.
[0028] Whether it's the manual annotation pathway or the AI inference pathway, the data format of the final output lesion area and lesion classification information is completely unified. They can share the entire set of processing logic for subsequent heat bar generation, direction correction, and mapping overlay, without problems such as data format incompatibility or mapping pipeline fragmentation, which greatly reduces the maintenance costs of system integration and batch processing.
[0029] In the concurrent gross specimen image analysis process, in a preferred embodiment of the present invention, extracting the sequence of vertical lines intersecting the tissue region corresponding to the ESD specimen from the gross specimen image includes: Boundary detection is performed on the fixation plate in the gross specimen image to crop out the effective image area of the fixation plate; Identify the vertical grid lines within the effective image area of the fixed plate; The tissue region mask corresponding to the ESD specimen is generated by segmenting the effective image area of the fixed plate. The vertical grid lines are then intersected with the tissue region mask to obtain all vertical lines that intersect with the tissue region mask. These vertical lines are then sorted in spatial order from left to right to form a vertical line sequence.
[0030] Specifically, in this embodiment, boundary detection is first performed on the fixed plate in the macro specimen image, and the effective image area of the fixed plate without external blank background is cropped based on the image grayscale and color channel threshold constraints, thereby eliminating the interference of irrelevant background on grid line recognition.
[0031] After the effective area is cropped, the identification of all vertical grid lines within the effective image area of the fixed plate begins. In a preferred embodiment of the present invention, the identification of vertical grid lines within the effective image area of the fixed plate includes: A vertical dark line mask is generated within the effective image area of the fixed plate, and a horizontal morphological opening operation is performed on the vertical dark line mask to remove wide patch-like interference. The vertical dark line mask is divided into multiple independent vertical columns along the horizontal direction. The column support and the longest continuous support ratio of each vertical column are calculated. The vertical columns that meet the preset conditions in both column support and the longest continuous support ratio are marked as valid line peaks. Within the effective image area of the fixed plate, a sampling area is defined at the top of the fixed plate, and invalid peaks corresponding to the printed red reference line are removed within the sampling area at the top of the fixed plate based on the color hue and saturation parameters. The median standard spacing of the lines is calculated based on all valid line peaks. A minimum spacing threshold is set based on the median standard spacing of the lines, and clustered pseudo-peaks with spacing less than the minimum spacing threshold are filtered out. When the distance between two adjacent vertical lines is greater than a set multiple of the standard distance between the median of the lines, missing vertical lines are generated at the equal division positions of the adjacent vertical lines, and the effectiveness of the supplemented vertical lines is verified by combining the column support of the local column.
[0032] The process involves performing a horizontal morphological opening operation on the vertical dark line mask to filter out wide-area patchy interference pixels caused by dirt and reflections within the image. After processing, the entire vertical dark line mask is divided into independent vertical columns. For each column, two quantitative indicators are calculated: column support and the longest continuous support percentage. Only vertical columns that simultaneously meet preset threshold conditions are retained, and their horizontal coordinates are marked as valid line peaks, initially filtering out invalid lines formed by scattered noise. Next, a dedicated sampling area is defined above the effective image area of the fixed plate. Invalid peaks corresponding to the printed red reference line are distinguished and removed based on the pixel HSV hue and saturation parameters to avoid interference from the red baseline line in identifying the black grid vertical lines. Based on all the retained valid line peaks, the spacing between all adjacent lines is calculated to obtain the median standard spacing. Then, a minimum spacing threshold is set based on this median spacing, and densely stacked pseudo-peaks with spacing less than this threshold are batch-removed. Finally, all adjacent vertical lines are traversed. When the distance between two lines is greater than the standard distance between the median of the lines by a set multiple, a supplementary missing vertical line is automatically generated at the equidistant position between the two lines. The column support value of the local vertical column at the corresponding position is extracted to verify whether the supplementary vertical line actually has grid features. This solves the problem of incomplete recognition caused by local grid wear and missing lines on the fixed plate.
[0033] The entire grid line recognition process described above takes very little time in actual batch processing of cases. Experimental results show that in batch processing of cases, the bottom plate boundary detection takes about 0.51 seconds, and the black grid vertical line detection (including wide patch removal, red line filtering, and spacing filling) takes about 1.22 seconds. The two combined account for less than 2% of the entire pipeline's 173.3 seconds, balancing recognition robustness and processing efficiency.
[0034] After extracting the vertical grid lines, a segmentation model is preferably used to segment the effective image area of the fixed plate into pixels, generating a tissue region mask that completely covers the ESD specimen tissue. The intersection of all the previously identified vertical grid lines with the tissue region mask is then performed, retaining only the vertical lines whose pixels overlap with the tissue region. These lines are then sorted according to their spatial coordinates from left to right in the image, forming an ordered sequence of vertical lines. This sequence serves as the unique reference set for subsequent slice number matching and precise thermal strip overlay.
[0035] After obtaining the lesion information and ordered vertical line sequence in a unified format in step S2, step S3 is executed to generate a one-dimensional heat bar. This step realizes the transformation of the two-dimensional lesion distribution into a one-dimensional visualization carrier adapted to the long strip tissue morphology. During the operation, a unified image reading magnification is set, and different lesion classifications are rendered as independent two-dimensional lesion masks. For each two-dimensional lesion mask, all pixels are traversed from top to bottom along each column in the vertical direction, and the maximum value is calculated for all pixels in the same column to complete the compressed projection of the two-dimensional planar lesion area to the one-dimensional vertical distribution data. When multiple different lesion classification masks are superimposed at the same vertical coordinate position, a fixed coloring priority is preset according to the clinically defined lesion severity. The priority is from low to high as low-grade intraepithelial neoplasia, high-grade intraepithelial neoplasia, and adenocarcinoma-like lesions. The color of high-grade lesions will cover the color of low-grade lesions in layer rendering, and finally a complete one-dimensional heat bar is generated. Different colors on the one-dimensional heat bar correspond to different lesion grading indicators, intuitively reflecting the distribution range and severity of the lesion along the long axis of the tissue strip. This solves the problem that traditional two-dimensional lesion graphics cannot intuitively adapt to the reading habits of large tissue strips. The heat bar generation operation for a single tissue strip takes only a few seconds and can support automated batch processing at the case level.
[0036] After generating the one-dimensional heat strip, step S4 is performed to perform forward and reverse calibration on the heat strip to eliminate the misalignment of tissue extension direction caused by the different shooting angles of pathological sections and gross specimens. This calibration process includes: Marker detection areas were delineated at the four corners of the digital images of whole pathological slides and gross specimen images, and localization markers were identified within each marker detection area; For each marker detection area, count the number of pixels in the area that match the color range of the first type of positioning marker and the color range of the second type of positioning marker, and calculate the percentage of pixels of each type of color in the corresponding marker detection area. The marker type corresponding to the color with a higher pixel ratio is determined as the marker type of the positioning marker carried by the corresponding marker detection area, and the end orientation is determined based on the marker type; By comparing the end orientations of the same type of positioning marker regions in the digital image of the whole pathological slide and the gross specimen image, if the end orientations are consistent, the one-dimensional thermal strip is used as the corrected one-dimensional thermal strip; if the end orientations are inconsistent, the one-dimensional thermal strip is flipped to obtain the corrected one-dimensional thermal strip.
[0037] Specifically, in this embodiment, four blocks are delineated at the four corners of the digital image of the whole pathological slide and the gross specimen image, as shown below. Figure 4The independent marker detection areas shown represent four marker detection areas formed by the two image types. Within each marker detection area of the fixed plate, only a single type of positioning marker is placed; the remaining pixels are white background pixels not included in the statistics. The positioning markers are divided into black and blue categories, corresponding to two different color ranges.
[0038] In each marker detection area, marker pixels are identified, and the image end orientation corresponding to the two types of positioning markers is further distinguished. Specifically, when identifying the end orientation, for any marker detection area, the total number of pixels matching the black marker color range and the blue marker color range in the area are counted respectively. The pixel ratio of the two types of color pixels in this area is calculated respectively. The two color ratio values of the two types of colors in the same detection area are compared. The marker type corresponding to the color with a significantly higher ratio is determined as the exclusive positioning marker of this end. Combining the marker types determined by the two diagonal detection areas of a single image, a one-to-one correspondence between the marker type and the spatial position of the diagonal end of the image is established, and the end orientation reference corresponding to the two types of markers in a single image is completely obtained.
[0039] After extracting the azimuth reference of the two images, the azimuth of the same type of positioning marker in the whole pathological slide digital image and the gross specimen image are compared. If the distribution orientations of the two are completely consistent, the current one-dimensional heat bar is directly used as the corrected one-dimensional heat bar. If the azimuth of the two are opposite, the one-dimensional heat bar is flipped and corrected as a whole. Finally, the corrected one-dimensional heat bar that perfectly matches the tissue extension direction in the gross specimen image is output.
[0040] This correction step can completely avoid the problems of inverted beginnings and ends and disordered spatial matching of lesion distribution after the thermal strips are superimposed, and greatly improve the spatial accuracy of the mapping results.
[0041] After obtaining the corrected one-dimensional thermal bar, step S5 is executed. Based on the slide number and vertical line sequence, precise matching is performed and the images are overlaid to generate the final control image. Specifically, when matching the vertical line number, the slide number N corresponding to the current corrected one-dimensional thermal bar is first read from the digital image of the corresponding whole pathological slide. Simultaneously, the total number n of vertical line sequences obtained in step S2 is retrieved, and two sets of matching rules are set to adapt to different clinical placement scenarios: When the slice number N is less than or equal to the total number of vertical lines n, the vertical line with the same number as N in the vertical line sequence is directly selected as the target overlay position. When the number of tissue bands in the specimen exceeds the effective grid size of the fixation plate, that is, when the slice number N is greater than the total number of vertical lines n, the following reverse backfill formula is used:
[0042] in, is the vertical line number, n is the total number of vertical lines, and N is the slice number.
[0043] The target vertical line number is calculated to conform to the industry practice of placing excess strips in reverse in clinical practice, thus avoiding situations where slices do not have corresponding grids or lesions cannot be mapped.
[0044] After determining the target vertical line number, the corrected one-dimensional heat bars are superimposed using a rounded rectangular fusion mask, limiting the overlay range. They are only drawn within the tissue region corresponding to that vertical line in the gross specimen image to prevent the heat bars from covering the blank background grid. If multiple corrected one-dimensional heat bars are generated from multiple pathological sections corresponding to the same vertical line number, the multiple heat bars will be slightly offset laterally and arranged side-by-side to avoid mutual obstruction affecting observation.
[0045] Understandably, heat bars from multiple pathological sections in the same case can be sequentially accumulated and superimposed onto the same gross specimen image, ultimately outputting a gross control image that fully maps all lesion information. The superimposed effect is as follows: Figure 5 As shown, the entire matching and overlay process eliminates the need for pathologists to manually number each mesh. Experimental results demonstrate that the complete mapping of nine tissue bands takes only about seventy seconds, significantly reducing the manual time required for microscopic and gross specimen comparison. The generated mapped gross comparison image can be directly used as an appendix in pathology reports and for specimen quality control archiving. Furthermore, the entire process fully preserves the substrate boundaries, mesh filler lines, orientation markers, and multi-layer intermediate visualization results rendered with heatstrips, forming a complete and traceable chain of evidence. This facilitates ESD specimen quality control review in hospitals and has significant clinical application value.
[0046] This invention also provides a system for mapping lesion information from ESD pathological sections to gross specimen images, using the methods described above, such as... Figure 6 As shown, the system includes: Image acquisition module 1 is used to acquire digital images of the pathological whole slides corresponding to the ESD specimen and gross specimen images obtained by taking pictures of the ESD specimen fixed on the fixation plate. Image analysis module 2, connected to image acquisition module 1, is used to acquire lesion area and lesion classification information from digital images of whole pathological sections, and extract the vertical line sequence intersecting with the tissue area corresponding to the ESD specimen from the gross specimen image; The heat bar generation module 3 is connected to the image analysis module 2 and is used to compress and project the lesion area along the tissue extension direction to generate a one-dimensional heat bar carrying a lesion classification label that represents the lesion classification information. The orientation alignment module 4 is connected to the image analysis module 2 and the heat bar generation module 3 respectively. It performs positive and negative correction on the one-dimensional heat bar and outputs a corrected one-dimensional heat bar that is consistent with the tissue extension direction in the gross specimen image. The mapping and fusion module 5 is connected to the image analysis module 2 and the orientation alignment module 4 respectively. It is used to match the slice number of the pathological whole slice digital image corresponding to the corrected one-dimensional heat bar to obtain the corresponding vertical line number in the vertical line sequence, and then superimpose the corrected one-dimensional heat bar onto the tissue area where the vertical line number is located in the gross specimen image to obtain a gross control image with lesion information.
[0047] Example 1 In this embodiment, a complete batch processing test was conducted using a real clinical case. The ESD specimen from this case consisted of 9 tissue strips and 1 macroscopic specimen image taken from a fixation plate. The clinical requirement was to automatically map the distribution of lesions along the long axis of the tissue under the microscope to the corresponding vertical strip positions on the macroscopic image, generating mapped macroscopic specimen images that could be used for reporting and quality control.
[0048] Data preparation: Gross specimen images: JPEG images taken with a fixation plate, including black grid vertical lines, center reference lines, black / blue directional markers, and tissue bands.
[0049] Full-slice digital images: 9 WSI images, scan magnification typically 20x, file number N corresponds one-to-one with strip number (N=1,…,9).
[0050] Lesion information carrier: For each WSI, artificial intelligence inference is pre-run to output a structured file N_ai_result.json (N=1,…,9), which contains bag-level classification (such as group3 / group4 / adenocarcinoma), polygon vertices obtained from attention heatmap post-processing, and fields such as heatmap_scale=2.5; the file format is compatible with the physician annotation file N.json, and can be read uniformly by the mapping subsystem.
[0051] Batch processing configuration: The case_batch_run.yaml file configures 9 sets of {wsi_path, json_path, line_id} pairs, as well as parameters such as the overall graph path and GPU device.
[0052] Processing flow: Step 1: Obtain Input Images: Read the macroscopic.jpg file as the macroscopic specimen image; load 1.kfb to 9.kfb files sequentially as full-slice digital images according to the configuration. This step only completes I / O and cannot be interchanged with S3 (the input path must be determined first).
[0053] Step 2: Obtain lesion area and lesion classification information, and use an artificial intelligence model for reasoning. The process is as follows: Perform the following sub-steps for each WSI (if N_ai_result.json already exists and --skip-infer is set, skip this sub-step and proceed directly to the next step).
[0054] Foreground extraction and image patch segmentation: Non-overlapping patch sampling is performed within the WSI tissue region at an equivalent magnification of approximately 20 times and a resolution of 224×224 pixels; Feature extraction: Load the pre-trained pathological basic model UNI (uni_v1, ViT-Large architecture), extract feature vectors for each patch, and freeze the model parameters; Bag-level classification and attention heatmap: The patch feature sequence is input into the TransMIL (transformermil) multi-instance learning network, which outputs slice-level lesion classification and attention weights, and maps them to a heatmap aligned with the reading magnification of 2.5 times; Post-processing of heat maps: [This refers to the processing of heat maps.] Figure 2 Value-based processing (threshold typically 153, adjustable to 120 for batch processing), first close operation then open operation, filter connected components and approximate polygons, coordinate conversion to WSI scan coordinate system; if classified as negative, the lesion polygon is not written, and the overall image remains unchanged during the mapping stage.
[0055] Step 3, Gross Specimen Image Analysis: This step is performed only once on the gross specimen.jpg file (all 9 bands share the same analysis result): Base plate boundary detection: Cutting the ROI of the fixed plate under grayscale / V channel threshold constraints; Black grid vertical line detection: Vertical dark line masking → Horizontal opening operation wide patch removal → Column support peak filtering → Top HSV filtering printed red line → By median spacing Eliminate short-spacing false peaks → When the spacing between adjacent vertical lines Replace missing vertical lines; Tissue mask segmentation: The SAM2.1-hiera-large segmentation model is used to generate a binary mask of the gross tissue. The intersection is calculated to obtain the sequence of vertical lines intersecting the tissue (numbered from left to right, n lines in total).
[0056] In this embodiment, the image resolution of the gross specimen took approximately 1.73 seconds (0.51 seconds for the base plate + 1.22 seconds for the vertical line), which is less than 2% of the total pipeline time of 173.3 seconds.
[0057] Step 4: Generate one-dimensional color heatstrips using column projection: At a magnification of 2.5x, render each lesion category in N_ai_result.json as a multi-layer binary mask. For the first Take the maximum value in the column direction:
[0058] When there are multiple categories in the same column, the coloring is based on priority: low-grade intraepithelial neoplasia (group3, yellow) < high-grade intraepithelial neoplasia (group4, red) < adenocarcinoma (purple).
[0059] Step 5: Align the direction of the tissue strips: Cut tissue strips from each WSI; check the black / blue HSV markers in the corner area of the macrograph and check the strips. If the blue end of the strip is inconsistent with the macrograph, flip the strip.
[0060] Step 6, Target Vertical Line Selection and Blending Output: Using the specimen_index mode: Let the number of available vertical lines intersecting with the organization be . Full slice logical sequence number Inferred from the filename. When Select the first Intersecting vertical lines; when And when line_overflow_reverse_fill is enabled, the physical index is:
[0061] Aligned one-dimensional color heatmaps are superimposed onto the tissue region along the target vertical line using a rounded rectangular fusion mask (corner_radius=16, alpha=0.6). Steps 4-6 are performed sequentially on the nine bands and cumulatively superimposed onto the same gross image, outputting batch_all_on_gross.jpg.
[0062] Runtime configuration and time consumption: GPU: NVIDIA RTX A4000; Number of stripes: 9; Pipeline completion time: 173.3 s; Total of 9 stripe mappings (pair.total): 69.0 s; Single strip mapping time: approximately 3.0–10.6 s (strip_9–strip_7); Output files: batch_all_on_gross.jpg, debug directory batch_map_debug / Output and effect evaluation: After mapping is completed, pathologists can view the distribution of high-risk lesions along the long axis of 9 tissue bands on a single gross image without having to manually number each vertical line.
[0063] Example 2 In this embodiment, the effectiveness of the dual-path lesion information acquisition and skip-reasoning, only-mapping batch processing strategy is verified.
[0064] Experimental Design: Based on the data and configuration of the same case in Example 1, the following comparative group was designed to verify the flexibility of the dual inlet, single pipeline and batch processing of the present invention.
[0065] Experimental Group 1 (Path A, Physician Annotation): Replace the input of step 2 with N.json exported by the pathologist (containing polygon vertices and hierarchical_annotation level fields), skip UNI+TransMIL inference; directly call the same mapping API map_json_to_gross / pipeline to execute steps 3 to 6. Verify that the mapping subsystem does not distinguish the source of lesion information.
[0066] Experimental Group 2 (Path B, AI Inference): This is the complete process of Example 1. Step 2 uses N_ai_result.json, and then steps 3 to 6 are executed.
[0067] Experiment 3 (Skip Inference, Mapping Only): Given that structured lesion information files have already been generated in Experiment 1 or Experiment 2, with `--skip-infer` set, the batch script `run_case_folder_full.py` will not repeat step 2. Steps 3 through 6 will only be executed on the 9 WSI images, accumulating and overlaying the gross images. This is suitable for scenarios where physician annotations or AI results have already been stored and only the gross image mapping needs to be updated.
[0068] Experimental Group 4 (Single-band mapping vs. case batch processing): Steps 1 to 6 were executed only on 3.kfb + 3_ai_result.json to obtain a single-band mapping map; compared with the cumulative results of 9 bands in Experimental Group 2, the correctness of multi-slice incremental overlay (incremental_final_save=true) was verified.
[0069] Comparison conclusion: The overall images output by Experiment 1 and Experiment 2 are consistent in terms of strip position and heat bar direction (under the same line_id and the same mapping parameters), proving the logic of dual-path shared mask rendering and vertical line fusion; The experimental group's three-pipeline time was significantly lower than the complete workflow including inference (step 2, a typical WSI inference of about 116 seconds), reducing the cost of repeated calculations; Experimental group four can be used for single-strip quality control, and experimental group two is used for case-level panoramic display; step 3 is executed only once in batch processing, and steps 4 to 6 are performed in a strip cycle, and step 6 cannot be placed before step 4.
[0070] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made using the content of this specification and illustrations should be included within the protection scope of the present invention.
Claims
1. A method for mapping lesion information from ESD pathological sections to gross specimen images, characterized in that, include: Step S1: Obtain the digital image of the whole pathological section corresponding to the ESD specimen and the gross specimen image obtained by fixing the ESD specimen on the fixation plate. Step S2: Obtain the lesion area and lesion classification information from the digital image of the whole pathological section, and extract the vertical line sequence that intersects with the tissue area corresponding to the ESD specimen from the gross specimen image; Step S3: Compress and project the lesion area along the tissue extension direction to generate a one-dimensional heat bar carrying a lesion grading label representing the lesion classification information; Step S4: Perform positive and negative correction on the one-dimensional heat strip and output the corrected one-dimensional heat strip that is consistent with the tissue extension direction in the gross specimen image. Step S5: Match the slice number of the digital image of the whole pathological slice corresponding to the corrected one-dimensional heat bar to obtain the vertical line number in the vertical line sequence, and superimpose the corrected one-dimensional heat bar onto the tissue area where the vertical line number is located in the gross specimen image to obtain a gross control image mapped with lesion information.
2. The method according to claim 1, characterized in that, The process of obtaining lesion area and lesion classification information from the digital image of the whole pathological section includes: Check if a manually annotated file exists corresponding to the digital image of the pathological whole slide; If the manually labeled file is found, the lesion area and lesion classification information are obtained from the manually labeled file, and invalid labeled data are removed; If the manually annotated file is not found, the digital image of the whole pathological slide is processed based on an artificial intelligence model to obtain the lesion area and the lesion classification information.
3. The method according to claim 1, characterized in that, The sequence of vertical lines extracted from the gross specimen image that intersects with the tissue region corresponding to the ESD specimen includes: Boundary detection is performed on the fixation plate in the gross specimen image to crop out the effective image area of the fixation plate; Identify the vertical grid lines within the effective image area of the fixed plate; The tissue region mask corresponding to the ESD specimen is generated by segmenting the effective image area of the fixed plate, and the vertical grid lines are intersected with the tissue region mask to obtain all vertical lines that intersect with the tissue region mask. The vertical lines are then sorted in spatial order from left to right to form the vertical line sequence.
4. The method according to claim 3, characterized in that, The identification of vertical grid lines within the effective image area of the fixed plate includes: A vertical dark line mask is generated within the effective image area of the fixed plate, and a horizontal morphological opening operation is performed on the vertical dark line mask to remove wide patch-like interference. The vertical dark line mask is divided into multiple independent vertical columns along the horizontal direction. The column support and the longest continuous support ratio of each vertical column are calculated respectively. The vertical column whose column support and the longest continuous support ratio both meet the preset conditions are marked as valid line peaks. Within the effective image area of the fixed plate, a sampling area is defined at the top of the fixed plate, and invalid peaks corresponding to the printed red reference line are removed within the sampling area at the top of the fixed plate according to the color hue and saturation parameters. The median standard spacing of the lines is calculated based on all valid line peaks. A minimum spacing threshold is set based on the median standard spacing of the lines, and clustered pseudo-peaks with spacing smaller than the minimum spacing threshold are filtered out. When the distance between two adjacent vertical lines is greater than a set multiple of the median standard distance between the lines, missing vertical lines are generated at the equal division positions of the adjacent vertical lines, and the effectiveness of the supplemented vertical lines is verified by combining the column support of the local column.
5. The method according to claim 1, characterized in that, Step S3 includes: At a preset reading magnification, each of the lesion classification information is rendered as an independent two-dimensional lesion mask; For each vertical column of the two-dimensional lesion mask, the maximum value of all pixels in that column is calculated along the row direction to complete the compressed projection of the two-dimensional image to one dimension. When multiple lesion classification information exist in the same vertical column, the one-dimensional heat bar is generated by rendering in layers according to the preset coloring priority of the lesion severity. Different colors obtained by layer rendering on the one-dimensional heat bar correspond to different lesion classification identifiers.
6. The method according to claim 1, characterized in that, Step S4 includes: Marking detection areas are delineated at the four corners of the digital image of the whole pathological slide and the gross specimen image, and positioning marks are identified in each of the marking detection areas, and the end orientation corresponding to each positioning mark is identified. By comparing the end orientations of the same type of positioning marker regions in the digital image of the whole pathological slide and the gross specimen image, if the end orientations are consistent, the one-dimensional thermal strip is used as the corrected one-dimensional thermal strip; if the end orientations are inconsistent, the one-dimensional thermal strip is flipped to obtain the corrected one-dimensional thermal strip.
7. The method according to claim 6, characterized in that, The step of identifying the end position corresponding to the positioning mark includes: For each of the marker detection regions, the number of pixels in the region that conform to the color range of the first type of positioning marker and the color range of the second type of positioning marker are counted, and the pixel ratio of the number of pixels of the two types of colors in the corresponding marker detection region is calculated. The marker type corresponding to the color with a higher pixel ratio is determined as the marker type of the positioning marker carried by the corresponding marker detection area, and the end orientation is determined based on the marker type.
8. The method according to claim 1, characterized in that, In step S5, the process of matching the vertical line numbers includes: When the slice number is less than or equal to the total number of vertical lines in the vertical line sequence, the vertical line number in the vertical line sequence that matches the slice number is used as the matched vertical line number. When the slice number is greater than the total number of vertical lines, the vertical line number is calculated using a reverse backfill method. The calculation formula for the reverse backfill method is as follows: ; in, is the vertical line number, n is the total number of vertical lines, and N is the slice number.
9. The method according to claim 1, characterized in that, In step S5, when the corrected one-dimensional heat strip is superimposed onto the tissue region where the vertical line number is located, a rounded rectangular fusion mask is used to superimpose it onto the tissue region. When the same vertical line number corresponds to multiple corrected one-dimensional heat strips, each corrected one-dimensional heat strip is horizontally offset and parallel.
10. A system for mapping lesion information from ESD pathological sections to gross specimen images, characterized in that, The system comprising the method described in any one of claims 1-9, wherein the system includes: The image acquisition module is used to acquire digital images of the pathological whole slides corresponding to the ESD specimen and gross specimen images taken by fixing the ESD specimen on the fixation plate. The image analysis module, connected to the image acquisition module, is used to acquire lesion area and lesion classification information from the digital image of the whole pathological slide, and to extract the vertical line sequence intersecting with the tissue area corresponding to the ESD specimen from the gross specimen image; A heat bar generation module, connected to the image analysis module, is used to compress and project the lesion area along the tissue extension direction to generate a one-dimensional heat bar carrying a lesion grading label that represents the lesion classification information. The orientation alignment module is connected to the image analysis module and the heat bar generation module respectively. It performs positive and negative correction on the one-dimensional heat bar and outputs a corrected one-dimensional heat bar that is consistent with the tissue extension direction in the gross specimen image. The mapping and fusion module, which is connected to the image analysis module and the orientation alignment module respectively, is used to match the slice number of the pathological whole slice digital image corresponding to the corrected one-dimensional heat bar to obtain the corresponding vertical line number in the vertical line sequence, and to superimpose the corrected one-dimensional heat bar onto the tissue area where the vertical line number is located in the gross specimen image to obtain a gross control image mapped with lesion information.
