An interactive annotation and segmentation method and device for whole-section pathological images

Abstract

This invention belongs to the field of pathological image annotation and segmentation technology, and relates to an interactive annotation and segmentation method and apparatus for whole-slice pathological images. The method involves downsampling a whole-slice pathological image to obtain multiple images corresponding to different image resolution levels; dividing each image into multiple image blocks; acquiring a target image based on the zoom level of the user's front-end interface viewport; acquiring image blocks within the target image based on the viewport's coverage area; generating a candidate image block set for each image block to annotate it; and obtaining positive and negative sample point annotations for each image block; inputting the annotated image blocks into an image encoding module to generate image feature embeddings; inputting the positive and negative sample point annotations into a prompt encoding module to generate prompt embedding representations; inputting the image feature embeddings and prompt embedding representations into a mask decoding module to generate a segmentation mask; and superimposing the segmentation masks of each image block onto the whole-slice pathological image to obtain the annotation and segmentation result of the whole-slice pathological image.

Description

Technical Field

[0001] This invention relates to the field of pathological image annotation and segmentation technology, and in particular to an interactive annotation and segmentation method and apparatus for whole-slice pathological images. Background Technology

[0002] The development of digital pathology has made whole-slide images (WSI) an important data foundation for clinical diagnosis and medical research. WSI typically boasts ultra-high resolution with hundreds of millions of pixels, containing rich histological and cellular morphological information. However, it also brings practical problems such as high annotation costs, low efficiency, and large scale spans. High-quality annotation is a prerequisite for training pathology AI models. Therefore, how to leverage more general and efficient methods to support large-scale annotation and segmentation processes has become a research focus in recent years.

[0003] Because WSI images have extremely high resolution, the data size of a single image can reach several gigabytes or even higher. Traditional annotation software relies on preprocessing or downsampling mechanisms when processing WSI images. However, there are problems such as response delay or insufficient detail expression during high-magnification browsing and fine structure annotation, which affects the efficiency of accurate annotation. Furthermore, the image becomes unclear at high magnification, which in turn affects the annotation accuracy and the accuracy of segmentation results. In recent years, general-purpose visual foundational models, represented by the Segment Anything Model (SAM) and its iterative versions SAM2 and SAM3, have made significant progress in the field of natural image segmentation. They exhibit strong general segmentation capabilities through large-scale training and prompt-driven mechanisms. Existing technologies propose to introduce such models into pathological image analysis tasks for annotation and segmentation. However, most of these models are trained on natural images or limited medical images. When applied to WSI images, due to factors such as staining differences, complex tissue structures, and dense cell density, the models struggle to learn the pathological semantic information in the images, leading to inaccurate annotation and segmentation results for some areas with irregular cell boundaries and tissue morphology. In addition, most segmentation models cut the image into many small blocks and input them into the model, performing annotation and segmentation on each image block separately. This fails to utilize the global and continuous structural information of the entire image. However, in WSI images, structures such as cells and glands are arranged in an orderly manner along the tissue direction. These tissues themselves have strong structure, continuity, and directionality. Therefore, if the segmentation model is used for WSI image annotation and segmentation, it will ignore cross-region information, ultimately leading to inaccurate annotation and segmentation results.

[0004] In summary, existing methods for labeling and segmenting whole-section pathological images suffer from inaccurate labeling and segmentation results. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem of inaccurate annotation and segmentation results in the existing whole-slice pathological image annotation and segmentation methods.

[0006] To address the aforementioned technical problems, this invention provides an interactive annotation and segmentation method for whole-slice pathological images, comprising:

[0007] The whole-slice pathological image is input into the image pyramid and downsampled multiple times to obtain multiple images corresponding to different image resolution levels; each image is divided into multiple image blocks of the same size, and the image resolution level number, two-dimensional index position and global spatial coordinates of each image block are obtained;

[0008] The zoom level, spatial position, and coverage of the front-end interface viewport are acquired in real time when the user browses the whole-slice pathological image. The image at the image resolution level corresponding to the zoom level of the current viewport is used as the target image. Image blocks within the coverage area of ​​the current viewport are obtained from the target image to obtain a set of image blocks.

[0009] For each image patch in the image patch set, a candidate image patch set is generated. The image patch is labeled based on the annotation results of each candidate image patch in the candidate image patch set, and the positive sample point annotations and negative sample point annotations of the image patch are obtained based on the annotation results.

[0010] Each labeled image patch is input into the image encoding module to generate an image feature embedding. The positive and negative sample point annotations of the labeled image patch are input into the cue encoding module to generate a cue embedding representation. The image feature embedding and cue embedding representation of the labeled image patch are input into the mask decoding module to generate a segmentation mask for the labeled image patch.

[0011] Based on the global spatial coordinates of each labeled image patch, the segmentation mask of each labeled image patch is superimposed onto the whole-section pathological image to obtain the labeled segmentation result of the whole-section pathological image.

[0012] Preferably, there is an overlapping area between every two adjacent image blocks in each image.

[0013] Preferably, the zoom level, spatial location, and coverage of the front-end interface viewport are acquired in real time when the user browses the whole-slice pathological image. The image at the image resolution level corresponding to the zoom level of the current viewport is used as the target image. Image blocks within the coverage area of ​​the current viewport are obtained from the target image. After obtaining the set of image blocks, the method further includes:

[0014] Calculate the change in viewport zoom based on the zoom levels of the current viewport and the previous viewport; calculate the viewport movement direction vector based on the spatial positions of the current viewport and the previous viewport.

[0015] Based on the zoom change and movement direction vector of the viewport, predict the zoom ratio and spatial position of the next viewport, thereby obtaining the image patch within the coverage area of ​​the next viewport and obtaining a set of neighborhood candidate image patches;

[0016] For each neighboring candidate image patch in the set of neighboring candidate image patches, the preloading priority of the neighboring candidate image patch is calculated based on the normalized distance between the neighboring candidate image patch and the current viewport boundary, the consistency between the relative position vector between the neighboring candidate image patch and the current viewport and the viewport's movement direction vector, the viewport's movement speed, and the frequency of the neighboring candidate image patch within the coverage area of ​​the historical viewport.

[0017] After sorting the neighboring candidate image blocks in the set of neighboring candidate image blocks in descending order according to the preloading priority, a preset number of neighboring candidate image blocks are preloaded.

[0018] Preferably, the formula for calculating the preloading priority of neighboring candidate image blocks is:

[0019] ,

[0020] in, This indicates the preloading priority of the i-th neighboring candidate image block; Represents the normalized distance between the i-th neighboring candidate image patch and the current viewport boundary; Indicates the speed at which the viewport moves; This indicates the consistency between the relative position vector of the i-th neighboring candidate image patch and the current viewport and the movement direction vector of the viewport; This represents the frequency of the i-th neighboring candidate image patch within the coverage area of ​​the historical viewport; The first weighting coefficient; This is the second weighting coefficient; This is the third weighting coefficient.

[0021] Preferably, a candidate image patch set is generated for each image patch in the image patch set, and the image patch is labeled based on the labeling results of each candidate image patch in the candidate image patch set, including:

[0022] Using each image patch in the image patch set as the center of a preset candidate box, a candidate image patch set is generated based on the image patches within the preset candidate box; or, using each image patch in the image patch set as the center and a preset distance as the radius to obtain a candidate range, a candidate image patch set for that image patch is generated based on the labeled image patches within the candidate range; or, for each image patch in the image patch set, a candidate image patch set for that image patch is generated based on image patches with the same image resolution level number as that image patch; or, for each image patch in the image patch set, the distance between the remaining image patches and that image patch is calculated based on the two-dimensional index position or global spatial coordinates, and a candidate image patch set for that image patch is generated based on image patches whose distance is less than a preset threshold.

[0023] For each image patch in the image patch set, the image patch is labeled using manual annotation based on the annotation results of each candidate image patch in the candidate image patch set, or the image patch is labeled using the annotation model based on the annotation results of each candidate image patch in the candidate image patch set, and then the annotation results output by the annotation model are corrected using manual annotation.

[0024] Preferably, the image block is labeled based on the labeling results of each candidate image block in the candidate image block set, including:

[0025] Candidate image blocks with the same image resolution level number as the image block and with spatially overlapping regions are obtained from the candidate image block set of the image block and are regarded as overlapping candidate image blocks at the same level; the image block is labeled based on the annotation results of the overlapping region between the overlapping candidate image blocks at the same level and the image block.

[0026] Candidate image blocks with the same global spatial center position but different image resolution level numbers from the candidate image block set are selected as candidate image blocks with the same spatial position; the image block is labeled based on the annotation results of the candidate image blocks with the same spatial position and the image block with the same pixel position.

[0027] The annotation results and feature information of the remaining candidate image blocks in the candidate image block set based on the image block are used to annotate the image block.

[0028] Preferably, obtaining the positive and negative sample point annotations of the image patch based on the annotation results includes:

[0029] Multiple labeled connected components are obtained based on each labeled region in the image patch;

[0030] In an image patch, connected components with an area smaller than the minimum target area threshold, an area larger than the maximum target area threshold, and / or a gradient within the region smaller than a preset gradient are labeled as negative sample points, while the remaining connected components in the image patch are labeled as positive sample points.

[0031] Preferably, each labeled image patch is input into an image encoding module to generate an image feature embedding; the positive and negative sample point annotations of the labeled image patch are input into a cue encoding module to generate a cue embedding representation; and the image feature embedding and cue embedding representation of the labeled image patch are input into a mask decoding module to generate a segmentation mask for the labeled image patch, including:

[0032] When the number of labeled image blocks is equal to a preset threshold, each labeled image block is input into the image encoding module to generate an image feature embedding. The positive and negative sample point annotations of the labeled image block are input into the prompt encoding module to generate a prompt embedding representation. The image feature embedding and prompt embedding representation of the labeled image block are input into the mask decoding module to generate a segmentation mask for the labeled image block.

[0033] The segmentation loss value is calculated based on the segmentation mask of the labeled image patch and its annotation result. The image encoding module, prompt encoding module and mask decoding module are iteratively trained using the labeled image patch to obtain the trained image patch segmentation model, thereby segmenting the labeled image patch and obtaining the segmentation mask of each labeled image patch.

[0034] Preferably, the segmentation loss value The calculation formula is:

[0035] ,

[0036] in, The mask segmentation loss represents the constraint that the segmentation mask of the labeled image patch must be consistent with its labeling result. express The weights; This represents the flow field regression loss related to constraint separation; express The weights; This represents the regression loss used to enhance the horizontal and vertical distance plots of the segmentation boundary; express The weight.

[0037] The present invention also provides an interactive annotation and segmentation device for whole-slice pathological images, comprising:

[0038] The image processing module is used to input the whole-slice pathological image into the image pyramid for multiple downsampling to obtain multiple images corresponding to different image resolution levels; each image is divided into multiple image blocks of the same size, and the image resolution level number, two-dimensional index position and global spatial coordinates of each image block are obtained;

[0039] The interactive image patch acquisition module is used to acquire the zoom level, spatial position and coverage of the front-end interface viewport when the user browses the whole slice pathological image in real time, and take the image at the image resolution level corresponding to the zoom level of the current viewport as the target image, and acquire the image patch within the coverage of the current viewport in the target image to obtain the image patch set.

[0040] The annotation module is used to generate a candidate image block set for each image block in the image block set, annotate the image block based on the annotation results of each candidate image block in the candidate image block set, and obtain the positive sample point annotations and negative sample point annotations of the image block based on the annotation results;

[0041] The prompt-driven segmentation module is used to input each labeled image block into the image encoding module to generate image feature embedding, input the positive and negative sample point labels of the labeled image block into the prompt encoding module to generate prompt embedding representation, and input the image feature embedding and prompt embedding representation of the labeled image block into the mask decoding module to generate a segmentation mask for the labeled image block.

[0042] The annotation segmentation result acquisition module is used to overlay the segmentation mask of each annotated image block onto the whole slice pathological image based on the global spatial coordinates of each annotated image block, so as to obtain the annotation segmentation result of the whole slice pathological image.

[0043] The interactive annotation and segmentation method for whole-slice pathological images provided in this application has the following beneficial effects:

[0044] 1. Firstly, by constructing a multi-resolution image pyramid structure and combining it with a dynamic image block loading mechanism, hierarchical parsing and on-demand scheduling of ultra-high resolution whole-slice pathological images are achieved. Only image blocks of the resolution level matching the current viewport scaling are loaded, avoiding loading and processing the entire whole-slice pathological image at once. High-resolution image blocks are automatically used when browsing at high magnification, ensuring image clarity and complete details while also considering response speed, thus improving annotation accuracy and efficiency. During annotation and segmentation, the model no longer relies on the original generalization ability of the pre-trained general segmentation model. Instead, it first filters associated multi-level candidate image blocks for each image block within the viewport range. Based on the information of the associated candidate image blocks and the annotation results, the image blocks are annotated. The reuse of inter-block structural information avoids the problem of inaccurate boundary and structural discontinuity in the annotation and segmentation results caused by ignoring the continuity and arrangement of tissues in the pathological image. Finally, positive and negative sample points are generated based on the annotation results of the image blocks, forming accurate prompts for the structure of the pathological image. The prompt-driven segmentation method makes the model focus on key pathological structures, making the segmentation results more consistent with the structural characteristics of pathological tissues and improving the accuracy of the segmentation results.

[0045] 2. By combining the viewport movement pattern of the front-end interface, the position and zoom level of the next viewport are predicted, thereby performing intelligent preloading of image blocks to ensure that the browsing process is without delay or blurring. This avoids the interactive lag problem when browsing large-size whole-slice pathological images region by region, and improves the efficiency of annotation and segmentation.

[0046] 3. When annotating candidate image blocks based on image blocks, the annotation results of overlapping candidate image blocks at the same level guide the annotation, achieving spatial smoothness and boundary consistency, and avoiding the fragmentation of annotation between blocks; the use of candidate image blocks at multiple scales with the same spatial center achieves complementary information between the upper and lower structures, ensuring the integrity of large-scale organizational morphology; finally, the annotation is guided by the global similarity region information of the remaining candidate image blocks, making full use of the complementary information characteristics between image blocks. Attached Figure Description

[0047] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0048] Figure 1 Flowchart of the interactive annotation and segmentation method for whole-slice pathological images provided in this application;

[0049] Figure 2 A schematic diagram illustrating the analysis of whole-slice pathological images of the pyramid index results provided in this application;

[0050] Figure 3 A schematic diagram of the annotation pattern based on image patch segmentation provided in this application;

[0051] Figure 4 A schematic diagram illustrating tissue identification and automatic filtering of blank image blocks provided for this application;

[0052] Figure 5 This is a schematic diagram of image block brushing provided in this application;

[0053] Figure 6 This application provides a schematic diagram illustrating the image patch size setting based on user needs.

[0054] Figure 7 A schematic diagram of manually labeled image blocks provided in this application;

[0055] Figure 8 Example image of the labeled segmentation result of the image patch provided in this application; wherein, Figure 8 (a) in the image is an unlabeled patch. Figure 8 (b) in the image represents the labeled segmented image patch;

[0056] Figure 9 Export example images of the image patch annotation and segmentation results provided in this application;

[0057] Figure 10 Example image of the labeled segmentation results provided in this application;

[0058] Figure 11 Example diagram of structured data exported for this application. Detailed Implementation

[0059] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0060] Please see Figure 1 , Figure 1 The flowchart shown is a method for interactive annotation and segmentation of whole-slice pathological images provided in this application. The method includes S10 to S50:

[0061] S10: Input the whole-slice pathological image into the image pyramid and downsample it multiple times to obtain multiple images corresponding to different image resolution levels; divide each image into multiple image blocks of the same size, and obtain the image resolution level number, two-dimensional index position and global spatial coordinates of each image block.

[0062] Specifically, the image pyramid includes at least two image levels with different scaling ratios. Each level is downsampled according to a preset ratio to support image browsing and annotation operations at different scaling ratios. Figure 2 The image shown is a schematic diagram of the whole-slice pathological image analysis of the pyramid index results provided in this application.

[0063] Furthermore, at each resolution level, the image is systematically divided into blocks according to a preset size, resulting in multiple fixed-size image blocks. For example, the size of the image block can be set to 512×512 pixels or 1024×1024 pixels, etc. Figure 3 The figure shown is a schematic diagram of the annotation mode based on image block segmentation provided in this application.

[0064] Preferably, considering the correlation between various tissue structures in the whole-section pathological image, there is an overlapping area between every two adjacent image blocks in each image, so as to reduce the loss of structural information at the boundary of image blocks and improve the continuity and accuracy of subsequent annotation and segmentation.

[0065] Through the above image processing steps, the entire whole-slice pathological image forms a ternary index structure of hierarchy-block index-spatial coordinates, which can realize unified coordinate mapping and image block positioning between different image resolution levels.

[0066] Optionally, in some embodiments, since there may be approximately 40% or 50% effective tissue regions in low-magnification images, tissue region detection can be performed on the low-magnification images first to generate a tissue region mask, i.e., to identify background blank areas. A tissue region index table is then established based on the tissue region mask, allowing for automatic filtering of background blank areas during subsequent image block scheduling. Figure 4 As shown in the image.

[0067] S20: Real-time acquisition of the zoom level, spatial position, and coverage of the front-end interface viewport when the user browses the whole-slice pathological image, and taking the image at the image resolution level corresponding to the zoom level of the current viewport as the target image, and obtaining the image blocks within the coverage area of ​​the current viewport in the target image to obtain the image block set.

[0068] Specifically, during the process of a user browsing a full-slice pathological image, the viewport of the front-end interface will continuously change according to the user's browsing position and zoom requirements. The full-slice pathological image is dynamically scheduled and loaded by obtaining the center coordinates, zoom level, and spatial range of the current viewport in real time. During the loading process, only the image block within the current viewport coverage area of ​​the corresponding image resolution level is obtained according to the zoom level, instead of the entire image. This can reduce the system's video memory and memory usage, and improve the interactive response speed of the browsing and annotation process.

[0069] Furthermore, in order to reduce the loading delay caused by frequent disk I / O operations, image blocks that may enter the viewport range at the next moment can be predicted based on the change of viewport position and loaded in advance. This allows for efficient on-demand loading and acquisition of ultra-high resolution whole-slice pathological images while ensuring smooth interaction.

[0070] Specifically, step S20 is followed by steps 1-1 to 1-4:

[0071] Step 1-1: Calculate the change in viewport scaling based on the scaling ratio of the current viewport and the previous viewport; calculate the viewport movement direction vector based on the spatial position of the current viewport and the previous viewport.

[0072] Step 1-2: Based on the zoom change of the viewport and the movement direction vector, predict the zoom ratio and spatial position of the next viewport, thereby obtaining the image patch within the coverage area of ​​the next viewport and obtaining a set of neighborhood candidate image patches.

[0073] For example, a convolutional neural network model can be trained using historical viewport zoom levels and spatial location data, enabling the model to capture the viewport movement and zoom patterns of users browsing full-slice pathological images. The trained convolutional neural network model can then be used to predict viewport zoom levels and spatial locations.

[0074] It should be noted that the coverage area of ​​the viewport is related to the size of the front-end display, so it is only necessary to predict the coordinates of the viewport center to obtain the image patch within its coverage area.

[0075] Steps 1-3: For each neighboring candidate image patch in the set of neighboring candidate image patches, calculate the preloading priority of the neighboring candidate image patch based on the normalized distance between the neighboring candidate image patch and the current viewport boundary, the consistency between the relative position vector between the neighboring candidate image patch and the current viewport and the viewport's movement direction vector, the viewport's movement speed, and the frequency of the neighboring candidate image patch within the coverage area of ​​the historical viewport.

[0076] Specifically, the formula for calculating the preloading priority of neighboring candidate image patches is as follows:

[0077] ,

[0078] in, This indicates the preloading priority of the i-th neighboring candidate image block; Represents the normalized distance between the i-th neighboring candidate image patch and the current viewport boundary; Indicates the speed at which the viewport moves; This indicates the consistency between the relative position vector of the i-th neighboring candidate image patch and the current viewport and the movement direction vector of the viewport; This represents the frequency of the i-th neighboring candidate image patch within the coverage area of ​​the historical viewport; The first weighting coefficient; This is the second weighting coefficient; This is the third weighting coefficient.

[0079] It should be noted that, .

[0080] Specifically, the normalized distance between the neighboring candidate image patch and the current viewport boundary is represented by the normalized value of the distance between the center point coordinates of the neighboring candidate image patch and each boundary of the current viewport; the viewport movement speed can be calculated based on the current viewport position, the next viewport position, and the time between the two movements; the consistency between the relative position vector between the neighboring candidate image patch and the current viewport and the viewport movement direction vector can be determined using the angle relationship between the two vectors, which is used to characterize the consistency of the neighboring candidate image patch's movement direction relative to the current viewport; the frequency of the neighboring candidate image patch within the coverage area of ​​the historical viewport refers to the frequency with which the neighboring candidate image patch appears within the viewport coverage area from the start of the user browsing the full-slice pathological image.

[0081] In a specific example of this application, , , The values ​​are 0.5, 0.3, and 0.2.

[0082] Steps 1-4: After sorting the neighboring candidate image blocks in the neighboring candidate image block set in descending order according to the preloading priority, a preset number of neighboring candidate image blocks are preloaded.

[0083] This application improves the browsing smoothness of large-size whole-slice pathological images by designing a neighborhood preloading mechanism, reduces the memory usage caused by loading invalid data, and effectively improves the user's interactive experience.

[0084] S30: Generate a candidate image block set for each image block in the image block set, annotate the image block based on the annotation results of each candidate image block in the candidate image block set, and obtain the positive sample point annotation and negative sample point annotation of the image block based on the annotation results.

[0085] Specifically, step S30 includes S300~S303:

[0086] S300: Using each image block in the image block set as the center of a preset candidate box, generate a candidate image block set based on the image blocks within the preset candidate box; or, using each image block in the image block set as the center and a preset distance as the radius to obtain a candidate range, generate a candidate image block set for that image block based on the labeled image blocks within the candidate range; or, for each image block in the image block set, generate a candidate image block set for that image block based on image blocks with the same image resolution level number as that image block; or, for each image block in the image block set, calculate the distance between the remaining image blocks and that image block based on the two-dimensional index position or global spatial coordinates, and generate a candidate image block set for that image block based on image blocks whose distance is less than a preset threshold.

[0087] S301: For each image patch in the image patch set, the image patch is labeled using manual annotation based on the annotation results of each candidate image patch in the candidate image patch set, or the image patch is labeled using the annotation model based on the annotation results of each candidate image patch in the candidate image patch set, and then the annotation results output by the annotation model are corrected using manual annotation.

[0088] Specifically, the image block is labeled based on the labeling results of each candidate image block in the candidate image block set, including steps 2-1 to 2-3:

[0089] Step 2-1: Obtain candidate image blocks from the candidate image block set that have the same image resolution level number as the image block and have spatially overlapping regions, and use them as overlapping candidate image blocks at the same level; annotate the image block based on the annotation results of the overlapping regions between the overlapping candidate image blocks at the same level and the image block.

[0090] Step 2-2: Obtain candidate image blocks from the candidate image block set that have the same global spatial center position as the image block but different image resolution level numbers, and use them as candidate image blocks with the same spatial position; annotate the image block based on the annotation results of the candidate image blocks with the same spatial position and the image block at the same pixel position.

[0091] Steps 2-3: Based on the annotation results and feature information of the remaining candidate image blocks in the candidate image block set (excluding overlapping candidate image blocks at the same level and candidate image blocks at the same spatial location), annotate the image block.

[0092] It should be noted that steps 2-3 can utilize either manual annotation or model annotation. Manual annotation can directly combine pathological knowledge with the feature information of the image patch itself for annotation. Model annotation can first calculate the feature similarity between the remaining candidate image patches and the current image patch, assign weights to each remaining candidate image patch based on feature similarity, and input each remaining candidate image patch, its features, weights, annotation results, and the current image patch and its features into the annotation model. This allows the model to learn the correlation between the feature distribution and the annotation results based on the candidate image patches with high similarity, thereby annotating the image patch. After model annotation, the model annotation results can be manually corrected to improve the accuracy of the annotation results. The annotation model can be a fine-tuned version of an open-source model on an open-source dataset, such as suda_cellpose, suda_hover-net, or suda_sam.

[0093] For example, after maintaining a set of candidate image blocks for each image block, a fine-grained annotation interface can be created based on the image blocks selected by the user on the front-end interface. This interface can automatically load relevant data for the selected image blocks, such as their position in the whole-slice pathological image and their image resolution level number. Alternatively, a series of selected image blocks can be loaded based on the user's selection mode on the front-end interface. Figure 5 As shown; after selecting an image patch, the size of the image patch can be set according to user needs for fine annotation, such as... Figure 6 As shown.

[0094] In manual annotation mode, you can use polygon tools (supporting point-by-point outlining, node addition, deletion and dragging, and closure constraints), brush / eraser tools (supporting adjustable brush radius, hardness and smoothing intensity), area filling / removal (supporting quick filling of closed areas and removal of mis-annotated areas), boundary smoothing and hole filling (post-processing such as smoothing of manual contours, cleaning of connected components, and hole filling), and instance-level management (operations such as selection, deletion, merging, splitting, and numbering of each target).

[0095] After annotation is completed, the image patches and their annotation results are converted into pixel-level label masks, and incrementally saved in the local cache and database, such as... Figure 7 The image shown is a schematic diagram of the manually annotated image block provided in this application.

[0096] S302: Obtain multiple labeled connected components based on each labeled region in the image patch.

[0097] S303: Label connected components in the image patch whose area is less than the minimum target area threshold or whose area is greater than the maximum target area threshold and / or whose gradient is less than the preset gradient as negative sample points, and label the remaining connected components in the image patch as positive sample points.

[0098] Alternatively, users can manually select positive and negative sample points.

[0099] S40: Input each labeled image block into the image encoding module to generate image feature embedding, input the positive and negative sample point annotations of the labeled image block into the prompt encoding module to generate prompt embedding representation, and input the image feature embedding and prompt embedding representation of the labeled image block into the mask decoding module to generate the segmentation mask of the labeled image block.

[0100] Based on the annotation results of the aforementioned steps, this application also designs a prompt-driven segmentation mechanism, which combines the annotation information of image blocks to generate segmentation prompt features, thereby guiding the model to output segmentation results.

[0101] Specifically, the image feature embedding representation output by the image coding module for:

[0102] ,

[0103] in, This represents the input labeled image patch; This indicates the image encoding module.

[0104] The hint embedding representation output by the hint encoding module for:

[0105] ,

[0106] in, This indicates the positive and negative sample point annotations of the labeled image patch; This indicates the prompt encoding module.

[0107] The segmentation mask representation output by the mask decoding module for:

[0108] ,

[0109] in, This indicates the mask decoding module.

[0110] Specifically, the labeled and segmented image patch data can be stored or exported in a structured manner. The data includes the original image patch and its image resolution level number, two-dimensional index location and global spatial coordinates, annotation results, user-manually annotated annotation information, segmentation results, etc. Figure 8 The image shown is an example of the labeled segmentation result of the image patch provided in this application, wherein, Figure 8 (a) in the image is an unlabeled patch. Figure 8 In the diagram, (b) represents the labeled segmented image patch, such as... Figure 9 The image shown is an example of the exported image block annotation and segmentation results provided in this application.

[0111] Optionally, in order to further optimize the segmentation accuracy of the prompt-driven segmentation mechanism and enable the model to gradually adapt to the data distribution characteristics of the current whole-slice pathological images, some embodiments of this application also design an online model optimization mechanism, which specifically includes steps 3-1 to 3-2:

[0112] Step 3-1: When the number of labeled image blocks is equal to the preset threshold, each labeled image block is input into the image encoding module to generate image feature embedding. The positive and negative sample point annotations of the labeled image block are input into the prompt encoding module to generate prompt embedding representation. The image feature embedding and prompt embedding representation of the labeled image block are input into the mask decoding module to generate the segmentation mask of the labeled image block.

[0113] Step 3-2: Calculate the segmentation loss value based on the segmentation mask of the labeled image patch and its annotation result, and use the labeled image patch to iteratively train the image encoding module, the prompt encoding module and the mask decoding module to obtain the trained image patch segmentation model, thereby segmenting the labeled image patch and obtaining the segmentation mask of each labeled image patch.

[0114] Specifically, segmentation loss value The calculation formula is:

[0115] ,

[0116] in, The mask segmentation loss represents the constraint that the segmentation mask of the labeled image patch must be consistent with its labeling result. express The weights; This represents the flow field regression loss related to constraint separation; express The weights; This represents the regression loss used to enhance the horizontal and vertical distance plots of the segmentation boundary; express The weight.

[0117] Specifically, Binary cross-entropy loss or Dice loss can be used for calculation.

[0118] Preferably, during model training, to ensure model stability and reduce computational overhead, a low-rank adaptation matrix can be introduced into the model's attention layer or linear mapping layer to perform a low-rank update on the original weight matrix. For example, for any linear transformation weight matrix... The updated weights are represented as ,in, , for matrix, for matrix, It is a rank parameter. During training, only A and B are updated, without updating the original weight matrix W, thus achieving lightweight model adaptation. Furthermore, as data accumulates gradually during the annotation process, the model can perform an iterative update each time a certain amount of labeled data is reached. This allows the model to continuously adapt to the current data distribution characteristics, improving the accuracy of subsequent annotation and segmentation, and enhancing the user experience.

[0119] S50: Based on the global spatial coordinates of each labeled image block, the segmentation mask of each labeled image block is superimposed onto the whole slice pathological image to obtain the labeled segmentation result of the whole slice pathological image.

[0120] like Figure 10 The image shown is an example of the labeled segmentation results provided in this application. Figure 11 The figure shown is an example diagram of structured data export provided in this application.

[0121] Based on the interactive annotation and segmentation method for whole-slice pathological images provided in the above embodiments, this application also provides an interactive annotation and segmentation device for whole-slice pathological images, which specifically includes:

[0122] The image processing module is used to input the whole-slice pathological image into the image pyramid for multiple downsampling to obtain multiple images corresponding to different image resolution levels; each image is divided into multiple image blocks of the same size, and the image resolution level number, two-dimensional index position and global spatial coordinates of each image block are obtained;

[0123] The interactive image patch acquisition module is used to acquire the zoom level, spatial position and coverage of the front-end interface viewport when the user browses the whole slice pathological image in real time, and take the image at the image resolution level corresponding to the zoom level of the current viewport as the target image, and acquire the image patch within the coverage of the current viewport in the target image to obtain the image patch set.

[0124] The annotation module is used to generate a candidate image block set for each image block in the image block set, annotate the image block based on the annotation results of each candidate image block in the candidate image block set, and obtain the positive sample point annotations and negative sample point annotations of the image block based on the annotation results;

[0125] The prompt-driven segmentation module is used to input each labeled image block into the image encoding module to generate image feature embedding, input the positive and negative sample point labels of the labeled image block into the prompt encoding module to generate prompt embedding representation, and input the image feature embedding and prompt embedding representation of the labeled image block into the mask decoding module to generate a segmentation mask for the labeled image block.

[0126] The annotation segmentation result acquisition module is used to overlay the segmentation mask of each annotated image block onto the whole slice pathological image based on the global spatial coordinates of each annotated image block, so as to obtain the annotation segmentation result of the whole slice pathological image.

[0127] The interactive annotation and segmentation method for whole-slice pathological images described above will be explained and illustrated through a specific embodiment:

[0128] In this embodiment, the front end uses an HTML+CSS+JS framework, the back end uses a Django+PyTorch framework, and the database uses SQLite.

[0129] The specific steps are as follows: The user uploads the histopathological slides to be labeled and segmented. The system automatically generates a pyramid slide index. The user browses the image at 20x magnification and clicks on a tissue region. The system backend calls the model to extract image blocks based on the clicked location and labels them. The user corrects the labeling results. For example, if the labeling mask has edge overflow, the user clicks on the overflow area. Based on the correction results, positive and negative labeled samples are obtained to train the segmentation model. The trained segmentation model is then used to label and segment the whole-slice pathological image.

[0130] This application achieves hierarchical analysis and on-demand scheduling of ultra-high resolution whole-slice pathological images by constructing a multi-resolution image pyramid structure and combining it with a dynamic image block loading mechanism. This avoids the memory pressure caused by loading the entire image, enabling the system to achieve continuous browsing and fine annotation of large-scale images under normal computing resource conditions, thereby improving the smoothness and stability of the annotation process.

[0131] Secondly, by introducing a prompt-driven interactive segmentation mechanism, the feature representation capabilities of the large visual model are combined with user interaction, enabling the system to generate high-quality segmentation results with minimal user intervention and supporting real-time correction and updates. This mechanism reduces the workload of repetitive manual drawing and improves annotation efficiency. Simultaneously, by introducing instance separation mechanisms based on pixel-level flow field prediction or horizontal / vertical distance maps into the instance segmentation module, the system possesses stronger instance discrimination capabilities when handling scenes with dense cell nuclei or adhered boundaries, thereby improving segmentation stability and boundary accuracy under complex tissue structures.

[0132] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An interactive annotation and segmentation method for whole-section pathological images, characterized in that, include: The whole-section pathological image is input into the image pyramid and downsampled multiple times to obtain multiple images corresponding to different image resolution levels; Each image is divided into multiple image blocks of the same size, and the image resolution level number, two-dimensional index position and global spatial coordinates of each image block are obtained; The zoom level, spatial position, and coverage of the front-end interface viewport are acquired in real time when the user browses the whole-slice pathological image. The image at the image resolution level corresponding to the zoom level of the current viewport is used as the target image. Image blocks within the coverage area of ​​the current viewport are obtained from the target image to obtain a set of image blocks. For each image patch in the image patch set, a candidate image patch set is generated. The image patch is labeled based on the annotation results of each candidate image patch in the candidate image patch set, and the positive sample point annotations and negative sample point annotations of the image patch are obtained based on the annotation results. Each labeled image patch is input into the image encoding module to generate an image feature embedding. The positive and negative sample point annotations of the labeled image patch are input into the cue encoding module to generate a cue embedding representation. The image feature embedding and cue embedding representation of the labeled image patch are input into the mask decoding module to generate a segmentation mask for the labeled image patch. Based on the global spatial coordinates of each labeled image patch, the segmentation mask of each labeled image patch is superimposed onto the whole-section pathological image to obtain the labeled segmentation result of the whole-section pathological image.

2. The interactive annotation and segmentation method for whole-slice pathological images according to claim 1, characterized in that, There is an overlapping area between every two adjacent image patches in each image.

3. The interactive annotation and segmentation method for whole-slice pathological images according to claim 1, characterized in that, The system acquires the zoom level, spatial location, and coverage of the front-end viewport when the user browses whole-slice pathological images in real time. It then uses the image at the resolution level corresponding to the current zoom level as the target image, extracts image patches within the coverage area of ​​the current viewport from the target image, and further obtains the image patch set, which includes: Calculate the change in viewport zoom based on the zoom levels of the current viewport and the previous viewport; calculate the viewport movement direction vector based on the spatial positions of the current viewport and the previous viewport. Based on the zoom change and movement direction vector of the viewport, predict the zoom ratio and spatial position of the next viewport, thereby obtaining the image patch within the coverage area of ​​the next viewport and obtaining a set of neighborhood candidate image patches; For each neighboring candidate image patch in the set of neighboring candidate image patches, the preloading priority of the neighboring candidate image patch is calculated based on the normalized distance between the neighboring candidate image patch and the current viewport boundary, the consistency between the relative position vector between the neighboring candidate image patch and the current viewport and the viewport's movement direction vector, the viewport's movement speed, and the frequency of the neighboring candidate image patch within the coverage area of ​​the historical viewport. After sorting the neighboring candidate image blocks in the set of neighboring candidate image blocks in descending order according to the preloading priority, a preset number of neighboring candidate image blocks are preloaded.

4. The interactive annotation and segmentation method for whole-slice pathological images according to claim 3, characterized in that, The formula for calculating the preloading priority of neighboring candidate image patches is: ， in, This indicates the preloading priority of the i-th neighboring candidate image block; Represents the normalized distance between the i-th neighboring candidate image patch and the current viewport boundary; Indicates the speed at which the viewport moves; This indicates the consistency between the relative position vector of the i-th neighboring candidate image patch and the current viewport and the movement direction vector of the viewport; This represents the frequency of the i-th neighboring candidate image patch within the coverage area of ​​the historical viewport; The first weighting coefficient; This is the second weighting coefficient; This is the third weighting coefficient.

5. The interactive annotation and segmentation method for whole-slice pathological images according to claim 1, characterized in that, For each image patch in the image patch set, a candidate image patch set is generated. Based on the annotation results of each candidate image patch in the candidate image patch set, the image patch is annotated, including: Using each image patch in the image patch set as the center of a preset candidate box, a candidate image patch set is generated based on the image patches within the preset candidate box; or, using each image patch in the image patch set as the center and a preset distance as the radius to obtain a candidate range, a candidate image patch set for that image patch is generated based on the labeled image patches within the candidate range; or, for each image patch in the image patch set, a candidate image patch set for that image patch is generated based on image patches with the same image resolution level number as that image patch; or, for each image patch in the image patch set, the distance between the remaining image patches and that image patch is calculated based on the two-dimensional index position or global spatial coordinates, and a candidate image patch set for that image patch is generated based on image patches whose distance is less than a preset threshold. For each image patch in the image patch set, the image patch is labeled using manual annotation based on the annotation results of each candidate image patch in the candidate image patch set, or the image patch is labeled using the annotation model based on the annotation results of each candidate image patch in the candidate image patch set, and then the annotation results output by the annotation model are corrected using manual annotation.

6. The interactive annotation and segmentation method for whole-slice pathological images according to claim 5, characterized in that, The image patch is annotated based on the annotation results of each candidate image patch in the candidate image patch set, including: Candidate image blocks with the same image resolution level number as the image block and with spatially overlapping regions are obtained from the candidate image block set of the image block and are regarded as overlapping candidate image blocks at the same level; the image block is labeled based on the annotation results of the overlapping region between the overlapping candidate image blocks at the same level and the image block. Candidate image blocks with the same global spatial center position but different image resolution level numbers from the candidate image block set are selected as candidate image blocks with the same spatial position; the image block is labeled based on the annotation results of the candidate image blocks with the same spatial position and the image block with the same pixel position. The annotation results and feature information of the remaining candidate image blocks in the candidate image block set based on the image block are used to annotate the image block.

7. The interactive annotation and segmentation method for whole-slice pathological images according to claim 5, characterized in that, Based on the annotation results, obtain the positive and negative sample point annotations for the image patch, including: Multiple labeled connected components are obtained based on each labeled region in the image patch; In an image patch, connected components with an area smaller than the minimum target area threshold, an area larger than the maximum target area threshold, and / or a gradient within the region smaller than a preset gradient are labeled as negative sample points, while the remaining connected components in the image patch are labeled as positive sample points.

8. The interactive annotation and segmentation method for whole-slice pathological images according to claim 1, characterized in that, Each labeled image patch is input into the image encoding module to generate an image feature embedding. The positive and negative sample point annotations of the labeled image patch are input into the cue encoding module to generate a cue embedding representation. The image feature embedding and cue embedding representation of the labeled image patch are input into the mask decoding module to generate a segmentation mask for the labeled image patch, including: When the number of labeled image blocks is equal to a preset threshold, each labeled image block is input into the image encoding module to generate an image feature embedding. The positive and negative sample point annotations of the labeled image block are input into the prompt encoding module to generate a prompt embedding representation. The image feature embedding and prompt embedding representation of the labeled image block are input into the mask decoding module to generate a segmentation mask for the labeled image block. The segmentation loss value is calculated based on the segmentation mask of the labeled image patch and its annotation result. The image encoding module, prompt encoding module and mask decoding module are iteratively trained using the labeled image patch to obtain the trained image patch segmentation model, thereby segmenting the labeled image patch and obtaining the segmentation mask of each labeled image patch.

9. The interactive annotation and segmentation method for whole-slice pathological images according to claim 8, characterized in that, Segmentation loss value The calculation formula is: ， in, The mask segmentation loss represents the constraint that the segmentation mask of the labeled image patch must be consistent with its labeling result. express The weights; This represents the flow field regression loss related to constraint separation; express The weights; This represents the regression loss used to enhance the horizontal and vertical distance plots of the segmentation boundary; express The weight.

10. An interactive annotation and segmentation device for whole-slice pathological images, characterized in that, include: The image processing module is used to input the whole-slice pathological image into the image pyramid for multiple downsampling to obtain multiple images corresponding to different image resolution levels; Each image is divided into multiple image blocks of the same size, and the image resolution level number, two-dimensional index position and global spatial coordinates of each image block are obtained; The interactive image patch acquisition module is used to acquire the zoom level, spatial position and coverage of the front-end interface viewport when the user browses the whole slice pathological image in real time, and take the image at the image resolution level corresponding to the zoom level of the current viewport as the target image, and acquire the image patch within the coverage of the current viewport in the target image to obtain the image patch set. The annotation module is used to generate a candidate image block set for each image block in the image block set, annotate the image block based on the annotation results of each candidate image block in the candidate image block set, and obtain the positive sample point annotations and negative sample point annotations of the image block based on the annotation results; The prompt-driven segmentation module is used to input each labeled image block into the image encoding module to generate image feature embedding, input the positive and negative sample point labels of the labeled image block into the prompt encoding module to generate prompt embedding representation, and input the image feature embedding and prompt embedding representation of the labeled image block into the mask decoding module to generate a segmentation mask for the labeled image block. The annotation segmentation result acquisition module is used to overlay the segmentation mask of each annotated image block onto the whole slice pathological image based on the global spatial coordinates of each annotated image block, so as to obtain the annotation segmentation result of the whole slice pathological image.

Images