Automatic construction of pathological image dataset and training of cell nucleus detection and classification method based on spatial transcriptome technology

Abstract

The application discloses a kind of based on spatial transcriptome technology automatic construction pathology image dataset and training cell nucleus detection and classification method, belong to image processing and artificial intelligence auxiliary pathological diagnosis field.The method is by obtaining spatial transcriptome public data set, after pretreatment, deconvolution cell type annotation and cell nucleus instance segmentation, automatic construction includes image block, weak supervision / semi-supervised label and cell nucleus boundary information dataset, reduce dependence on artificial annotation.Further, design detection and classification model, adopt multiscale deformable attention encoder, decoupled detection and classification decoder, and introduce limited deformable cross attention mechanism in classification decoder, combined with KL divergence classification loss, realize from regional level proportion label learning instance level cell nucleus class.The application realizes end-to-end automation, improves cell nucleus detection and classification precision and efficiency, provides high-quality pre-training model basis for downstream pathological analysis.

Description

Technical Field

[0001] This invention belongs to the field of image processing, and in particular relates to a cell nucleus detection and classification method based on an automated process for constructing a weakly supervised dataset for training. Background Technology

[0002] In tumor pathology diagnosis, whole-slide images (WSI) play a crucial role. Pathologists need to visually identify the types of cell nuclei in the images for diagnosis, a process that is typically time-consuming and labor-intensive. To aid diagnosis, researchers have attempted to construct datasets containing cell nucleus segmentation and type annotation to train artificial intelligence models. However, existing datasets largely rely on manual annotation by pathologists, resulting in problems such as a limited number of annotated samples (usually less than 10,000 image patches), a limited number of identifiable cell types, and inconsistent annotation quality across different datasets, thus hindering further improvements in model performance.

[0003] In recent years, the rise of spatial transcriptomics technology has provided a new approach to solving the aforementioned annotation bottlenecks. This technology can detect gene expression information in local regions of pathological images. By using deconvolution tools and combining them with single-cell transcriptomics data, the proportion of each cell type within that region can be calculated. Further integration with cell nucleus segmentation tools allows for the automated acquisition of cell nucleus boundary information. Based on this, large-scale, weakly supervised pathological image cell nucleus datasets can be constructed, significantly reducing the reliance on manual annotation.

[0004] Faced with the massive, weakly supervised data generated by this approach, designing efficient and robust detection and classification models becomes crucial. Attention-based models (such as Transformers) have demonstrated superior large-scale learning capabilities compared to traditional CNNs and RNNs in natural language and image tasks. Among them, the DETR (Detection Transformer) model, based on Deformable Attention, has significant advantages in image detection and classification tasks: firstly, it performs better in detecting small targets; secondly, it is more computationally efficient—the computational complexity of Deformable Attention is O(HWC). 2 The O(H) is far lower than that of the standard self-attention mechanism. 2 W 2 C) It has a significant advantage, especially when the feature map size HW is much larger than the number of channels C; third, the model converges faster during training.

[0005] However, directly applying such advanced models to weakly supervised pathological image data based on spatial transcriptomics still faces challenges such as mutual interference between detection and classification tasks, and how to effectively extract instance-level features from weakly supervised signals. Therefore, there is an urgent need for a cell nucleus detection and classification method that can adapt to training on multiple types of data and is optimized for the characteristics of weakly supervised pathological images. Summary of the Invention

[0006] The purpose of this invention is to provide a cell nucleus detection and classification method that can adapt to training with multiple types of data and is optimized for the characteristics of weakly supervised pathological images. By using pathological images and gene expression paired data provided by spatial transcriptomics technology, and with the help of existing tools, a weakly supervised dataset is automatically constructed. Based on the advantages of Deformable DETR, this advanced model can be trained on a large amount of weakly supervised dataset to detect and classify cell nucleus types only from pathological images.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of this invention provides a method for automatically constructing a pathological image dataset based on spatial transcriptomics technology. This method provides suitable input data for subsequent model training and includes the following steps:

[0009] S1: Obtain a publicly available spatial transcriptome dataset for pathological image-gene expression matching;

[0010] S2: Preprocess the raw data to unify the scale of pathological images, and generate a first intermediate file containing image blocks and their spatial coordinates according to different spatial transcriptomics technology platforms;

[0011] S3: Using deconvolution tools and combined with single-cell transcriptome reference data, cell type annotation is performed on the gene expression data obtained in step S2 to generate cell type ratio labels or single-cell type labels, and the annotation results are quality controlled.

[0012] S4: Using a pre-trained cell nucleus segmentation tool, perform cell nucleus instance segmentation on the pathological image processed in step S2 to obtain cell nucleus boundary and bounding box information;

[0013] S5: Integrate the cell type labels generated in step S3 with the cell nuclear boundary information obtained in step S4, and store them as a structured data file for model training.

[0014] The core of this scheme lies in generating a special supervisory signal: for each image region, it provides the overall proportional distribution of various cell nuclei within the region, rather than the precise category of each cell nucleus. This data format is a key input for adapting to subsequent weakly supervised learning models.

[0015] The second aspect of this invention provides a cell nucleus detection and classification method trained on the aforementioned constructed dataset. This method is specifically designed for processing the data constructed using the first technical solution. Its training process takes the aforementioned data as input and includes the following steps:

[0016] A: Construct a data interface to read image patches, weakly supervised scale labels, and cell nucleus bounding boxes from the above structured dataset;

[0017] B: Construct a feature extraction network, a backbone for extracting multi-scale features, and an encoder for further feature extraction;

[0018] C: Construct a detection decoder to predict the detection box of the cell nucleus from the features;

[0019] D: Construct a classification decoder. Its core design lies in using detection box information to extract local features of each cell nucleus instance from global features and predict its category.

[0020] E: Construct a joint loss function that includes a key weakly supervised classification loss term. This loss function guides the model's learning by calculating the difference (such as KL divergence) between the model's aggregated distribution of all predicted cell classes within the entire image patch and the image patch-level scale labels provided by the dataset.

[0021] The core innovation of this approach lies in its model architecture and loss function, which are specifically optimized for the "regional proportion labels" generated by the first approach. Through a decoupled "detection-classification" architecture and a constrained attention mechanism, the model can effectively infer the category of a single cell from weakly supervised signals; while the weakly supervised classification loss establishes a mathematical bridge between region-level labels and instance-level predictions, making it possible to train the model using automatically generated labels.

[0022] There is a close technical correlation and interdependence between the first and second schemes mentioned above: the specific form of data generated by the first scheme (weakly supervised / semi-supervised labels) is the prerequisite and foundation for the effective operation of the second scheme; while the model architecture and loss function of the second scheme are specialized designs for the data characteristics of the first scheme, providing a technical guarantee for fully mining the value of this data and achieving the final application goal. Together, they constitute a complete technical closed loop from "automated data production" to "specialized model". Compared with the prior art, the beneficial effects of this invention are:

[0023] 1. End-to-end automation of data construction and model training has been achieved: It completely eliminates the reliance on manual annotation by pathology experts and forms a highly efficient pipeline of "spatial transcriptome data → automated annotation → dedicated model training", which greatly reduces the threshold and cost of building an AI pathology analysis system.

[0024] 2. The high degree of model-data fit improves learning efficiency and performance: The dedicated model architecture (such as the classification decoder) and loss function (such as the weakly supervised classification loss) in the second scheme are tailored to the characteristics of the data (regional proportion labels) generated by the first scheme, which enables the model to learn more directly and effectively from imprecise supervision signals. Compared with using a general model, it can achieve better detection and classification accuracy under the same data.

[0025] 3. A scalable technical framework has been formed: This joint solution is not only applicable to specific technical platforms (such as 10XVisium, Xenium), but its paradigm of "automatically generating weakly supervised data + dedicated weakly supervised learning model" can provide a reference for other tasks that need to use weak labels for fine-grained image analysis.

[0026] 4. It provides valuable pre-trained models for downstream tasks: The large-scale dataset constructed by the first scheme and the model trained by the second scheme can serve as a powerful pre-training foundation. Through fine-tuning, it can be quickly adapted to various supervised downstream pathological analysis tasks, significantly improving performance in small sample scenarios (as shown in Example 6). Attached Figure Description

[0027] Figure 1 This is an automated flowchart for constructing weakly supervised and semi-supervised datasets of pathological images in Embodiment 1 of the present invention, illustrating the entire process from data acquisition, preprocessing, deconvolution annotation, cell nucleus segmentation to data integration and storage.

[0028] Figure 2 This is an example image of cell membrane boundaries in Xenium technology platform data, demonstrating subcellular level detection accuracy.

[0029] Figure 3 The example output image for the cell nucleus segmentation tool shows the results of cell nucleus boundary recognition.

[0030] Figure 4 This is a flowchart of the data processing on the Xenium technology platform, illustrating the process of generating a gene expression matrix at the cell level from raw data.

[0031] Figure 5 This is a diagram showing the overall structure of the cell nucleus detection and classification model trained on weakly supervised data in Embodiment 2 of the present invention, illustrating the complete process from input image to detection and classification output.

[0032] Figure 6This is a schematic diagram illustrating the structure and working mechanism of the decoder part of the model in this invention, with a focus on the restricted deformable cross attention mechanism introduced in the weakdecoder.

[0033] Figure 7 This is a schematic diagram of the internal structure of a restricted deformable cross attention layer, illustrating how the attention sampling points are restricted to the detection box using the tanh activation function.

[0034] Figure 8 This is a graph showing the trend of the model's weak loss decreasing with the number of training epochs on the training set.

[0035] Figure 9 This is a graph showing the trend of the model's weak loss decreasing with the number of training epochs on the validation set.

[0036] Figure 10 This is a visualization of restricted deformable attention on a cell nucleus image, showing how the attention is focused on the characteristic regions at the cell nucleus boundary and inside.

[0037] Figures 11a-11f This is a graph showing the trend of F1 scores of various cells changing with the number of training rounds during supervised fine-tuning after weakly supervised pre-training. Detailed Implementation

[0038] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0040] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0041] Example 1: A method for automatically constructing pathological image datasets based on spatial transcriptomics technology

[0042] This embodiment details a complete workflow for automatically constructing a weakly supervised or semi-supervised pathological image dataset that can be used to train cell nucleus detection and classification models using publicly available spatial transcriptome data. The workflow follows a paradigm of "data acquisition → preprocessing → annotation generation → image segmentation → data integration," with the specific steps as follows:

[0043] S1: Data Acquisition

[0044] Download the pre-organized spatial transcriptomics dataset HEST-1K from a public data platform (Hugging Face is used in this example). This dataset integrates data from 131 public resources, covering 26 human organs and 25 cancer types, totaling 367 samples. The dataset also provides matched full-view digital pathology slides (WSI) and gene expression data, and includes data from multiple spatial transcriptomics technology platforms such as 10X Visium and Xenium, providing a data foundation for the automated construction of large-scale training sets.

[0045] S2: Data Preprocessing

[0046] The raw data was preprocessed using the official toolkit HEST-Library provided by the HEST-1K dataset. The core objective was to standardize the data scale and generate intermediate files that would facilitate subsequent processing.

[0047] 1. Pathological image standardization: The WSI resolution of all samples was standardized to 40x magnification (corresponding to 0.25 micrometers / pixel).

[0048] 2. Data processing for different platforms:

[0049] For the 10X Visium platform: Each sample's WSI is cropped into 256×256 pixel image patches. An H5 file is generated for each sample, containing three datasets:

[0050] "barcode": A unique identifier sequence used for storage and corresponding one-to-one with the sequencing region. Example: "AAACAGAGCGACTCCT-1".

[0051] “coords”: Used to record the coordinates of the center of each image patch on WSI.

[0052] "img" is used to store the pixel values ​​of the obtained image block.

[0053] For Xenium data: Because this technology uses immunofluorescence, the detection area reaches the subcellular level, allowing for the location of each detected gene and precise acquisition of cell membrane boundaries (greater than the cell nucleus). Data examples are shown below. Figure 2 , Figure 3 As shown, the cell membrane and nuclear boundaries are illustrated. At this point, it's sufficient to combine the gene statistics within the same cell to obtain a cell gene expression matrix (dimension: number of cells multiplied by number of genes), and then store it in an H5ad file for later use. Figure 4 The following is a flowchart of the process.

[0054] S3: Cell type annotation and quality control based on deconvolution

[0055] Cell type proportion labels at the spot (55-micrometer circular region) level and cell type labels at the single cell nucleus level were obtained from gene expression data of two different technology platforms using deconvolution methods, and data quality control was performed. The specific process is as follows:

[0056] a. Generating weakly supervised labels for 10X Visium data: The gene expression matrix from the H5ad file was read, and the deconvolution tools RCTD and cell2location were used. Combined with the single-cell transcriptome reference dataset of the corresponding tissue, the proportion of each cell type within each spot (a circular region approximately 55 micrometers in diameter) was calculated. To improve label quality, the cosine similarity of the proportion vectors obtained from the two methods was calculated: Only Spots with a similarity greater than 0.9 are retained as high-confidence weak supervision labels.

[0057] b. Generate semi-supervised labels for Xenium data: Use the single-cell annotation tool tacco to annotate the single-cell gene expression matrix obtained in S2, and add a cell_type column to the .obs attribute of the H5ad file.

[0058] Subsequently, quality control is performed: Due to the limitations of this spatial transcriptomics technology, some cells may not have any genes detected or may have very few genes detected. There are a large number of cells with less than 50% gene detection, which will affect the accuracy of cell annotation. Therefore, additional quality control is required. This invention filters out cells with less than 50% of the genes detected and / or less than 100 total gene expression for quality control.

[0059] S4: Cell Nucleus Instance Segmentation

[0060] Cell nucleus segmentation was performed on pathological images using a pre-trained CellViT model (weight file: CellViT-256-x40.pth).

[0061] For 10X Visium, inference is performed directly on 256x256 image blocks.

[0062] For the entire WSI corresponding to Xenium, the WSI inference mode is used to obtain the full image segmentation result.

[0063] The segmentation results (including polygon boundaries and bounding boxes) are saved in GeoJSON format, which makes it easy to crop WSI according to coordinates to form different types of datasets (different image patch sizes, different locations, different cell types, etc.).

[0064] S5: Data Integration and Structured Storage

[0065] The image patches, cell type labels, and cell nucleus segmentation information obtained in the above steps are aligned and integrated. For Xenium data, to ensure consistency with the Visium data format, a tool is used to crop the WSI images into 256x256 image patches sequentially (from top to bottom, from left to right). Finally, the data for each sample is integrated into an H5 file, the structure of which includes:

[0066] img: Stores image block data.

[0067] annotation: Store weakly supervised labels (10X Visium) or leave blank (Xenium semi-supervised data).

[0068] mask: Stores the instance segmentation mask corresponding to the image patch.

[0069] patch_instance: A single-channel mask where different pixel values ​​are used to distinguish and index different cell nucleus instances in the image patch.

[0070] sample_barcode: The associated spatial barcode identifier.

[0071] Through the process described in this embodiment, two types of datasets are ultimately constructed automatically: a weakly supervised dataset based on 10X Visium data (each image patch has region-level cell type proportion labels), and a semi-supervised dataset based on Xenium data (some cells in each image patch have precise single-cell type labels). These two datasets provide large-scale, high-quality training resources for the subsequent development of cell nucleus detection and classification models suitable for weak / semi-supervised learning.

[0072] Example 2: A method for constructing and training a cell nucleus detection and classification model based on a weakly supervised dataset.

[0073] This embodiment describes training a deep learning model for cell nucleus detection and classification using the weakly supervised dataset constructed in Embodiment 1. Figure 5 As shown, the model building and training steps are as follows:

[0074] A: Build a dataset loading module to read pathological image blocks and their corresponding annotations.

[0075] This module uses a custom Dataset class built on the PyTorch framework to read the structured H5 format dataset generated in Example 1. Its core functionalities include:

[0076] Data reading: Load img (image patch), annotation (weakly supervised proportional label vector), mask (instance segmentation mask), and patch_instance (instance index mask) from the H5 file.

[0077] Data encapsulation: The output follows the COCO format and includes an image tensor, the corresponding weakly supervised label (a C-dimensional vector, where C is the number of cell types), and a detection box for each cell nucleus instance.

[0078] Image augmentation: During training, image patches are normalized by specifying uniform mean and variance for each color channel. (In this implementation, mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]). This embodiment only performs normalization and does not use other spatial augmentations to avoid disrupting the correspondence between spatial labels.

[0079] B: Construction of Feature Extraction Network

[0080] The feature extraction network consists of two parts: a backbone network and an encoder. It is responsible for extracting multi-level, global feature representations from the input image.

[0081] The overall structure of the model is as follows Figure 5 Details such as Figure 6 , 7 .

[0082] 1. Backbone: Employs a ResNet-50 model pre-trained on ImageNet. Inputting a 256×256 image patch, the Backbone outputs feature maps for the final four stages, with spatial resolutions of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, each with 256 channels. These multi-scale features provide the foundation for subsequent processing of cell nuclei of different sizes.

[0083] 2. Encoder: The encoder consists of N_e layers (6 layers in this embodiment) stacked together. The core of each layer is a multi-scale deformable attention module (MSDeformAttn), whose calculation formula is:

[0084]

[0085] This module takes a learnable query or the output features of the previous layer as input, and performs adaptive sampling and feature aggregation on multi-scale backbone features to efficiently fuse global context information. In this embodiment, the total length of the feature sequence input to the Encoder (i.e., the sum of the number of pixels after flattening all scale feature maps) is 5440.

[0086] C. Construction of the Detection Decoder

[0087] The detector decoder is responsible for decoding the spatial location (bounding box) of each cell nucleus instance from the global features output by the encoder. In this embodiment, the number of layers N_d in the detector decoder is set to 3.

[0088] Core modules: see Figure 6 At the heart of each decoder layer is a multi-scale deformable cross-attention module. Its mathematical form is the same as the multi-scale deformable attention (MSDeformAttn) in the encoder, but there is a key difference: in the decoder, the query comes from the output of the previous layer (or a learnable object query), while the key / value pairs originate from the final output features of the encoder. This design allows the decoder to focus on the regions of encoded features most relevant to a specific detection target.

[0089] Reference point mechanism: The decoder uses a series of reference points as anchors for prediction. The initial reference point can be set as the image center or generated from learnable parameters. During training, an "Iterative Bounding Box Refinement" strategy can be adopted, where the center coordinates of the bounding box predicted by the current layer can be used as reference points for the next layer's decoding, thereby gradually optimizing the localization.

[0090] Output: After decoding through N_d layers, the model outputs N prediction results (N is the preset maximum number of targets, usually greater than the expected number of cell nuclei in the image). Each prediction result contains two parts:

[0091] 1. A bounding box: represented as a 4-dimensional vector [x, y, w, h], where Let w be the coordinates of the center point, and h be the width and height.

[0092] 2. A binary logits: representing the confidence score of whether the predicted box is "foreground" (i.e., cell nucleus) or "background".

[0093] 3. Integration with the classification decoder: The bounding box information predicted by the detection decoder (especially the center coordinates and width and height) will be used to initialize the query and position encoding of the classification decoder, thereby decoupling the detection task from the classification task and enabling information transfer.

[0094] D. Construction of the Weak Decoder

[0095] The classification decoder is the core innovative component of this invention, specifically designed to learn cell nucleus category information from weakly supervised labels. Its function is to utilize the localization information provided by the detection decoder to extract local features of each cell nucleus instance from the global features output by the encoder, and predict its category probability distribution. In this embodiment, the number of layers N_w in the classification decoder is set to 3.

[0096] 1. Structure Overview and Initialization: The classification decoder consists of N_w layers stacked together. Its query initialization depends on the output of the detection decoder. Specifically, the width and height (w, h) of the bounding box predicted by the detection decoder are mapped to the initial query vector through a linear layer; simultaneously, the center coordinates of the bounding box are... It is then converted into a corresponding positional embedding through another linear layer, which is used to provide spatial location information in the attention mechanism.

[0097] 2. Restricted Deformable Cross-Attention Module: The key innovation of the classification decoder lies in its first layer, which employs a Restricted Deformable Cross-Attention module. This module represents a significant improvement over the standard deformable cross-attention module:

[0098] a. Standard procedure: The module predicts a set of sampling offsets based on the query. And the corresponding attention weights.

[0099] b. Restriction mechanism: By introducing the tanh activation function The predicted sampling offset is strictly limited to the (-1,1) interval.

[0100] c. In-frame sampling: The final sampling point coordinates are calculated using the following formula after limiting the offset:

[0101] ;

[0102] in, The predicted offset. The center coordinates of the bounding box are: Let be the width and height of the bounding box. This calculation ensures that all sampling points fall within the currently predicted cell nucleus bounding box, forcing the model to focus on the morphology and internal features of the instance itself, thereby enhancing the ability to extract instance-level representations.

[0103] 3. Hierarchical Connections and Subsequent Layers: The output features of the first layer (Restricted Deformable Cross-Attention) module are passed to subsequent decoding layers (corresponding to...) through residual connections and other methods. Figure 6 The structure of subsequent layers is similar to that of the detector-decoder layers, but the cross-attention module is also replaced with a (Restricted Deformable Cross-Attention) module to maintain focus on local features of the instance.

[0104] 4. Output: After being processed by the N_w layer classification decoder, the output features are transformed into the class probability distribution of each predicted instance on C cell types through a linear layer and a Softmax activation function.

[0105] In the above scheme, the classification decoder, through a "detection-guided classification" strategy, uses detection box information to constrain the attention range, enabling it to effectively infer the category information of each specific cell nucleus instance from weakly supervised, image patch-level proportional labels. Together with the detection decoder, it forms a decoupled "detection-classification" dual-branch architecture, which is key to the invention's ability to perform efficient training using automatically generated labels.

[0106] Furthermore, the core purpose of the constraint mechanism in the above scheme is to force the model's "attention" to focus on the bounding box of the currently predicted cell nucleus instance, actively shielding it from irrelevant tissue background and other cellular interference outside the box. This enables the classification decoder to extract purer and more discriminative instance-level morphological and texture features, thereby significantly improving the accuracy of fine-grained cell nucleus classification under weak supervision.

[0107] E. Loss Function and Model Training

[0108] To enable effective training of the model on weakly supervised datasets, this embodiment designs a joint loss function and corresponding training strategy to ensure that the model can simultaneously learn accurate cell nucleus localization and proportional label-based cell nucleus classification.

[0109] 1. Prediction-True Value Matching (Hungarian Matching)

[0110] Before calculating the loss, the Hungarian matching algorithm is first used to establish the correspondence between the model predictions and the ground truth annotations. This algorithm performs optimal bipartite graph matching between the N predictions (including bounding boxes and foreground / background scores) output by the detector decoder and the M ground truth cell nucleus instances (composed of bounding boxes transformed from segmentation masks) in the image patch, ensuring that each ground truth instance is assigned to the most suitable prediction.

[0111] In some specific implementations, the weights of each cost are set as follows: the classification confidence weight (cost_class) is 2, the bounding box L1 loss weight (cost_bbox) is 5, and the GIoU loss weight (cost_giou) is 2.

[0112] 2. Detecting Losses

[0113] For each successfully matched prediction, calculate the detection loss. The loss is a weighted sum of the following three components:

[0114] Focal loss: This is used to resolve class imbalance issues that may exist between the foreground (cell nucleus) and the background. The formula is: in This represents the confidence score output of the model for a certain category.

[0115] In some specific implementation schemes, the weighting coefficient Set it to 1.0, and set the hyperparameter focal_alpha to 0.25.

[0116] L1 loss: Calculates the absolute error between the predicted bounding box coordinates and the true bounding box coordinates, used for basic location regression. Its formula is: ,in, Indicates the model output, Indicates a label.

[0117] In the specific implementation plan, the L1 loss weighting coefficient Set it to 5.0.

[0118] GIoU loss: Calculates the generalized intersection-union (GU) loss between the predicted and ground truth bounding boxes, providing a better measure of the overlap and relative position of the boxes. Its formula is: Where A and B are the predicted box and the ground truth box, respectively, and C represents the smallest enclosing box, which is the smallest rectangle that can simultaneously contain both A and B. Represents the area of ​​a region (or the size of a set of points). and Intersection and Union Set subtraction (difference).

[0119] In some specific implementations, the GIoU loss weighting coefficient Set it to 2.0.

[0120] In this scheme, the total detection loss is: .

[0121] 3. Loss in weakly supervised classification

[0122] This is crucial for implementing weakly supervised learning. For each matched prediction item classified as foreground, its predicted class probability distributions are summed and averaged to obtain the aggregated prediction distribution q for the entire image patch. The KL divergence (Kullback-Leibler Divergence) between this distribution and the image patch-level weakly supervised proportional labels p (i.e., annotations in the H5 file) provided by the dataset is calculated as the classification loss, with the specific formula as follows: ,

[0123] in, For label distribution, To predict the distribution, For dimensional indexing.

[0124] This loss function establishes a mathematical connection between the regional proportional supervision signal and the model's sum of class predictions for all instances, guiding the model to learn the correct classification ability in the absence of instance-level labels.

[0125] In some specific implementation schemes, the weighting coefficient of the weakly supervised classification loss Set it to 1.0.

[0126] 4. Model Training Configuration

[0127] The model training process is guided by optimizing both the detection loss and the weakly supervised classification loss. During training, gradient descent-based optimization algorithms (such as AdamW) are used to iteratively update the model parameters, continuously reducing the overall loss target, ultimately enabling the model to automatically detect and classify cell nuclei from pathological images.

[0128] Example 3: Model Training and Validation on Weakly Supervised Datasets

[0129] This embodiment details how to apply the weakly supervised pathological image dataset constructed in Embodiment 1, based on the cell nucleus detection and classification model architecture and training method described in Embodiment 2, to perform specific model training and preliminary performance verification. This embodiment aims to demonstrate the complete application chain of the described data construction process and model method.

[0130] S1: Set up the test dataset

[0131] A subset of data from lung and kidney tissue samples in the weakly supervised dataset constructed using the automated process in Example 1 was selected for training and validation in this example. This subset contains approximately 31,000 pathological image patches of 256×256 pixels, covering 10 predefined cell types. Before training began, this dataset was randomly divided into training and validation sets in an approximately 8:2 ratio.

[0132] S2: Model Instantiation and Parameter Configuration

[0133] Based on the model architecture described in Example 2, a specific cell nucleus detection and classification model is instantiated, and the following parameters are configured:

[0134] Backbone network: ResNet-50 pre-trained on ImageNet.

[0135] Number of encoder layers: Set the number of encoder layers N_e to 6.

[0136] Number of decoder layers: Since the detection task on pathological images is relatively simple, the number of layers can be reduced. Set the number of layers of the detection decoder to N_d as 3;

[0137] Set the number of layers N_w in the classification decoder to 3 (including the first layer, Defomable crossattention).

[0138] Task header settings: The detection branch outputs 2 categories (foreground / background); the classification branch outputs 10 categories (corresponding to 10 cell types).

[0139] Matcher cost weights: In the Hungarian matcher, the cost weight of classification confidence (cost_class) is set to 2, the cost weight of bounding box L1 loss (cost_bbox) is set to 5, and the cost weight of bounding box GIoU loss (cost_giou) is set to 2.

[0140] S3: Training Process Implementation

[0141] Train the model using the prepared training and validation sets. The key configurations and processes for training are as follows:

[0142] 1. Optimizer settings: Learning rate: 1e-4; Backbone learning rate: 1e-5; Optimizer: AdamW; Learning rate decay: cosine; Decay rate: 1e-4; Number of training epochs: 20.

[0143] 2. Loss function weights: The total training loss is the detection loss defined in Example 2. Loss of weakly supervised classification Together they constitute the whole. The weighting coefficients for each loss are set as follows:

[0144] The Focal Loss weight (class_loss_coef) in the detection branch is 1.0.

[0145] The bounding box L1 loss weight (bbox_loss_coef) is 5.0.

[0146] The bounding box GIoU loss weight (giou_loss_coef) is 2.0.

[0147] The weight of the weakly supervised classification loss (weak_class_coef) is 1.0.

[0148] 3. Training hyperparameters: The hyperparameter (focal_alpha) is set to 0.25, the batch size is set to 8, and a total of 20 training rounds are performed.

[0149] S4. Training Monitoring: Records the model training process. In this embodiment, the loss value at each step of the model training process is recorded in real time using the Wandb tool and displayed as an average over 1000 steps. Figure 8 and Figure 9 As shown, as training progresses, the main losses (such as weakly supervised classification loss) on both the training and validation sets show a steady decreasing trend and gradually converge, indicating that the model can effectively learn on the weakly supervised dataset provided in Example 1 without overfitting, thus verifying the feasibility of the training method described in Example 2.

[0150] S5: Model Saving and Application

[0151] After training, the model parameters from the last training epoch are saved. This model is capable of detecting cell nuclei in pathological images and predicting the probability of them belonging to 10 predefined types. This model can be used as a base model for subsequent cell nucleus analysis studies, or, as shown in other embodiments, for fine-tuning in downstream tasks.

[0152] Through the specific implementation of this embodiment, it is demonstrated that the automated data construction process of Embodiment 1 can produce a large-scale weakly supervised dataset that can be used for training. At the same time, it is verified that the dedicated model architecture and training method proposed in Embodiment 2 can successfully learn using such data, realizing a complete technical closed loop from "automated data construction" to "dedicated model training".

[0153] Example 4: Independent Testing and Interpretability Verification of Weakly Supervised Training Models

[0154] This embodiment aims to conduct independent performance testing and internal mechanism interpretability analysis on the weakly supervised dataset constructed through Embodiment 1 and the cell nucleus detection and classification model trained by the methods described in Embodiments 2 and 3, in order to verify its basic capabilities.

[0155] S1: Test Environment and Data Preparation

[0156] To objectively evaluate model performance, the fold3 portion of the Pannuke dataset, a publicly available benchmark dataset independent of the training data and featuring authoritative, finely annotated data, was selected as the test set. This dataset contains five main cell nucleus categories: neoplastic, non-tumor epithelial, inflammatory, connective / stromal, and necrosis.

[0157] S2: Model Loading and Configuration

[0158] Construct a model architecture completely identical to that of Example 2. Load the model parameters from the 20th training epoch, saved after training in Example 3, into this model. Since this example used 10 subdivided cell types during training, while the Pannuke dataset has 5 major categories, the 10 probability distributions predicted by the model need to be mapped and integrated into the above 5 major categories based on biological prior knowledge before evaluation.

[0159] S3: Performance Evaluation Process and Results

[0160] 1. Inference and Matching: Input the test set images into the loaded model to obtain the predicted bounding box and its class probability distribution for each cell nucleus instance. Using the Hungarian matching algorithm defined in step E of Example 2, optimal matching is performed between the instances predicted by the model and the manually labeled real instances in the test set.

[0161] 2. Performance Metric Calculation: For successfully matched predictions, calculate their performance metrics on both the detection and classification tasks. The detection task primarily evaluates localization accuracy, while the classification task evaluates the correctness of type identification. Common evaluation metrics include precision (P), recall (R), and F1 score (F).

[0162] 3. Evaluation Results: As shown in Table 2, the model achieved preliminary but effective performance on the Pannuke fold3 test set. In detection, the model demonstrated strong localization capabilities. In classification, despite the training signal being a weakly supervised proportional label, the model was still able to distinguish key categories such as neoplasms to a certain extent. These results indicate that the model trained using the method described in this invention can effectively learn cell nucleus detection and classification knowledge from automatically generated weakly supervised data.

[0163] S4: Model interpretability verification

[0164] To verify the rationality and effectiveness of the Restricted Deformable Cross-Attention module introduced in the Weak Decoder in this invention, this embodiment provides a visual analysis of the model's attention mechanism.

[0165] Attention extraction: When the model performs inference on the test image, record the attention weights and corresponding sampling point coordinates generated by the restricted deformable cross attention module in the classification decoder.

[0166] Visualization analysis: Sampling points with attention weights higher than a threshold (e.g., 0.2) are selected, and their coordinates are mapped back to the original pathological image for visualization. Figure 10 As shown.

[0167] Results Analysis: Visualization results show that the high-weighted attention sampling points are not randomly distributed, but densely concentrated on the boundary contours of the cell nucleus and its discriminative internal regions. This phenomenon intuitively demonstrates the effectiveness of the restricted deformable cross-attention module's design, which uses the tanh activation function to limit sampling offsets to the detection box. It successfully guides the model to focus its "attention" on the morphological and textural features of individual cell nuclei, rather than irrelevant regions of the entire image. This not only enhances the model's ability to extract instance-level discriminative features but also makes the model's decision-making process more interpretable, aligning with the intuition of pathological analysis focusing on the morphology of individual cells.

[0168] This embodiment verifies the effectiveness and interpretability of the model trained based on the automated process of this invention through independent benchmarking and visualization of the internal mechanisms.

[0169] Example 5: Performance Validation of the Model under Supervised Training Paradigm

[0170] This embodiment aims to verify the performance ceiling of the model architecture constructed in Embodiment 2 when sufficient and accurate manual annotations are obtained (i.e., supervised learning), and to explore its advantages after the "detection-classification" decoupling and attention constraint improvement.

[0171] S1: Experimental Setup and Data Division

[0172] This experiment uses the fully supervised Pannuke dataset. The samples in this dataset are pre-divided into three parts: fold1, fold2, and fold3. This embodiment follows the standard usage conventions of this dataset: fold1 is used as the training set, fold2 as the validation set for tuning hyperparameters and monitoring the training process, and fold3 as the final test set to evaluate model performance. The model needs to identify the five cell nucleus categories defined by Pannuke.

[0173] S2: Model Tuning and Training

[0174] 1. Architecture Adjustment: The model architecture based on Example 2 is modified for adaptability. The main changes include: setting the number of categories output by the Weak Decoder to 5; and replacing the weakly supervised classification loss (KL divergence loss) in the loss function with a standard classification loss suitable for supervised learning (focal loss is used in this example).

[0175] 2. Training Parameters: The optimizer was set to AdamW. The initial learning rate was set to 3e-4, with a lower learning rate of 3e-5 used for the backbone network for fine-tuning. Training was conducted for a total of 100 epochs, with other training details (such as batch size) remaining consistent with Example 3.

[0176] 3. Training objective: The goal of this experiment is to verify the capabilities of the model architecture itself. Therefore, no pre-trained weights are used, and all model parameters are randomly initialized and trained from scratch.

[0177] S3: Performance Evaluation and Results Analysis

[0178] After training, the parameters of the model that performed best on the validation set are loaded, and a comprehensive evaluation is performed on the independent fold3 test set. The evaluation process is the same as in Example 4, S3.

[0179] The test results are shown in Table 3. Compared with the model performance trained under weak supervision in Example 4 (Table 2), the model in this example shows a significant improvement in all metrics. For example, the F1 score for the detection task improved from 0.74 to 0.845, and the F1 score for the classification of neoplasm cells improved from 0.48 to 0.718. This fully demonstrates that the model architecture proposed in this invention (including a multi-scale deformable attention encoder, a decoupled detection and classification decoder, and a restricted deformable cross-attention module) is a powerful and efficient framework. When equipped with accurate instance-level labels, this architecture can fully realize its performance potential and achieve excellent performance in cell nucleus detection and classification tasks. In particular, the detection precision metric is outstanding, reflecting the accuracy of model localization, and indirectly verifying that the "detection-classification" decoupling design helps reduce interference between tasks.

[0180] The results of this embodiment show that the model architecture provided by the present invention is not only suitable for weakly supervised learning, but also compatible with and adept at fully supervised learning, and has good versatility and superior performance.

[0181] Example 6: Verification of the facilitating effect of weakly supervised pre-trained models on supervised fine-tuning

[0182] This embodiment aims to verify a key application value: whether the weakly supervised pre-trained model generated by the automated process described in Embodiments 1 to 3 can serve as an efficient initial parameter to significantly improve the training speed and final performance of the model on downstream supervised tasks with limited data.

[0183] S1: Experimental Design and Model Initialization

[0184] 1. Downstream Task Setting: Select a supervised cell nucleus classification task as the downstream scenario. Refer to the setup in Example 5, using the Pannuke dataset and employing the same partitioning method.

[0185] 2. Key Initialization Strategy: Construct a model architecture identical to that in Example 5 (a 5-class supervised model). Its parameter initialization employs a transfer learning strategy:

[0186] First, load all parameters of the weakly supervised pre-trained model from the 20th round of training saved in Example 3.

[0187] Then, since the classification head of the pre-trained model is designed for 10 subcategories, while the downstream task has 5 categories, the weights of the final linear classification layer of the classification decoder in the pre-trained model are discarded.

[0188] Finally, this new 5-class linear layer is randomly initialized, while all other parts of the model (including the backbone network, encoder, decoder, etc.) inherit parameters from the weakly supervised pre-trained model.

[0189] S2: Fine-tuning the training process

[0190] The Pannuke dataset partitioning described in Example 5 (i.e., fold1 as the training set) was used to perform supervised fine-tuning on the initialized model. The hyperparameter settings during training, such as the optimizer, learning rate, and number of epochs, remained consistent with the training settings for the weakly supervised model in Example 3. The fine-tuning process consisted of only 20 epochs.

[0191] S3: Evaluation and Analysis of Fine-Tuning Effects

[0192] 1. Performance Comparison: After 20 rounds of fine-tuning, the model was evaluated on the fold3 test set of the Pannuke dataset (the same test set used in Examples 4 and 5), and the results are shown in Table 4. Comparing Table 4 (pre-training + 20 rounds of fine-tuning) with Table 3 (Example 5, 100 rounds of supervised training from scratch), it can be seen that the model pre-trained with weak supervision, after only 20 rounds of fine-tuning, has reached or even surpassed the performance of the model trained from scratch for 100 rounds in most metrics. For example, the detection F1 score reached 0.863 (vs. 0.845), and the epithelial cell classification F1 score reached 0.736 (vs. 0.660). This demonstrates that the pre-trained model provides highly valuable initial feature representations.

[0193] 2. Convergence rate analysis: Figure 11a~f shows the curves of F1 scores for each category changing with training epochs during the fine-tuning process. It can be observed that the performance of all categories rapidly increases and stabilizes within 5 epochs after the start of fine-tuning. This indicates that the model does not learn from scratch, but rather rapidly adapts to the high-quality features obtained during pre-training, greatly accelerating the convergence process.

[0194] S4: Conclusions and Application Significance

[0195] This embodiment verifies one of the core values ​​of the "automatic construction of weakly supervised datasets → training of specialized models" technology loop provided by this invention: producing a general, highly representative pathological image pre-trained model. This pre-trained model learns universal cell nucleus morphology, texture, and contextual features through massive amounts of weakly supervised data. When applied to downstream specific supervised tasks where labeled data may be limited (as shown in this embodiment), it can significantly reduce the demand for labeled data; significantly accelerate model training convergence speed; and effectively improve the model's final performance ceiling.

[0196] In summary, this embodiment verifies one of the core values ​​of the "automatic construction of weakly supervised datasets" → "training dedicated models" technology closed loop provided by this invention: producing a general, highly representative pathological image pre-trained model. This pre-trained model learns universal cell nuclear morphology, texture, and contextual features through massive amounts of weakly supervised data. When applied to downstream specific supervised tasks where labeled data may be limited, it can: 1. significantly reduce the need for labeled data; 2. significantly accelerate model training convergence speed; and 3. effectively improve the model's final performance ceiling. This provides a strong foundation for rapidly developing various pathological image analysis tools for specific organs, diseases, or research purposes, fully demonstrating the important practical significance of this invention's method in promoting AI-assisted pathological diagnosis research and application.

[0197] Table 1: Comparison of detection and classification performance metrics of supervised training models on the Pannuke dataset.

[0198]

[0199] Table 2: Performance metrics of the model on the Pannuke dataset after weakly supervised training.

[0200]

[0201] Table 3: Performance metrics of the model on the Pannuke dataset after supervised training.

[0202]

[0203] Table 4: Performance metrics of the model on the Pannuke dataset after supervised fine-tuning following weakly supervised pre-training.

[0204]

[0205] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatically constructing a pathological image dataset based on spatial transcriptomics technology, characterized in that, Includes the following steps: S1: Obtain a publicly available spatial transcriptome dataset of pathological images and gene expression matching as the raw data; S2: The raw data is preprocessed to standardize the scale of pathological images and generate a first intermediate file containing image blocks and their spatial coordinates according to different spatial transcriptomics technology platforms. Specifically, for 10X Visium platform data, the preprocessing includes standardizing the full-resolution pathological images to a uniform magnification and cutting them into image blocks of fixed size, generating an H5 file containing the pixel values ​​of the image blocks, their corresponding spatial coordinates, and unique identifiers. For Xenium platform data, the preprocessing includes statistically merging gene expression data within the same cell to generate a single-cell level gene expression matrix file. S3: Using deconvolution tools and combined with single-cell transcriptome reference data, cell type annotation is performed on the gene expression data obtained in step S2 to generate cell type ratio labels or single-cell type labels, and the annotation results are quality controlled; among them, for 10X Visium platform data, high-confidence regions are screened by calculating the cosine similarity between the results of different deconvolution methods. The cosine similarity calculation formula is: in, and These are the cell proportion vectors obtained by the two deconvolution methods, respectively. For Xenium platform data, cell type annotation of gene expression data using deconvolution tools specifically includes: Obtain an H5ad file containing a single-cell gene expression matrix; The H5ad file is annotated with cell type at the single-cell level using an annotation tool that can combine single-cell data of the corresponding tissue, and a cell_type column is added to the observation data of the H5ad file to record the annotation results of each cell; Quality control is performed on the annotation results, including filtering out less than 50% of the gene types detected in the cell regions and / or less than 100 total gene expression numbers. The annotation results of the completed quality control were used as semi-supervised labels for the Xenium platform data. S4: Using a pre-trained cell nucleus segmentation tool, perform cell nucleus instance segmentation on the pathological image processed in step S2 to obtain cell nucleus boundary and bounding box information; S5: Integrate the cell type labels generated in step S3 with the cell nuclear boundary information obtained in step S4, and store them as a structured data file for model training.

2. The method for automatically constructing pathological image datasets based on spatial transcriptomics technology according to claim 1, characterized in that: In step S5, the dataset is stored in H5 format, including image patches, semi-supervised annotations, segmentation masks, instance indexes, and sample identifiers.

3. A method for training cell nucleus detection and classification based on a dataset constructed according to claim 1 or 2, characterized in that: include: A: Construct a dataset loading module to read pathological image patches and their corresponding annotations; B: Construct a feature extraction network, including a backbone for extracting multi-scale features and an encoder for further feature extraction; C: Construct a detection decoder to predict the detection box of the cell nucleus; D: Construct a classification decoder to predict the class distribution of cell nuclei; wherein, the classification decoder employs a restricted deformable cross-attention module, and the sampling offset is restricted to the detection box by the tanh activation function, the calculation formula of which is: ； in, The predicted offset. The center coordinates of the bounding box are: The width and height of the bounding box; E: Construct the loss function, including detection loss and weakly supervised classification loss, for model training; where the detection loss includes focus loss, L1 loss, and GIoU loss, and the total detection loss is: The weakly supervised classification loss uses KL divergence loss: in, For label distribution, To predict the distribution, For dimensional indexes, calculations are performed on... Perform small value smoothing to avoid logarithmic inconsistencies.

4. The method for detecting and classifying training cell nuclei according to claim 3, characterized in that: The Encoder in the feature extraction network employs a multi-scale deformable attention module, and its calculation formula is as follows: ； in, To obtain the sampling offset point through linear layer calculation, For attention weights, and satisfying Reference point After normalization Add offset later Then, bilinear interpolation sampling is performed in the representation. and These are the learnable model weights, where m is the attention head. q is the index of the scale level, q is the reference point, and k is the number of sampling points.

Images