Tumor cell infiltration depth prediction method and system based on pathological image reconstruction

Abstract

The invention discloses a tumor cell infiltration depth prediction method and system based on pathological image reconstruction. The method comprises the following steps: marking a shallow region and a deep region on a full pathological image, and constructing a spatial relation graph; training an image segmentation model and a spatial relationship extraction model by using the shallow region, the deep region and the spatial relationship graph; obtaining a shallow pathology image and a deep pathology image through an image segmentation model, and extracting a shallow pathology image space relation graph through a space relation extraction model; taking the superficial layer pathological image and the superficial layer pathological image space relation graph as input, taking the deep layer pathological image as output, and training to obtain a trained information reconstruction model; inputting the superficial layer pathological image into the trained spatial relation extraction model to obtain a superficial layer pathological image spatial relation graph; and inputting the superficial layer pathological image and the spatial relation graph of the superficial layer pathological image into the trained information reconstruction model to obtain a deep layer pathological image, and predicting the tumor cell infiltration depth based on the deep layer pathological image.

Description

Technical Field

[0001] The present invention relates to the technical field of tumor detection, and in particular to a method and system for predicting tumor cell infiltration depth based on pathological image reconstruction. Background Art

[0002] Biopsy is the standard for tumor screening, diagnosis and follow-up. For example, breast, colorectal and lung biopsies are widely used in clinical practice. Biopsy can evaluate the type, location and size of tumors. However, it is limited to the fact that the samples are taken from the shallow human tissue at the biopsy site, and it is difficult to directly evaluate the depth of tumor cell infiltration in this site. Taking the biopsy of the colorectal wall as an example, the colorectal wall is divided into four anatomical layers from shallow to deep: mucosal layer, submucosal layer, muscular layer, and serosa layer. When taking colonoscopic biopsy samples, the colonoscopic biopsy samples are only taken from the mucosal layer of the shallow layer of the colorectal wall. This results in the pathological images made from the colorectal biopsy specimens can only reflect the tumor cell information of the mucosal layer, and cannot reflect the deeper layers, such as: submucosal layer, muscular layer, and serosa layer, so it is impossible to know whether these deep areas are infiltrated by tumor cells. The benign and malignant nature of the tumor is mainly related to the degree of epithelial cell atypia and the depth of infiltration. At the same time, the depth of infiltration of tumor cells is also an important reference for the clinical staging of tumors. Therefore, the depth of invasion is closely related to the choice of subsequent treatment options. Therefore, it is impossible to obtain a complete pathological image of the tumor through biopsy, and therefore it is impossible to accurately assess the depth of invasion of tumor cells, which limits the diagnosis of the malignancy of the tumor before surgery, thus seriously affecting the choice of treatment options. It can be seen that the existing biopsy technology has the problem of inaccurate detection results of the depth of invasion of tumor cells. Summary of the invention

[0003] The present invention provides a method and system for predicting tumor cell infiltration depth based on pathological image reconstruction, so as to solve the problem of inaccurate detection results in existing tumor cell infiltration depth detection.

[0004] In order to achieve the above object, the present invention is implemented by the following technical solutions: In a first aspect, the present invention provides a method for predicting tumor cell infiltration depth based on pathological image reconstruction, comprising: Obtaining a full-length pathological image of the patient, marking a shallow region and a deep region on the full-length pathological image, and marking corresponding cell types and anatomical layers in the shallow region and the deep region, and constructing a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical layer spatial relationship diagram describing the spatial relationship of the anatomical layers; Constructing an image segmentation model, and using the shallow region and the deep region as training sets to train the image segmentation model to obtain a trained image segmentation model; Constructing a spatial relationship extraction model, and using the cell type spatial relationship map and the anatomical level spatial relationship map to train the spatial relationship extraction model to obtain a trained spatial relationship extraction model; Segmenting the full-slice pathology image by a trained image segmentation model to obtain a shallow pathology image and a deep pathology image, and extracting a shallow pathology image spatial relationship graph of the shallow pathology image by a trained spatial relationship extraction model; An information reconstruction model is constructed, and the shallow pathology image and the spatial relationship diagram of the shallow pathology image are used as input and the deep pathology image is used as output to train the information reconstruction model to obtain a trained information reconstruction model; The shallow pathology image obtained by biopsy is input into the trained spatial relationship extraction model to obtain the corresponding shallow pathology image spatial relationship map, the shallow pathology image and the corresponding shallow pathology image spatial relationship map are input into the trained information reconstruction model to obtain the deep pathology image, and the tumor cell infiltration depth is predicted based on the deep pathology image.

[0005] Optionally, marking the shallow layer area and the deep layer area on the pathological full-film image includes: The image of the shallow region and the image of the deep region are extracted from the pathological full-slice image in a direction perpendicular to the anatomical structure of the disease, and the shallow region and the deep region are annotated in the pathological full-slice image by manual annotation.

[0006] Optionally, the marking of corresponding cell types and anatomical levels in the shallow region and the deep region, and constructing a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical level spatial relationship diagram describing the spatial relationship of the anatomical levels, include: The cell types and anatomical layers are manually annotated in the shallow and deep areas respectively. By annotating the spatial relationships between the cell types and the anatomical layers, a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical level spatial relationship diagram describing the spatial relationship of the anatomical layers are constructed. The spatial relationships between the cell types and the anatomical layers include: adjacency, infiltration and inclusion.

[0007] Optionally, the using the shallow region and the deep region as training sets to train the image segmentation model to obtain a trained image segmentation model includes: The image segmentation model is constructed by inputting manually annotated full-length pathological images containing shallow and deep areas as a training set to train the image segmentation model to obtain a trained image segmentation model.

[0008] Optionally, the step of training a spatial relationship extraction model using the cell type spatial relationship graph and the anatomical level spatial relationship graph to obtain a trained spatial relationship extraction model includes: The clustering of the cell types and the anatomical hierarchy are used as nodes to construct a spatial relationship graph of the spatial relationship between nodes, and the pathological image and its corresponding spatial relationship graph are used as data sets to train a spatial relationship extraction model to obtain a trained spatial relationship extraction model.

[0009] Optionally, the training information reconstruction model uses a total cost function weighted by two cost functions: pixel-level reconstruction and clinical-level reconstruction; The difference between the deep pathology image calculated by the pixel-level reconstruction cost function and the real deep pathology image, including pixel, perception and discrimination costs; the difference between the deep pathology image calculated by the clinical-level reconstruction cost function and the real deep pathology image in terms of tumor cell types and quantity levels at the same anatomical level; The cell types and quantity levels at the anatomical level refer to the quantity levels of the same type of tumor cells at the same anatomical level of the deep pathology image. The quantity levels are artificially divided. The clinical-level reconstruction cost function compares the output deep pathology images and the differences in the quantity levels of the same type of tumor cells at the same anatomical level in the real deep pathology images.

[0010] Optionally, the method of training the information reconstruction model using the shallow pathology image and the shallow pathology image spatial relationship map as input and the deep pathology image as output to obtain a trained information reconstruction model includes: Shallow pathology images, shallow pathology image spatial relationship diagrams and deep pathology images are used as data sets to train the information reconstruction model. The shallow pathology images and shallow pathology image spatial relationship diagrams in the data sets are used as input data of the information reconstruction model, and the deep pathology images in the data sets are used as output data of the information reconstruction model.

[0011] Optionally, the predicting the tumor cell infiltration depth based on the deep pathological image includes: Deep pathological images are analyzed manually using image diagnosis or automatic image analysis models to obtain the tumor cell types in deep areas and deep anatomical layers, and to obtain information on the depth of tumor cell infiltration.

[0012] The image analysis model uses a segmentation model or a classification model, which is trained using deep pathology images and manually annotated data on cell types and anatomical levels.

[0013] The image segmentation model segments and counts tumor cells at each layer of deep pathology images to obtain the presence and number of tumor cells in each anatomical layer, thereby determining the depth of infiltration.

[0014] The image classification model classifies the infiltration depth of deep pathology images, inputs deep pathology images, and outputs the predicted label of the anatomical level of the infiltration depth.

[0015] Optionally, knowledge distillation is performed on the spatial relationship extraction model, the information reconstruction model, and the image analysis model to train an infiltration depth prediction model, and the shallow pathology image is input into the trained infiltration depth prediction model to output the anatomical level prediction label of the infiltration depth.

[0016] In a second aspect, an embodiment of the present application provides a tumor cell infiltration depth prediction system based on pathological image reconstruction, including a processor, a memory, and a communication bus; Memory, used to store computer programs; The processor is used to implement any method step described in the first aspect when executing a program stored in the memory.

[0017] Beneficial effects: The tumor cell infiltration depth prediction method based on pathological image reconstruction provided by the present invention uses surgical pathology full-film images with complete tumor information to train an information reconstruction model; the model uses the superficial tumor pathology images obtained by biopsy to reconstruct the deep tumor pathology images that cannot be obtained and are missing, thereby accurately obtaining the infiltration depth of tumor cells. This method will solve the defect that the current biopsy cannot evaluate the infiltration depth of tumor cells, so that the infiltration depth of tumor cells and the degree of tumor malignancy can be obtained during preoperative biopsy, thereby scientifically selecting tumor treatment plans in the early stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is one of the flow charts of the method for predicting tumor cell infiltration depth based on pathological image reconstruction according to the preferred embodiment of the present invention; Figure 2 This is the second flowchart of the method for predicting tumor cell infiltration depth based on pathological image reconstruction according to the preferred embodiment of the present invention; Figure 3 A schematic diagram of the steps of training an image segmentation model provided in a preferred embodiment of the present invention; Figure 4 A schematic diagram of the steps of training a spatial relationship extraction model provided by a preferred embodiment of the present invention; Figure 5 One of the schematic diagrams of the steps of reconstructing a model with training information provided in a preferred embodiment of the present invention; Figure 6 A second schematic diagram of the steps of reconstructing a model based on training information provided in a preferred embodiment of the present invention; Figure 7 A schematic diagram of the steps for estimating the depth of tumor cell infiltration provided in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one" or "one" do not indicate quantity restrictions, but indicate the existence of at least one. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship also changes accordingly.

[0021] See also Figure 1-7 The present application embodiment provides a method for predicting tumor cell infiltration depth based on pathological image reconstruction, comprising: Obtaining a full-length pathological image of the patient, marking a shallow region and a deep region on the full-length pathological image, and marking corresponding cell types and anatomical layers in the shallow region and the deep region, and constructing a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical layer spatial relationship diagram describing the spatial relationship of the anatomical layers; Constructing an image segmentation model, and using the shallow region and the deep region as training sets to train the image segmentation model to obtain a trained image segmentation model; Constructing a spatial relationship extraction model, and using the cell type spatial relationship map and the anatomical level spatial relationship map to train the spatial relationship extraction model to obtain a trained spatial relationship extraction model; Segmenting the full-slice pathology image by a trained image segmentation model to obtain a shallow pathology image and a deep pathology image, and extracting a shallow pathology image spatial relationship graph of the shallow pathology image by a trained spatial relationship extraction model; An information reconstruction model is constructed, and the shallow pathology image and the spatial relationship diagram of the shallow pathology image are used as input and the deep pathology image is used as output to train the information reconstruction model to obtain a trained information reconstruction model; The shallow pathology image obtained by biopsy is input into the trained spatial relationship extraction model to obtain the corresponding shallow pathology image spatial relationship map, the shallow pathology image and the corresponding shallow pathology image spatial relationship map are input into the trained information reconstruction model to obtain the deep pathology image, and the tumor cell infiltration depth is predicted based on the deep pathology image.

[0022] In the above embodiment, a surgical pathology full-length image with complete tumor information is used to train an information reconstruction model; the model uses the superficial tumor pathology images obtained by biopsy to reconstruct the deep tumor pathology images that cannot be obtained and are missing, thereby accurately obtaining the infiltration depth of tumor cells. This method will solve the defect that the current biopsy cannot evaluate the infiltration depth, so that the infiltration depth of tumor cells and the degree of tumor malignancy can be obtained during preoperative biopsy, so as to scientifically select tumor treatment plans in the early stage. The specific method is as follows: Figure 2 shown.

[0023] Optionally, marking the shallow layer area and the deep layer area on the pathological full-film image includes: The image of the shallow region and the image of the deep region are extracted from the pathological full-slice image in a direction perpendicular to the anatomical structure of the disease, and the shallow region and the deep region are annotated in the pathological full-slice image by manual annotation.

[0024] Optionally, the labeling of corresponding cell types and anatomical levels in the shallow region and the deep region includes: The cell types and anatomical layers are manually annotated in the shallow and deep areas respectively.

[0025] Optionally, the using the shallow region and the deep region as training sets to train the image segmentation model to obtain a trained image segmentation model includes: The image segmentation model is constructed by inputting manually annotated full-length pathological images containing shallow and deep areas as a training set to train the image segmentation model to obtain a trained image segmentation model.

[0026] In the above embodiment, the training of the image segmentation model specifically includes: Obtain full-length pathology images Pathological specimens were prepared from the tumor tissue removed during surgery. Pathological diagnosis confirmed the presence of tumor cells in the specimens, and that the tumor cells had infiltrated from the superficial to the deep layers.

[0027] Among them, the superficial layer is the outer anatomical layer of the tumor that can be obtained by biopsy, and the deep layer is a combination of several anatomical layers from shallow to deep along the direction of tumor invasion. It cannot be obtained by biopsy but can be removed during surgery.

[0028] Pathological sections are prepared by staining the specimens with pathological staining techniques to enhance the distinction between tumors and normal cells. The staining method can be HE staining or other immunohistochemical staining techniques.

[0029] By scanning the slices with a 20x or 40x magnification pathology scanner, the full-slice pathology image of the surgical specimen is obtained, and a high-definition full-slice pathology image dataset Data-Full is obtained.

[0030] Tumor cells exist in the superficial layers of all specimens, and the depth of tumor cell infiltration in the deep layers of different specimens varies, that is, the anatomical layers infiltrated are different, reflecting the diversity of the data.

[0031] Training an image segmentation model 1. Take out some surgical pathology full-slide images from Data-Full, manually annotate the image regions and cell types, and obtain the annotated segmentation dataset Data-Prompt, which contains segmentation requirements, region templates and cell templates describing the segmentation results.

[0032] The annotation efficiency is improved by downsampling the 20x or 40x magnified full-length pathology images to 2x magnification.

[0033] The annotation method includes regional annotation, in which the whole pathological film image is annotated into shallow layer, deep layer 1, deep layer 2, deep layer 3, etc., wherein deep layer 1, deep layer 2, deep layer 3, etc. constitute the anatomical region of the depth of tumor infiltration, specifically including outlining the boundaries of the image region where each anatomical layer is located.

[0034] The annotation method includes cell annotation, specifically including the cell type and location of tumor cells and other cells.

[0035] Segmentation requirements refer to the segmentation range and segmentation object. The segmentation range indicates the target area to be segmented, such as the name of the shallow layer or the deep layer; the segmentation object refers to the object to be segmented, such as tumor cells, tumor stromal cells, etc., which are used as prompt words to train the image segmentation model.

[0036] 2. Use the labeled image set Data-Prompt to train a multimodal pathology image segmentation model, input the full-length surgical pathology image, and obtain the segmented regions, cell regions, and cell types at each level.

[0037] Use pre-trained cell segmentation encoder and region segmentation encoder of pathological images to speed up training.

[0038] Analyze segmentation cues using a pre-trained language encoder.

[0039] The image features and the language features of the prompt words are aligned and fused, and then input into the segmentation model for segmentation.

[0040] The encoder of the basic model of the neural network pre-trained on a large number of pathological images is used as the cell, region, and language encoder of the segmentation model.

[0041] The prompt word refers to segmentation range + segmentation object. The segmentation range indicates the target area to be segmented, such as the name of the shallow layer or deep layer; the segmentation object refers to the segmented object, such as tumor cells, tumor stromal cells, etc.

[0042] The decoder of the segmentation model refers to a neural network that converts features into segmentation results, such as UNet, Transformer and other network structures.

[0043] The difference between the segmentation result generated by the model and the manually annotated segmentation template is used as the cost, and the Dice cost, cross entropy and Dice combined cost are calculated to train the segmentation model.

[0044] A phased training method is adopted. In the first step, the encoder is fixed and only the decoder is trained; in the second step, the encoder and decoder are trained together.

[0045] Optionally, the spatial relationship extraction model is trained using the pathological image, the spatial relationship diagram of the spatial relationship between the cell types in the pathological image and the anatomical level to obtain a trained spatial relationship extraction model, including: The clusters of various cell types and the anatomical levels are used as nodes to construct a spatial relationship diagram of the spatial position relationship between nodes, and the nodes and spatial relationships are used as data sets to train a spatial relationship extraction model to obtain a trained spatial relationship extraction model.

[0046] In the above embodiment, the training of the spatial relationship extraction model specifically includes: 1. Create a spatial relationship dataset The spatial relationship between various objects in the image is represented by a spatial relationship graph. Each node of the spatial relationship graph represents an object, mainly cell name and region name; the edge between nodes represents the spatial relationship between two objects, such as adjacency, infiltration and inclusion, that is, the relationship between cells and tissue structures.

[0047] Preferably, clusters formed by various cells are used as nodes to reduce the number of cell nodes.

[0048] The pathological images were scaled to 5x and 20x magnified images to produce spatial spatial relationship diagrams at two different scales.

[0049] The segmentation model is used to segment various types of cells and regions in the image as nodes; the edges are used to represent the relationship between nodes, and a dataset of spatial relationships, Data-Seg-Image, is artificially created.

[0050] 2. Training the Spatial Relationship Extraction Model The paired pathological images and spatial relationship graphs of Data-Seg-Image are used as training data to train the spatial relationship extraction model.

[0051] Optionally, the encoder and decoder can be implemented using a network structure of a codec such as Transformer.

[0052] Optionally, the predictor includes a target detector and a relationship predictor. The target detector predicts the location of the target and classifies the target; the relationship predictor predicts the relationship between the targets. The target detector and the relationship predictor are implemented using network structures such as a target localization network and a convolutional neural network.

[0053] Training spatial relationship extraction models of two scales, such as 5x and 20x, to extract spatial relationship graphs of spatial relationships at different scales. Optionally, the predictors of the two scales adopt weight sharing.

[0054] The difference between the categories of the extracted spatial relationship graph nodes and the node categories annotated by Data-Seg-Image, and the difference between the spatial relationship and the spatial relationship annotated by Data-Seg-Image are used as the cost function.

[0055] Using the trained spatial relationship extraction model, the spatial relationships of all pathological full-slice images Data-Full are extracted to obtain a spatial relationship graph Data-Full-Image that describes the spatial relationships.

[0056] Optionally, the shallow pathology image and the spatial relationship map of the shallow pathology image are used as input, and the deep pathology image is used as output to train the information reconstruction model to obtain a trained information reconstruction model, including: The shallow pathology images and deep pathology images of the same full-slice pathology image are used as data sets to train the information reconstruction model. The shallow pathology images and the spatial relationship graph of the shallow pathology images in the data set are used as input data of the information reconstruction model, and the deep pathology images of the full-slice pathology image where the shallow pathology images are located are used as output data of the information reconstruction model.

[0057] In the above embodiment, the training of the information reconstruction model specifically includes: 1. Data preparation The image segmentation model is used to complete the segmentation of the full-length pathological image, denoted as Data-Seg.

[0058] The spatial relationship extraction model is used to extract the spatial relationship graph of the spatial relationship of the full-length pathological image, which is recorded as Data-Full-Image.

[0059] The paired pathological images and spatial relationship graphs in Data-Full and Data-Full-Image are used for training the information reconstruction model.

[0060] 2. Aligning the features of the spatial relationship graph and the image Training method: Mask the nodes in the shallow layer of the tumor in the spatial relationship graph ( Figure 5 ), while masking the deep pathological images ( Figure 5 The deep information of the spatial relationship graph and the shallow information of the pathological image are used to reconstruct the complete tumor (including shallow and deep tumor tissue) image.

[0061] Two types of cost functions, pixel-level reconstruction and clinical-level reconstruction, are used to calculate the difference between the reconstructed deep pathology image and the true deep pathology image.

[0062] The goal of the graph encoder and encoder of the training information reconstruction model is to align the features of the spatial relationship graph and the image features.

[0063] Swap the masked positions of the spatial relationship graph and the pathological image, and train the graph encoder and encoder again. For example, in the first training cycle, mask the shallow layer of the spatial relationship graph and the deep layer of the pathological image; in the second training cycle, mask the deep layer of the spatial relationship graph and the shallow layer of the pathological image. And so on.

[0064] 3. Reconstruct the model using training information The information reconstruction model is used to predict the pathological image of deep tumor tissue from the pathological image of shallow tumor tissue. To this end, the deep spatial relationship map (because it cannot be obtained by biopsy) is masked and the information reconstruction model is established together with the shallow pathological image.

[0065] Neural networks, such as U-Net, Transformer and other neural network models, are used as the backbone network.

[0066] The information reconstruction model includes graph encoder, encoder, decoder and other components.

[0067] The graph encoder aligns the spatial relationship graph features and image features, which are trained in step 2.

[0068] The encoder and decoder can be implemented using the encoder and decoder structure of a neural network, such as the encoder and decoder of the Transformer's multi-head attention.

[0069] An image generation model framework, such as a generative adversarial network, a diffusion model, or other image generation network models, is used as the network framework of the decoder of the information reconstruction model.

[0070] The cross-attention mechanism is applied to the encoder and decoder to improve the reconstruction accuracy.

[0071] The training goal is to compare the differences between the reconstructed deep pathology images and the real deep pathology images. The training method can be generative adversarial, conditional noise diffusion, etc.

[0072] Two types of cost functions, pixel-level reconstruction and clinical-level reconstruction, are used to calculate the difference between the reconstructed deep pathology image and the true deep pathology image.

[0073] 4. Cost Function for Pixel-Level Reconstruction The pixel-level reconstruction cost function is used to calculate the difference between the reconstructed deep pathology image and the real deep pathology image, including the weighted sum of pixel cost, perceptual cost, and discrimination cost, to reflect the difference between the two types of images.

[0074] The pixel cost uses the mean square error to measure the pixel-level difference between the two; the perceptual cost uses the pre-trained VGG-16 network to calculate the perceptual difference to better capture the structure and content information of the image; the identification cost uses the discriminator to judge the authenticity of the reconstructed image.

[0075] 5. Cost Function of Clinical-Grade Reconstruction Strategy Clinical-grade reconstruction means that the real deep-layer images and the reconstructed deep-layer images have the same pathological diagnostic effectiveness.

[0076] In the first step, an image segmentation model is used to segment the tumor cells in each layer in the real deep pathology image and the reconstructed deep pathology image.

[0077] In the second step, the number of tumor cells in each layer is calculated and divided into different levels, such as many, few, and none.

[0078] The third step is to calculate the cost function. For example, in the same layer of the real pathological image and the reconstructed pathological image, the closer the tumor cell level is, the lower the cost is, regardless of the specific location and exact number of tumor cells.

[0079] The cost functions of the first two reconstruction strategies are combined to achieve information reconstruction that integrates the advantages of the two strategies.

[0080] The image is divided into small image blocks (covering images at all levels) perpendicular to the direction of shallow and deep layers for training models in order to increase the amount of data and reduce hardware requirements.

[0081] Optionally, the predicting the tumor cell infiltration depth based on the deep pathological image includes: Deep pathological images are analyzed manually using image diagnosis or automatic image analysis models to obtain the types of tumor cells in deep areas and deep layers, and to obtain information on the depth of tumor cell infiltration.

[0082] In the above embodiment, the steps of the image analysis model are as follows: Figure 7 shown.

[0083] Example 1: The image analysis model for estimating the depth of tumor cell invasion can be obtained through the image segmentation model, which can be divided into the following steps: 1. The shallow pathological images obtained by biopsy are input into the spatial relationship extraction model and information reconstruction model to obtain the deep pathological images.

[0084] 2. The reconstructed deep pathology image is input into the image segmentation model (obtained in the second step of this embodiment) to output the result of tumor cell segmentation.

[0085] 3. The presence and quantity of tumor cells at each layer are used to determine the depth of tumor cell infiltration.

[0086] The prompt words of the image segmentation model are segmentation level + the name of the tumor cell.

[0087] The depth of infiltration is determined based on whether there are tumor cells in a certain layer, that is, whether there are tumor cells separated from this layer; the number of tumor cells, that is, how many tumor cells are separated from this layer. If tumor cells appear in a certain layer, it is considered to have infiltrated into this layer.

[0088] Example 2: The estimation of tumor cell invasion depth can be obtained through image classification model, which can be divided into the following steps: 1. The shallow pathological image obtained by biopsy is input into the spatial relationship extraction model and information reconstruction model to obtain the deep pathological image.

[0089] 2. Annotate deep pathology images with the depth (level) of infiltration as a classification label.

[0090] 3. Use deep pathology images as input and the classification label of invasion depth as output to train the image classification model.

[0091] 4. Input the deep pathology image into the trained image classification model and output the category of infiltration depth.

[0092] Example 3: Estimating the tumor cell invasion depth can be done by performing knowledge distillation on the spatial relationship extraction model, information reconstruction model, and image classification model to obtain an invasion depth prediction model, which can be divided into the following steps: 1. The shallow pathological images obtained by biopsy are input into the spatial relationship extraction model, information reconstruction model and image classification model in sequence.

[0093] 2. Construct an infiltration depth prediction model based on the classification model framework.

[0094] 3. Use the image classification model of Example 2 as the parent model and the infiltration depth model as the child model. Input the shallow pathology images obtained by biopsy and implement knowledge distillation to minimize the output difference between the parent model and the child model.

[0095] The above output includes feature output and classification result output.

[0096] 4. Input the shallow pathological images obtained by biopsy into the trained infiltration depth prediction model to obtain the category of infiltration depth of tumor cells.

[0097] Preferably, shallow pathological images are extracted from Data-Prompt and labeled with invasion depth labels as an augmented data set to further train the invasion depth prediction model.

[0098] The embodiment of the present application also provides a tumor cell infiltration depth prediction system based on pathological image reconstruction, including a processor, a memory and a communication bus; Memory, used to store computer programs; The processor is used to implement any method step in the method for predicting tumor cell infiltration depth based on pathological image reconstruction when executing the program stored in the memory.

[0099] The above-mentioned tumor cell infiltration depth prediction system based on pathological image reconstruction can implement various embodiments of the above-mentioned tumor cell infiltration depth prediction method based on pathological image reconstruction, and can achieve the same beneficial effects, which will not be described in detail here.

[0100] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for predicting tumor cell infiltration depth based on pathological image reconstruction, characterized in that: include: Obtaining a full-length pathological image of the patient, marking a shallow region and a deep region on the full-length pathological image, and marking corresponding cell types and anatomical layers in the shallow region and the deep region, and constructing a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical layer spatial relationship diagram describing the spatial relationship of the anatomical layers; Constructing an image segmentation model, and using the shallow region and the deep region as training sets to train the image segmentation model to obtain a trained image segmentation model; Constructing a spatial relationship extraction model, and using the cell type spatial relationship map and the anatomical level spatial relationship map to train the spatial relationship extraction model to obtain a trained spatial relationship extraction model; Segmenting the full-slice pathology image by a trained image segmentation model to obtain a shallow pathology image and a deep pathology image, and extracting a shallow pathology image spatial relationship graph of the shallow pathology image by a trained spatial relationship extraction model; An information reconstruction model is constructed, and the shallow pathology image and the spatial relationship diagram of the shallow pathology image are used as input and the deep pathology image is used as output to train the information reconstruction model to obtain a trained information reconstruction model; The shallow pathology image obtained by biopsy is input into the trained spatial relationship extraction model to obtain the corresponding shallow pathology image spatial relationship map, the shallow pathology image and the corresponding shallow pathology image spatial relationship map are input into the trained information reconstruction model to obtain the deep pathology image, and the tumor cell infiltration depth is predicted based on the deep pathology image.

2. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The marking of the shallow layer area and the deep layer area on the pathological full-film image includes: The image of the shallow region and the image of the deep region are extracted from the pathological full-slice image in a direction perpendicular to the anatomical structure of the disease, and the shallow region and the deep region are annotated in the pathological full-slice image by manual annotation.

3. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The step of marking corresponding cell types and anatomical levels in the shallow region and the deep region, and constructing a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical level spatial relationship diagram describing the spatial relationship of the anatomical levels, comprises: The cell types and anatomical layers are manually annotated in the shallow and deep areas respectively. By annotating the spatial relationships between the cell types and the anatomical layers, a cell type spatial relationship diagram describing the spatial relationship of the cell types and an anatomical layer spatial relationship diagram describing the spatial relationship of the anatomical layers are constructed. The spatial relationships between the cell types and the anatomical layers include: adjacency, infiltration and inclusion.

4. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The method of using the shallow region and the deep region as training sets to train the image segmentation model to obtain a trained image segmentation model includes: The image segmentation model is constructed by inputting manually annotated full-length pathological images containing shallow and deep areas as a training set to train the image segmentation model to obtain a trained image segmentation model.

5. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The method of training a spatial relationship extraction model using the cell type spatial relationship graph and the anatomical level spatial relationship graph to obtain a trained spatial relationship extraction model includes: The clustering of the cell types and the anatomical hierarchy are used as nodes to construct a spatial relationship graph of the spatial relationship between nodes, and the pathological image and its corresponding spatial relationship graph are used as data sets to train a spatial relationship extraction model to obtain a trained spatial relationship extraction model.

6. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The training information reconstruction model uses a total cost function weighted by two cost functions: pixel-level reconstruction and clinical-level reconstruction; The difference between the deep pathology image calculated by the pixel-level reconstruction cost function and the real deep pathology image, including pixel, perception and discrimination costs; the difference between the deep pathology image calculated by the clinical-level reconstruction cost function and the real deep pathology image in terms of tumor cell types and quantity levels at the same anatomical level; The cell types and quantity levels at the anatomical level refer to the quantity levels of the same type of tumor cells at the same anatomical level of the deep pathology image. The quantity levels are artificially divided. The clinical-level reconstruction cost function compares the output deep pathology images and the differences in the quantity levels of the same type of tumor cells at the same anatomical level in the real deep pathology images.

7. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The method of training the information reconstruction model by taking the shallow pathology image and the shallow pathology image spatial relationship map as input and the deep pathology image as output to obtain the trained information reconstruction model includes: Shallow pathology images, shallow pathology image spatial relationship diagrams and deep pathology images are used as data sets to train the information reconstruction model. The shallow pathology images and shallow pathology image spatial relationship diagrams in the data sets are used as input data of the information reconstruction model, and the deep pathology images in the data sets are used as output data of the information reconstruction model.

8. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The predicting of the tumor cell infiltration depth based on the deep pathological image includes: Deep pathological images are analyzed manually using image diagnosis or automatic image analysis models to obtain the tumor cell types in deep areas and deep anatomical layers, and to obtain information on the depth of tumor cell infiltration. The image analysis model uses a segmentation model or a classification model, which is trained using deep pathology images and manually annotated data on cell types and anatomical levels. The image segmentation model segments and counts tumor cells at each layer of deep pathology images to obtain the presence and number of tumor cells in each anatomical layer, thereby determining the depth of infiltration. The image classification model classifies the infiltration depth of deep pathology images, inputs deep pathology images, and outputs the predicted label of the anatomical level of the infiltration depth.

9. The method for predicting tumor cell infiltration depth based on pathological image reconstruction according to claim 1, characterized in that: The spatial relationship extraction model, information reconstruction model and image analysis model are subjected to knowledge distillation to train the invasion depth prediction model. The shallow pathological images are input into the trained invasion depth prediction model to output the prediction label of the anatomical level of the invasion depth.

10. A tumor cell infiltration depth prediction system based on pathological image reconstruction, characterized in that: including a processor, a memory and a communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

Images