Scoring method of Tils

Abstract

The invention provides a scoring method for tumor infiltration lymph. The method comprises the following steps: obtaining a semantic segmentation image corresponding to each pathological image by using a semantic segmentation model based on multi-scale feature information of a pathological image data set; wherein the pathological image data set comprises a plurality of pathological images, each pixel point in the semantic segmentation image is labeled with a semantic category label, and the semantic category label comprises a matrix region label; utilizing a Tils counting model to obtain a density map corresponding to each pathological image based on the multi-scale feature information of the pathological image data set, the semantic segmentation image and preset priori knowledge; wherein each pixel value in the density map represents the probability that the immune cells exist in the corresponding pixel point, and the preset priori knowledge comprises the probability that the immune cells exist in each semantic region; according to the semantic segmentation image and the density map, determining the matrix region area and the number of immune cells of each pathological image, and calculating the Tils score of each pathological image according to the matrix region area and the number of immune cells.

Description

Technical Field

[0001] The present invention relates to the technical field of pathological image processing, and in particular to a Tils scoring method. Background Art

[0002] At present, cancer (e.g., breast cancer, lung cancer, etc.) has become one of the most common diseases worldwide, and cancer treatment depends largely on the biological characteristics of the tumor. In recent years, more and more evidence has shown that the patient's immune system plays an important role in determining treatment options and long-term survival rates. As a carrier of complex interactions between tumor cells and the host immune system, the tumor microenvironment (TME) has received increasing attention in the field of immuno-oncology in recent years. Tumor-infiltrating lymphocytes (Tils) refer to the process of immune cells infiltrating into the tumor and surrounding matrix areas in the TME. In this process, immune cells play a key role in identifying and attacking tumor cells and inhibiting tumor growth and spread. Therefore, the Tils scoring method is used to quantitatively evaluate the number and distribution of immune cells in the tumor microenvironment to understand the patient's immune status and treatment response, thereby assisting doctors in determining appropriate treatment strategies and evaluating patient prognosis, which is of great significance for improving patient survival.

[0003] There are some semi-quantitative methods of Tils scoring in the prior art to evaluate the distribution of immune cells in the matrix area near tumor cells. However, the above methods usually rely on subjective visual evaluation by pathologists, so accuracy, evaluation efficiency, and objectivity are difficult to guarantee. At the same time, since the current process of pathologists manually annotating Tils scores is semi-quantitative, it is not necessary to accurately detect the number of immune cells.

[0004] In addition, there are some automated Tils scoring methods based on deep learning in the prior art. The area of ​​the matrix region near the tumor cells is first determined by the semantic segmentation model, and then the Tils counting model is used to calculate the number of immune cells in the matrix region. However, it is difficult for the existing automated Tils scoring methods to guarantee the accuracy of Tils scoring. On the one hand, the existing automated Tils scoring methods usually train the semantic segmentation model and the Tils counting model independently, while ignoring the correlation between the two models. In fact, there is a close correlation between the two models. For example, accurate segmentation of the matrix region helps to locate immune cells more accurately, and the counting results of immune cells can also verify the accuracy of the matrix region segmentation in turn. Therefore, ignoring the correlation between the two models may lead to deviations in the Tils scoring results. On the other hand, the existing Tils counting model uses the idea of ​​the target detection algorithm to count immune cells, and the dense distribution and mutual occlusion of immune cells in the tumor microenvironment will seriously affect the accuracy of the target detection algorithm, thereby reducing the accuracy of the Tils scoring. Summary of the invention

[0005] In view of this, the present application provides a Tils scoring method to improve the accuracy of Tils scoring.

[0006] The present application provides a Tils scoring method, and the Tils scoring method includes: A semantic segmentation model is used to obtain a semantic segmentation image corresponding to each pathological image based on multi-scale feature information of a pathological image dataset; wherein the pathological image dataset includes a plurality of pathological images, each pixel in the semantic segmentation image is annotated with a semantic category label, and the semantic category label includes a matrix region label; A density map corresponding to each pathological image is obtained by using the Tils counting model based on the multi-scale feature information of the pathological image dataset, the semantic segmentation image, and preset prior knowledge; wherein each pixel value in the density map represents the probability of the existence of immune cells at the corresponding pixel point, and the preset prior knowledge includes the probability of the existence of immune cells in each semantic area; The matrix area and the number of immune cells of each pathological image are determined according to the semantic segmentation image and the density map, so as to calculate the Tils score of each pathological image according to the matrix area and the number of immune cells.

[0007] The above-mentioned Tils scoring method incorporates the semantic segmentation image output by the semantic segmentation model and the probability of immune cells appearing in each semantic area into the training process of the Tils counting model, thereby realizing the associated training of the semantic segmentation model and the Tils counting model. At the same time, the Tils counting model in this application draws on the idea of ​​the crowd counting algorithm to better handle the occlusion problem of immune cells, and the obtained immune cell counting results are more accurate, thereby improving the accuracy of the Tils score. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0009] Figure 1 It is a flow chart of the Tils scoring method provided in the embodiments of the present application.

[0010] Figure 2 This is the first sub-flowchart of the Tils scoring method provided in the embodiment of the present application.

[0011] Figure 3 This is the second sub-flowchart of the Tils scoring method provided in the embodiment of the present application.

[0012] Figure 4 It is a structural diagram of the semantic segmentation model provided in the embodiment of the present application.

[0013] Figure 5 It is a structural diagram of the Tils counting model provided in the embodiment of the present application.

[0014] Description of the accompanying drawings: The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0016] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar planning objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate. In other words, the described embodiments are implemented according to a sequence other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations may also include other content. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to only those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] It should be noted that the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0018] Please see Figure 1 , Figure 1 The process of the Tils scoring method is illustrated. The present application provides a scoring method for tumor-infiltrating lymphocytes (Tils), which can achieve a more accurate Tils scoring function, thereby assisting doctors to quickly and accurately analyze and understand the tumor microenvironment and improve the patient's prognosis and survival. The Tils scoring method includes steps S101 to S105.

[0019] Step S101, using a semantic segmentation model to obtain a semantic segmentation image corresponding to each pathological image based on multi-scale feature information of a pathological image dataset; wherein the pathological image dataset includes a plurality of pathological images, and each pixel point in the semantic segmentation image is annotated with a semantic category label, and the semantic category label includes a matrix region label.

[0020] In step S101, the pathological image dataset includes several pathological images, which are whole-slide imaging (WSI) of several patients (e.g., breast cancer patients, liver cancer patients, lung cancer patients, etc.) collected. The pathological image dataset is divided into a pathological image training set, a pathological image verification set, and a pathological image test set, which are used for training, verification, and testing of a semantic segmentation model and a Tils counting model, respectively. It can be understood that the collected pathological image dataset is an original pathological image dataset, and the pathological image dataset needs to be annotated by a pathologist to meet the requirements of the semantic segmentation model and the Tils counting model for the input pathological image dataset. The annotation of the pathological image training set is explained as follows. Specifically, the pathological image training set includes several training pathological images, and the pathologist annotates the several training pathological images, including semantically annotating each semantic region in each training pathological image to obtain a pathological image training set with semantic annotations. The pathologist annotates the several training pathological images, and also includes point annotating the immune cells in each training pathological image to obtain a pathological image training set with point annotations. In addition, the present application uses a Gaussian distribution function to convert point annotations into density annotations to obtain a pathological image training set with density annotations. In other words, the semantic segmentation model and the Tils counting model are trained based on the pathological image training set with semantic annotations and density annotations.

[0021] Furthermore, the pathological image includes a variety of different semantic regions (for example, matrix regions, tumor regions, and background tissue regions). The semantic segmentation image is the result of segmenting different semantic regions in the pathological image, wherein each pixel in the semantic segmentation image is annotated with a semantic category label. The semantic category label is a pixel-level category label used to identify and distinguish different semantic regions in the pathological image. Different semantic regions correspond to different semantic category labels. Pixels with the same semantic category label are combined together to form corresponding semantic regions. It can be understood that the semantic category labels in the present application include matrix region labels, tumor region labels, and background tissue region labels that correspond one-to-one to semantic regions (matrix regions, tumor regions, and background tissue regions). In addition, the specific process of the semantic segmentation model obtaining the semantic segmentation image corresponding to each pathological image based on the multi-scale feature information of the pathological image data set will be described in detail below (steps S201-step S207).

[0022] Step S103, using the Tils counting model to obtain a density map corresponding to each pathological image based on the multi-scale feature information of the pathological image data set, the semantic segmentation image, and the preset prior knowledge; wherein each pixel value in the density map represents the probability of the existence of immune cells at the corresponding pixel point, and the preset prior knowledge includes the probability of the existence of immune cells in each semantic area.

[0023] In step S103, the Tils counting model in the present application uses the idea of ​​the crowd counting algorithm to implement the counting function of immune cells. The goal of the crowd counting algorithm is to count the number of people in the image. The goal of the Tils counting model is to count the number of immune cells in the pathological image. Both of them are aimed at counting specific objects in the image, so the algorithm ideas are interoperable. At the same time, the crowd counting algorithm performs Gaussian smoothing on the center position of the head of each person in the image to obtain a density map representing the density distribution of people. Correspondingly, the Tils counting model also uses a similar method to generate a density map representing the density distribution of immune cells. In this embodiment, each pixel value in the density map generated by the Tils counting model represents the probability value (between 0 and 1) of the existence of immune cells at the corresponding pixel point. The larger the probability value, the greater the probability that the current pixel point belongs to an immune cell. In addition, by setting a preset threshold M, pixels with probability values ​​higher than the preset threshold M are regarded as having immune cells.

[0024] It can be understood that the density map represents the presence of immune cells through continuous probability values, rather than simply judging the presence or absence of immune cells. This makes it possible that even if there are immune cells that are blocked, the probability value of the surrounding area can still reflect the presence of the blocked immune cells. In other words, the present application is less affected by the problem of immune cell occlusion by counting through density maps, and can accurately reflect the distribution of immune cells in the tumor microenvironment. The Tils counting model in the existing method usually adopts the idea of ​​target detection algorithm to realize the immune cell counting function, that is, by generating a bounding box to mark the position and size of immune cells in the image, and the occlusion of immune cells and other problems will seriously affect the accuracy of the immune cell counting results. Therefore, the Tils counting model of the present application can better handle the occlusion problem of immune cells compared to the Tils counting model of the prior art, and the immune cell counting results are more accurate. In addition, the specific process of the Tils counting model to obtain the density map corresponding to each pathological image based on the semantic segmentation image and the preset prior knowledge will be described in detail below (step S301-step S305).

[0025] Step S105 , determining the matrix area and the number of immune cells of each pathological image according to the semantic segmentation image and the density map, so as to calculate the Tils score of each pathological image according to the matrix area and the number of immune cells.

[0026] In step S105, since each pixel in the semantic segmentation image is marked with a corresponding semantic category label, the number of pixels whose semantic category label is a matrix region label in the semantic segmentation image is counted, and the total number of pixels obtained is the matrix region area S of each pathological image. At the same time, since each pixel value in the density map represents the probability value of the existence of immune cells in the corresponding pixel, and since the Tils score is only for immune cells in the matrix region, the pixel values ​​in the density map except the matrix region are set to 0, and then the number of pixels whose pixel values ​​in the remaining density map are higher than the preset threshold M is counted to obtain the number of immune cells N in the matrix region. In addition, since the area of ​​a single immune cell size under a microscope is an empirical constant C, the product N*C of the number of immune cells N and the empirical constant C represents the area of ​​immune cells in the matrix region. Finally, the ratio of the area N*C of immune cells in the matrix region to the matrix region area S is used as the Tils score R of each pathological image, and the specific formula is: R=N*C / S. Among them, the R value is an indicator for predicting tumor prognosis and treatment effect. Understandably, the higher the R value, the greater the number of immune cells in the same area of ​​the matrix, which also indicates that the immune system responds more actively to the tumor.

[0027] In the present embodiment, the semantic segmentation model in step S101 and the Tils counting model in step S103 are obtained through training. The training of the semantic segmentation model and the Tils counting model is based on the pre-trained initial semantic segmentation model and the initial Tils counting model. Specifically, the pre-training includes fine-tuning the parameters of the semantic segmentation model and the Tils counting model to obtain the initial semantic segmentation model and the initial Tils counting model. It can be understood that the semantic segmentation model and the Tils counting model before pre-training are suitable for a variety of images, while the initial semantic segmentation model and the initial Tils counting model obtained after pre-training are more suitable for the pathological images of the present application. Therefore, training based on the initial semantic segmentation model and the initial Tils counting model can improve the efficiency of training. The structure of the semantic segmentation model and the Tils counting model, as well as the process of training the semantic segmentation model and the Tils counting model will be further introduced below.

[0028] Please refer to Figure 4 , Figure 4The structure of the semantic segmentation model is shown in Figure 1. The semantic segmentation model uses the Visual Attention Network (VAN) as the backbone network and builds a Convolutional Neural Networks (CNN) model based on the U-Net structure. Specifically, the U-Net structure includes an encoder part (i.e., feature extraction layer), a decoder part, and a bottleneck part between the encoder and decoder parts.

[0029] The encoder part adopts a bottom-up feature extraction path. The encoder part consists of four stages (corresponding to Figure 4 The four VAN blocks in the image are used to extract features from the input pathological image layer by layer through these four stages to obtain multi-scale feature information. Among them, the feature map output by the VAN block of each stage is passed to the VAN block of the next stage (from low-level feature map to high-level feature map). The four stages correspond to feature information of different scales and levels. In this process, the resolution and detail information of the high-level feature map gradually decrease, while the high-level semantic information gradually becomes richer. In order to solve the problem of low resolution and little detail information of high-level feature maps, the Feature Pyramid Network (FPN) is used for feature fusion. The specific structure of the Feature Pyramid Network will be further explained below.

[0030] The feature output of each stage of the encoder part is linked to the decoder part through the feature pyramid network. Specifically, the feature pyramid network adopts a top-down feature fusion path. First, the high-level feature map is upsampled (such as bilinear interpolation, deconvolution, etc.), and then the upsampled high-level feature map is fused with the lower-level feature map. The fusion method is not limited here. At the same time, the top-down feature fusion path is connected to the bottom-up feature extraction path through a lateral connection. The lateral connection combines the high-resolution information of the low-level feature map with the rich semantic information of the high-level feature map to form a new feature map and input it to the decoder part. The decoder part restores the resolution of the feature map layer by layer, and finally performs semantic segmentation on the feature map output by the decoder part to output a semantic segmentation image with the same resolution as the input pathological image.

[0031] In addition, the bottleneck part uses the pyramid pooling module (PPM) to provide multi-scale pooling operations. Specifically, the pyramid pooling module is located after the last stage of the encoder part. The pyramid pooling module uses the feature information output by the last stage to further extract the global feature information of the pathological image through multi-scale pooling operations. Among them, the highest-level feature map after upsampling is fused with the global feature information and then fused with the lower-level feature map. It can be understood that the highest-level feature map contains rich semantic information, but lacks detailed information. This defect is compensated by fusing with the global feature information, making the feature fusion more comprehensive and accurate.

[0032] Please refer to Figure 2 , Figure 2 The training process of the semantic segmentation model is schematically shown. The pathological image training set includes a number of training pathological images, and the Tils scoring method also includes training the semantic segmentation model, and the training of the semantic segmentation model specifically includes steps S201 to S207.

[0033] Step S201: input a pathological image training set, and perform feature extraction on each training pathological image to obtain multi-scale feature information.

[0034] In step S201, the semantic segmentation model includes a feature extraction layer (i.e., the encoder part), and the multi-scale feature information of the pathological image data set is obtained after the pathological image data set is input into the feature extraction layer for feature extraction. In this embodiment, the input pathological image training set is a pathological image training set with density annotation. At the same time, the input pathological image training set is also a pathological image training set including several training pathological image slices. It can be understood that several training pathological images in the pathological image training set are original training pathological images, and the original training pathological images need to be preprocessed. Specifically, several training pathological images in the pathological image training set are cut to form several training pathological image groups, each training pathological image group includes multiple training pathological image slices, and each training pathological image group is input one by one to the semantic segmentation model for training. It can be understood that in some other feasible embodiments, preprocessing the training pathological images can also include image denoising, image enhancement and other operations, which are no longer limited here.

[0035] Step S203, extracting global feature information of the multi-scale feature information, and fusing the multi-scale feature information and the global feature information to obtain fused feature information. Specifically, extracting global feature information of the multi-scale feature information through a pyramid pooling module, and fusing the multi-scale feature information and the global feature information to obtain fused feature information.

[0036] Step S205, segment each training pathology image into different semantic regions according to the fused feature information to obtain a semantic segmentation image corresponding to each training pathology image. Specifically, the fused feature image is mapped into a probability image, each pixel in the probability image has a corresponding probability distribution, and the probability distribution includes the probability that the pixel belongs to each semantic region category, and the semantic region category with the highest probability is selected as the final semantic category label of the pixel, so that the semantic segmentation model outputs a semantic segmentation image corresponding to each training pathology image.

[0037] Step S207, calculating the loss value of the semantic segmentation image, and optimizing the semantic segmentation model based on the loss value of the semantic segmentation image.

[0038] In step S207, the semantic segmentation model learns how to distinguish each semantic region in the pathological image through continuous training. The loss value of the semantic segmentation image is an indicator to measure whether the current semantic segmentation model can distinguish each semantic region in the pathological image well. In other words, the loss value of the semantic segmentation image also represents the gap between the predicted semantic segmentation image and the real semantic segmentation image. In this embodiment, the image obtained by the pathologist semantically annotating each semantic region in the pathological image is a real semantic segmentation image. The semantic segmentation model is trained with the real semantic segmentation image as a label to obtain the predicted semantic segmentation image. It can be understood that the larger the loss value of the semantic segmentation image, the larger the gap between the predicted semantic segmentation image output by the semantic segmentation model and the real semantic segmentation image. Therefore, it is necessary to minimize the loss value of the semantic segmentation image through algorithms such as gradient descent algorithms, so as to optimize the parameters of the semantic segmentation model so that the predicted semantic segmentation image generated by the semantic segmentation model gradually approaches the real semantic segmentation image.

[0039] Further, the loss value of the semantic segmentation image is determined based on the cross entropy loss function and the Dice loss function. Specifically, the calculation formula of the loss value L of the semantic segmentation image is, L=aL1+bL2, wherein L1 is the loss value calculated by the cross entropy loss function, L2 is the loss value calculated by the Dice loss function, and a and b are the hyperparameters of the cross entropy loss function and the Dice loss function, respectively. In this embodiment, the hyperparameters of the cross entropy loss function and the Dice loss function are the same. For example, the hyperparameters a and b of the cross entropy loss function and the Dice loss function are both set to 1. It can be understood that the hyperparameters of the two loss functions are the same, which means that the two loss functions have the same weight when calculating the loss value L.

[0040] Please refer to Figure 5 , Figure 5The structure of the Tils counting model is shown schematically. The Tils counting model includes a feature extraction layer and a prior interaction layer, wherein the Tils counting model and the above-mentioned semantic segmentation model share the same feature extraction layer to extract features from the pathological image training set to obtain multi-scale feature information, and further process the multi-scale feature information to obtain the intermediate layer output of the Tils counting model. The prior interaction layer uses the above-mentioned intermediate layer output and the semantic segmentation image output by the semantic segmentation model as input, and the prior interaction layer also presets prior knowledge (i.e., the probability of the existence of immune cells in each semantic region). The prior interaction layer fuses the three to obtain the density map corresponding to each training pathological image.

[0041] Please refer to Figure 3 , Figure 3 The training process of the Tils counting model is shown in the figure. The Tils scoring method further includes training the Tils counting model, which specifically includes steps S301 to S305.

[0042] Step S301, based on the multi-scale feature information, obtain the Tils appearance information in each training pathological image. Specifically, the multi-scale feature information extracted by the feature extraction layer is convolved to obtain the Tils appearance information, which is the intermediate layer output of the Tils counting model, and the Tils appearance information is input into the prior interaction layer. The Tils appearance information includes the contour information of immune cells.

[0043] Step S303 , combining Tils appearance information, semantic segmentation image, and preset prior knowledge to obtain a density map corresponding to each training pathological image.

[0044] In step S303, the prior interaction layer combines the Tils appearance information, the semantic segmentation image, and the preset prior knowledge to obtain the corresponding density map . Density plot The calculation formula is: ; Where C is the total number of semantic regions, and the value of c ranges from 1 to C. represents the probability of immune cells appearing in each semantic region, represents the semantic segmentation image output by the semantic segmentation model, It is the intermediate layer output of the Tils counting model (i.e., Tils appearance information). It can be understood that the Tils counting model in the present application obtains the density map corresponding to each pathological image based on the semantic segmentation image and the preset prior knowledge. In other words, the density map in the present application is not obtained by directly processing the pathological image, but is integrated with the semantic segmentation image output by the semantic segmentation model, and the probability of immune cells appearing in each semantic area. Because immune cells are unevenly distributed in different semantic areas. Therefore, the density map obtained based on the semantic segmentation image and the distribution probability of immune cells in each semantic area can more accurately reflect the distribution of immune cells in the tumor microenvironment, thereby improving the accuracy of the immune cell counting results.

[0045] Step S305, calculating the loss value of the density map, and optimizing the Tils counting model based on the loss value of the density map.

[0046] In step S305, the loss value of the density map is determined based on the minimum mean square error function, wherein the loss value of the density map refers to the gap between the predicted density map output by the Tils counting model and the true density map. In the present embodiment, the pathologist performs point annotation on the pathological image training set, and then converts the point annotation into the image obtained by density annotation as a true density map. The Tils counting model is trained with the true density map as a label to obtain a predicted density map. It can be understood that the larger the loss value of the density map, the larger the gap between the predicted density map output by the semantic segmentation model and the true density map. Therefore, it is necessary to optimize the parameters of the Tils counting model by minimizing the loss value of the density map, so that the predicted density map generated by the Tils counting model gradually approaches the true density map.

[0047] In the above embodiment, the semantic segmentation image output by the semantic segmentation model and the probability of immune cells appearing in each semantic area are integrated into the training process of the Tils counting model, thereby realizing the associated training of the semantic segmentation model and the Tils counting model. At the same time, the Tils counting model in this application draws on the idea of ​​the crowd counting algorithm to better handle the occlusion problem of immune cells, and the obtained immune cell counting results are more accurate, thereby improving the accuracy of the Tils score.

[0048] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0049] The above examples are only preferred embodiments of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. A Tils scoring method, characterized in that: Tils' scoring methodology includes: A semantic segmentation model is used to obtain a semantic segmentation image corresponding to each pathological image based on multi-scale feature information of a pathological image dataset; wherein the pathological image dataset includes a plurality of pathological images, each pixel in the semantic segmentation image is annotated with a semantic category label, and the semantic category label includes a matrix region label; A density map corresponding to each pathological image is obtained by using the Tils counting model based on the multi-scale feature information of the pathological image dataset, the semantic segmentation image, and preset prior knowledge; wherein each pixel value in the density map represents the probability of the existence of immune cells at the corresponding pixel point, and the preset prior knowledge includes the probability of the existence of immune cells in each semantic area; The matrix area and the number of immune cells of each pathological image are determined according to the semantic segmentation image and the density map, so as to calculate the Tils score of each pathological image according to the matrix area and the number of immune cells.

2. The Tils scoring method according to claim 1, characterized in that: The semantic segmentation model and the Tils counting model share a feature extraction layer, and the multi-scale feature information of the pathological image dataset is obtained after the pathological image dataset is input into the feature extraction layer for feature extraction.

3. The Tils scoring method according to claim 2, characterized in that: The pathological image data set includes a pathological image training set, and the pathological image training set includes a plurality of training pathological images. The Tils scoring method also includes training the semantic segmentation model, and training the semantic segmentation model specifically includes: Input a pathological image training set, and perform feature extraction on each training pathological image to obtain multi-scale feature information; Acquire global feature information of each training pathological image based on the multi-scale feature information, and fuse the multi-scale feature information with the global feature information to obtain fused feature information; Segmenting each of the training pathological images into different semantic regions according to the fused feature information to obtain a semantic segmentation image corresponding to each of the training pathological images; A loss value of the semantic segmentation image is calculated, and the semantic segmentation model is optimized based on the loss value of the semantic segmentation image.

4. The Tils scoring method according to claim 3, characterized in that: The loss value of the semantic segmentation image is calculated based on a cross entropy loss function and a Dice loss function; wherein the values ​​of the hyperparameters of the cross entropy loss function and the Dice loss function are the same.

5. The Tils scoring method according to claim 3, characterized in that: The Tils scoring method further includes training the Tils counting model, and training the Tils counting model specifically includes: Acquire Tils appearance information of each training pathological image based on the multi-scale feature information; The Tils appearance information, the semantic segmentation image, and the preset prior knowledge are integrated to calculate a density map corresponding to each training pathological image; A loss value of the density map is calculated, and the Tils counting model is optimized based on the loss value of the density map.

6. The Tils scoring method according to claim 5, characterized in that: The loss value of the density map is calculated based on a minimum mean square error function.

7. The Tils scoring method according to claim 5, characterized in that: The training of the semantic segmentation model and the tils counting model is performed based on a pre-trained initial semantic segmentation model and an initial tils counting model.

8. The Tils scoring method according to claim 3, characterized in that: The input pathological image training set is a pathological image training set with density annotation. Before the input pathological image training set, the Tils scoring method further includes: Performing point annotation on immune cells in each training pathological image to obtain a pathological image training set with point annotation; The point annotations are converted into density annotations using a Gaussian distribution function to obtain a pathological image training set with density annotations.

9. The Tils scoring method according to claim 3, characterized in that: The input pathological image training set is a pathological image training set including a plurality of training pathological image slices. Before the input pathological image training set, the Tils scoring method further includes: Each training pathological image is sliced ​​to obtain a pathological image training set including a plurality of training pathological image slices.

10. The Tils scoring method according to claim 1, characterized in that: Determining the matrix area and the number of immune cells in each pathological image according to the semantic segmentation image and the density map specifically includes: Counting the number of pixels in the semantic segmentation image whose semantic category label is the matrix area to obtain the matrix area of ​​each pathological image; The pixel values ​​in the density map except for the matrix area are set to 0, and the number of pixel points in the remaining density map whose pixel values ​​are higher than a preset threshold is counted to obtain the number of immune cells in the matrix area.

Images