Method and device for automatic quantification of intratumoral tumor infiltrating lymphocytes

Abstract

The present application belongs to the technical field of intelligent detection, and provides an automatic quantification method and device for light-weight tumor-infiltrating lymphocytes in tumors. A bladder cancer staining pathological image is acquired and pre-processed to obtain effective color blocks; after the effective color blocks are labeled, a training set and a test set of the effective color blocks are constructed; two independent light-weight deep learning models for identifying tumor color blocks and lymphocyte aggregation color blocks, respectively, are constructed; a bladder cancer pathological image of a patient is inputted, and the overlapping area of the tumor color blocks and the lymphocyte aggregation color blocks accounts for the proportion of the total tumor color block area; the score of the tumor-infiltrating lymphocytes in the tumor of the bladder cancer patient is evaluated and / or a prognosis diagnosis is made for the bladder cancer patient. The present application assists clinicians in evaluating the tumor-infiltrating lymphocytes in the tumor, provides a reference basis for digital and individualized diagnosis and treatment decisions for patients, and provides a solid scientific foundation and technical support for the development of precision medicine.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent detection, and particularly relates to an automatic quantification method and device for light-weight intra-tumoral tumor infiltrating lymphocytes. BACKGROUND

[0002] Bladder cancer is a common urological malignancy. According to pathological diagnosis, bladder cancer can be divided into muscle-invasive bladder cancer (MIBC) and non-muscle-invasive bladder cancer (NMIBC). MIBC is prone to metastasis, and the 5-year survival rate is low after tumor progression. Radical cystectomy combined with platinum-based chemotherapy is the standard treatment for MIBC, but a considerable proportion of patients will develop primary or secondary platinum resistance, which is a challenge for treatment. Although immunotherapy has become a viable salvage treatment for patients with chemotherapy resistance, most patients worsen after an initial promising response. Evaluating immune infiltration in tumors, most commonly referred to as tumor infiltrating lymphocytes (TILs), is increasingly important in screening patients who may benefit from immunosuppressants.

[0003] In recent years, a number of studies have suggested that tumor infiltrating lymphocyte evaluation should be included as a biomarker in routine histopathology reports. Intra-tumoral TILs (iTILs) are defined as lymphocytes in tumor nests, which are not interposed in the stroma, directly contact and interact with cancer cells, while stromal TILs (sTILs) are dispersed in the stroma between cancer cells and do not directly contact cancer cells. Since intra-tumoral TILs usually exist in a lower number and are detected in fewer cases, they are more heterogeneous and difficult to observe on routine HE-stained pathology images, most studies consider stromal TILs to be a superior and more reproducible evaluation indicator. However, in neoadjuvant triple-negative breast cancer (TNBC), both intra-tumoral TILs and stromal TILs can predict pathological response to neoadjuvant platinum-based chemotherapy. Therefore, there is an urgent need to develop a precise and efficient method to effectively evaluate intra-tumoral TILs. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides an automatic quantification method and device for light-weight intra-tumoral tumor infiltrating lymphocytes, which aims to solve the problems mentioned in the background art.

[0005] In a first aspect, the present application provides an automatic quantification method for light-weight intra-tumoral tumor infiltrating lymphocytes, comprising the following steps:

[0006] Step S1: Obtain and preprocess the bladder cancer staining pathological image to obtain effective color blocks, the effective color blocks including tumor color blocks, non-tumor color blocks, lymphocyte aggregation color blocks and non-lymphocyte aggregation color blocks;

[0007] After labeling the effective color blocks, a training set and a test set of the effective color blocks are constructed;

[0008] Two independent lightweight deep learning models are constructed, the lightweight deep learning model being composed of an input layer, a start layer, four residual separated convolution blocks, a global average pooling layer and a full connection layer in series, the lightweight deep learning model being trained based on the training set and the test set of the effective color blocks, and two independent lightweight deep learning models respectively identifying tumor color blocks and lymphocyte aggregation color blocks being obtained;

[0009] Step S2: input the pathological image of the patient with bladder cancer, identify the tumor color blocks and the lymphocyte aggregation color blocks of the pathological image of the patient with bladder cancer by using the two independent lightweight deep learning models respectively identifying the tumor color blocks and the lymphocyte aggregation color blocks, and calculate the proportion of the overlapping area of the tumor color blocks and the lymphocyte aggregation color blocks to the total tumor color block area according to the spatial coordinates of the tumor color blocks and the lymphocyte aggregation color blocks of the pathological image of the patient with bladder cancer;

[0010] Step S3: evaluate the score of tumor-infiltrating lymphocytes in the tumor of the patient with bladder cancer and / or make a prognosis diagnosis on the patient with bladder cancer according to the proportion of the overlapping area of the tumor color blocks and the lymphocyte aggregation color blocks to the total tumor color block area.

[0011] Further, in step S1, the training set and the test set of the effective color blocks include the training set and the test set of the tumor color blocks and the non-tumor color blocks, and the training set and the test set of the lymphocyte aggregation color blocks and the non-lymphocyte aggregation color blocks.

[0012] Further, in step S1, the training set and the test set of the effective color blocks include the training set and the test set of the tumor color blocks and the non-tumor color blocks, and the training set and the test set of the lymphocyte aggregation color blocks and the non-lymphocyte aggregation color blocks.

[0013] Step S101: based on the training set and the test set of the tumor color blocks and the non-tumor color blocks, and the training set and the test set of the lymphocyte aggregation color blocks and the non-lymphocyte aggregation color blocks, train two initial lightweight deep learning models respectively to obtain a lightweight deep learning model identifying tumor color blocks and a lightweight deep learning model identifying lymphocyte aggregation color blocks;

[0014] Step S102: visualize the weight regions of the lightweight deep learning model identifying tumor color blocks and the lightweight deep learning model identifying lymphocyte aggregation color blocks by gradient-weighted class activation mapping;

[0015] Step S103: Collect known bladder cancer pathological images as an external verification set to verify the identification efficiency of the tumor patch by the lightweight deep learning model and the lymphocyte aggregation patch by the lightweight deep learning model.

[0016] Further, in step S1, the lightweight deep learning model is composed of an input layer, a starting layer, four residual separated convolution blocks, a global average pooling layer, and a full connection layer in series, specifically:

[0017] The input layer is used to receive the bladder cancer pathological image and transmit it to the starting layer.

[0018] The starting layer is provided with a two-dimensional convolution layer containing 32 3x3 convolution kernels, which quickly extracts shallow edge texture through ordinary convolution, increases the channel to 32, batch normalization to suppress internal covariate shift, and then introduces nonlinearity through the ReLU activation function, and then reduces the spatial size by 2 times through 2x2 two-dimensional maximum pooling.

[0019] The four residual separated convolution blocks each include a main path, a residual branch, and a compression excitation reinforcement module; the main path sequentially passes through 3x3 separable convolution, batch normalization, ReLU activation function, and 3x3 separable convolution for feature extraction; the residual branch adds the residual of the main path output after adjusting the channel through 1x1 convolution and batch normalization; the compression excitation reinforcement module generates channel weights through global average pooling and multiplies the original features; and finally outputs a 64x64x128 feature matrix.

[0020] The global average pooling layer converts the 64x64x128 feature matrix into a 128-dimensional vector through two-dimensional global average pooling.

[0021] The full connection layer is composed of a feature dimension increasing layer, a random inactivation layer, and a classification full connection layer in series, and the classification result is obtained by processing through the Softmax activation function.

[0022] Further, the bladder cancer pathological image received by the input layer is a 512x512 pixel, 3-channel RGB image.

[0023] Further, in each residual separation convolution block, the main path first passes through a 3*3 separable convolution, the 3*3 separable convolution first performs channel-by-channel convolution, and then 1*1 convolution is performed; batch normalization is then performed, nonlinearity is introduced through a ReLU activation function, 3*3 separable convolution is performed again for feature extraction; the residual branch adjusts the number of channels through 1*1 convolution, so that it matches the number of channels of the output features of the main path, and after batch normalization, the residual is added to the output features of the main path; the compression excitation reinforcement module generates channel weights through global average pooling and multiplies them with the original features; the four residual separation convolution blocks are connected in turn, continuously extracting, optimizing and strengthening the features, and finally outputting a 64*64*128 feature matrix.

[0024] Further, in step S3, the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is positively correlated with the score of tumor-infiltrating lymphocytes in the tumor of the bladder cancer patient.

[0025] Further, in step S3, the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is positively correlated with the survival rate of the bladder cancer patient.

[0026] In a second aspect, the present application provides a computer device, comprising at least one processor, and a memory connected in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic quantification method of the lightweight tumor-infiltrating lymphocytes in the tumor.

[0027] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the automatic quantification method of the lightweight tumor-infiltrating lymphocytes in the tumor.

[0028] The present application has the following beneficial effects:

[0029] (1) The lightweight deep learning model for identifying the tumor color block and the lightweight deep learning model for identifying the lymphocyte aggregation color block can identify, label and calculate the tumor color block and the lymphocyte aggregation color block. The proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is intelligently quantified, and then correlation analysis is performed, so that the score of tumor-infiltrating lymphocytes in the tumor of the bladder cancer patient and / or the prognosis of the bladder cancer patient can be evaluated. The clinician can assist in evaluating the tumor-infiltrating lymphocytes in the tumor, providing a reference basis for digital and individualized diagnosis and treatment decision-making for patients, and providing a solid scientific foundation and technical support for the development of precision medicine.

[0030] (2) The lightweight deep learning model for identifying tumor patches and the lightweight deep learning model for identifying lymphocyte aggregation patches have the following characteristics: ① Efficient and accurate feature extraction: shallow edge texture is quickly extracted through ordinary convolution in the initial layer, and feature extraction is gradually deepened through four residual separation convolution blocks, combining the lightweight characteristics of separable convolution and the gradient stability mechanism of residual connection, which effectively avoids the degradation problem of deep network while ensuring the integrity of feature extraction, providing high-quality feature support for subsequent classification tasks. ② Significant reduction in parameter and calculation cost: the parameter amount is reduced to 1 / 9 of the ordinary convolution, significantly reducing the model parameter amount and calculation amount (FLOPs), the parameter amount is reduced by 99% compared with VGG-16 and by 95% compared with ResNet-50, the calculation amount is reduced by 94% compared with VGG-16 and by 74% compared with ResNet-50, significantly reducing the hardware resource demand of model training and deployment. ③ Enhanced feature expression ability, excellent generalization ability and classification performance: the compression excitation module is introduced, which learns the channel weight adaptively and strengthens the key feature channel and suppresses the redundant channel, so that the model focuses more on the core features related to the task, improving the relevance and effectiveness of feature expression; the global average pooling layer reduces the dimension while retaining the global features, reducing the risk of overfitting and improving the generalization ability of the model, which can maintain the same accuracy as ResNet-50 (≈93%) in similar binary classification tasks, ensuring high-precision classification results. ④ Outstanding deployment advantages: under the premise of ensuring high precision, the model inference delay and memory usage are only 1 / 5-1 / 7 of ResNet-50, achieving a balance between high precision and ultra-low resources, making it easier to deploy and apply in resource-constrained devices or scenarios, especially suitable for clinical pathological image analysis and other practical scenarios that require real-time performance and hardware conditions. BRIEF DESCRIPTION OF DRAWINGS

[0031] The exemplary embodiments of the present application can be more completely understood in reference to the following drawings:

[0032] Figure 1 The flowchart of the automatic quantification method of intratumoral tumor infiltrating lymphocytes provided by the embodiments of the present application.

[0033] Figure 2 The training results of the lightweight deep learning model for patch classification of the embodiments of the present application, wherein:

[0034] Figure 2 A in the above formula is the manual annotation result of the bladder cancer staining pathological image, highlighting the areas containing tumor patches and lymphocyte aggregation patches;

[0035] Figure 2 B in the above formula is the ROC curve of the lightweight deep learning model for identifying tumor patches;

[0036] Figure 2C in the figure is the ROC curve of the lightweight deep learning model for identifying lymphocyte aggregation color blocks.

[0037] Figure 3 Gradient-weighted class activation mapping visualization results of tumor color blocks, non-tumor color blocks, lymphocyte aggregation color blocks and non-lymphocyte aggregation color blocks of the embodiments of the present application, wherein:

[0038] Figure 3 A in the figure is a tumor color block;

[0039] Figure 3 B in the figure is a lymphocyte aggregation color block;

[0040] Figure 3 C in the figure is a region where a tumor color block and a lymphocyte aggregation color block coexist;

[0041] Figure 3 D in the figure is a region of a non-tumor color block and a non-lymphocyte aggregation color block.

[0042] Figure 4 Calculation of the proportion of the overlapping area of tumor color blocks and lymphocyte aggregation color blocks to the total tumor color block area, and correlation analysis with tumor-infiltrating lymphocytes in tumors of the embodiments of the present application, wherein:

[0043] Figure 4 A in the figure is a calculation method of the proportion of the overlapping area of tumor color blocks and lymphocyte aggregation color blocks to the total tumor color block area;

[0044] Figure 4 B in the figure is a manual assessment of tumor-infiltrating lymphocytes in tumors by a pathologist;

[0045] Figure 4 C in the figure is that the proportion of the overlapping area of tumor color blocks and lymphocyte aggregation color blocks to the total tumor color block area is significantly positively correlated with the score of tumor-infiltrating lymphocytes in tumors of bladder cancer patients, R=0.960, P<0.001.

[0046] Figure 5 Relationship between the proportion of the overlapping area of tumor color blocks and lymphocyte aggregation color blocks to the total tumor color block area and the prognosis of bladder cancer patients of the embodiments of the present application, wherein:

[0047] Figure 5 A in the figure is a single factor Cox analysis of the proportion of the overlapping area of tumor color blocks and lymphocyte aggregation color blocks to the total tumor color block area;

[0048] Figure 5 B in the figure is a Kaplan-Meier survival curve showing that the proportion of the overlapping area of tumor color blocks and lymphocyte aggregation color blocks to the total tumor color block area is significantly positively correlated with the survival rate of bladder cancer patients, P=0.0012.

[0049] Figure 5 C in the multivariate Cox analysis shows that the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block in the total tumor color block area is an independent prognostic factor of the patient with bladder cancer, P=0.003.

[0050] Figure 6 It is a structural schematic diagram of the lightweight deep learning model of the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the technical problems, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs; the terms used herein are only for the purpose of describing specific embodiments and not limiting the present application.

[0053] As shown in Figure 1 The embodiment of the present application provides an automatic quantification method of lightweight intratumoral tumor infiltrating lymphocytes, which comprises the following steps:

[0054] Step S1: acquiring and pre-processing the bladder cancer staining pathological image to obtain effective color blocks, the effective color blocks including tumor color blocks, non-tumor color blocks, lymphocyte aggregation color blocks and non-lymphocyte aggregation color blocks;

[0055] After labeling the effective color blocks, a training set and a test set of the effective color blocks are constructed;

[0056] Two independent lightweight deep learning models are constructed, the lightweight deep learning model being composed of an input layer, a start layer, four residual separated convolution blocks, a global average pooling layer and a full connection layer in series, the lightweight deep learning model being trained based on the training set and the test set of the effective color blocks to obtain two independent lightweight deep learning models respectively identifying the tumor color blocks and the lymphocyte aggregation color blocks;

[0057] Step S2: inputting the pathological image of the patient with bladder cancer, using the two independent lightweight deep learning models respectively identifying the tumor color blocks and the lymphocyte aggregation color blocks to identify the tumor color blocks and the lymphocyte aggregation color blocks of the pathological image of the patient with bladder cancer; and calculating the proportion of the overlapping area of the tumor color blocks and the lymphocyte aggregation color blocks in the total tumor color block area according to the spatial coordinates of the tumor color blocks and the lymphocyte aggregation color blocks of the pathological image of the patient with bladder cancer;

[0058] Step S3: According to the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area, the proportion of tumor-infiltrating lymphocytes in the tumor of the bladder cancer patient is evaluated and / or a prognosis diagnosis is made for the bladder cancer patient.

[0059] In some embodiments, in step S1, the training set and test set of the effective color block include the training set and test set of the tumor color block and the non-tumor color block, and the training set and test set of the lymphocyte aggregation color block and the non-lymphocyte aggregation color block.

[0060] In some embodiments, step S1: training a lightweight deep learning model based on the training set and test set of the effective color block, obtaining two independent lightweight deep learning models for identifying tumor color blocks and lymphocyte aggregation color blocks, specifically:

[0061] Step S101: Based on the training set and test set of the tumor color block and the non-tumor color block, and the training set and test set of the lymphocyte aggregation color block and the non-lymphocyte aggregation color block, two initial lightweight deep learning models are trained respectively to obtain a lightweight deep learning model for identifying tumor color blocks and a lightweight deep learning model for identifying lymphocyte aggregation color blocks;

[0062] Step S102: Visualize the weight area of the lightweight deep learning model for identifying tumor color blocks and the lightweight deep learning model for identifying lymphocyte aggregation color blocks by gradient-weighted class activation mapping;

[0063] Step S103: Collect known bladder cancer pathological images as an external validation set to verify the identification efficiency of the lightweight deep learning model for identifying tumor color blocks and the lightweight deep learning model for identifying lymphocyte aggregation color blocks for tumor color blocks and lymphocyte aggregation color blocks, respectively.

[0064] The gradient-weighted class activation mapping visualization results of the tumor color block, the non-tumor color block, the lymphocyte aggregation color block and the non-lymphocyte aggregation color block are shown in Figure 3

[0065] In some embodiments, in step S1, the lightweight deep learning model is composed of an input layer, a starting layer, four residual separated convolution blocks, a global average pooling layer and a full connection layer in series, specifically:

[0066] Input layer: used to receive the bladder cancer pathological image and transmit it to the starting layer;

[0067] ​The starting layer is provided with a two-dimensional convolution layer containing 32 3x3 convolution kernels, which quickly extracts shallow edge texture through ordinary convolution, increases the channel to 32, batch normalization to suppress internal covariate shift, ReLU activation function to introduce nonlinearity, and 2x2 two-dimensional maximum pooling to reduce the spatial size by 2, reduce the amount of calculation and bring a certain translation invariance;

[0068] The four residual separable convolution blocks each include a main path, a residual branch and a compression excitation reinforcement module; the main path sequentially passes through 3x3 separable convolution, batch normalization, ReLU activation function and 3x3 separable convolution for feature extraction; the residual branch adds the residual of the main path output after adjusting the channel through 1x1 convolution and batch normalization; the compression excitation reinforcement module generates channel weights through global average pooling and multiplies them with the original features to adaptively strengthen important channels and suppress redundant channels; and finally outputs a 64x64x128 feature matrix;

[0069] The global average pooling layer converts the 64x64x128 feature matrix into a 128-dimensional vector through two-dimensional global average pooling, thereby improving the generalization ability;

[0070] The fully connected layer is composed of a feature dimension increasing layer, a random inactivation layer and a classification fully connected layer in series, and the classification result is obtained by processing through a Softmax activation function.

[0071] Compared with the traditional lightweight deep learning model, the parameter quantity of the lightweight deep learning model of the embodiment is reduced by 99% compared with VGG-16, and by 95% compared with ResNet-50, the calculation amount (FLOPs) is reduced by 94% compared with VGG-16, and by 74% compared with ResNet-50. In the same type of binary classification task, the lightweight deep learning model of the embodiment maintains the same accuracy (≈93%) as ResNet-50, but the inference delay and memory occupancy are only 1 / 5-1 / 7 of the latter, thereby achieving the deployment advantage of high precision + ultra-low resource.

[0072] In some embodiments, the bladder cancer pathological image received by the input layer is a 512x512 pixel, 3-channel RGB image.

[0073] In some embodiments, in each residual separable convolution block, the main path first passes through a 3*3 separable convolution, which reduces the parameter quantity to 1 / 9 of the ordinary convolution, and then performs a 1*1 convolution; then batch normalization is performed, nonlinearity is introduced through a ReLU activation function, and then a 3*3 separable convolution is performed again for feature extraction; the residual branch adjusts the number of channels through a 1*1 convolution, so that it matches the number of channels of the output features of the main path, and after batch normalization, the output features of the main path are added to the residual branch to make the gradient directly return, avoid the degradation of the deep network, and keep the gradient stable even if the model depth is increased; the compression excitation reinforcement module generates channel weights through global average pooling and multiplies them with the original features; the four residual separable convolution blocks are connected in turn to continuously extract, optimize and strengthen the features, and finally output a 64*64*128 feature matrix.

[0074] In some embodiments, in step S3, the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is positively correlated with the proportion of tumor-infiltrating lymphocytes in the tumor of the bladder cancer patient.

[0075] In some embodiments, in step S3, the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is positively correlated with the survival rate of the bladder cancer patient.

[0076] In some embodiments, step S1 specifically comprises the following steps: 301 bladder cancer hematoxylin-eosin (HE) staining pathological images are collected, 50 of which are randomly selected, and two professional pathologists manually label the tumor and lymphocyte aggregation area according to the Chinese Clinical Diagnosis and Treatment Guidelines for Tumors and the Chinese Standard Diagnosis and Treatment Quality Control Index Standard for Primary Colorectal Cancer, and convert the labeled images into labeled images; the digitally labeled images are cut into 512*512 pixel color blocks, and the color blocks with incorrect size, low clarity, blank background or memory of 0 are removed, and then the color blocks are divided into a tumor color block and a non-tumor color block dataset, a lymphocyte aggregation color block and a non-lymphocyte aggregation color block dataset according to the manually labeled mask, and then randomly allocated according to the 8:2 principle to form a training set and a test set for training two independent lightweight deep learning models, and then collect other bladder cancer HE staining pathological images not involved in the training as a validation set to verify the recognition efficiency of the two independent lightweight deep learning models, obtain a lightweight deep learning model for recognizing tumor color blocks and a lightweight deep learning model for recognizing lymphocyte aggregation color blocks, and the training results are as shown in Figure 2

[0077] ​In some embodiments, the pathologist manually evaluates the tumor infiltrating lymphocytes in the tumor according to the solid tumor tumor infiltrating lymphocyte (TILs) evaluation guidelines, and performs correlation analysis on the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area and the tumor infiltrating lymphocytes in the tumor.

[0078] The calculation of the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area and the correlation analysis with the tumor infiltrating lymphocytes in the tumor are shown in Figure 4 The results show that the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is significantly positively correlated with the score of the tumor infiltrating lymphocytes in the tumor of the bladder cancer patient, R=0.960, P<0.001.

[0079] In some embodiments, the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is correlated with the prognosis information of the patient to verify the prediction ability of the index constructed by the application on the prognosis of the bladder cancer patient.

[0080] The relationship between the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area and the prognosis of the bladder cancer patient is shown in Figure 5 The results show that the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is significantly positively correlated with the survival rate of the bladder cancer patient, P=0.0012.

[0081] In some embodiments, the structural diagram of the lightweight deep learning model is shown in Figure 6 The input layer, the starting layer, the four residual separated convolution blocks, the global average pooling layer and the full connection layer are specifically as follows:

[0082] The input layer: used for receiving the bladder cancer pathological images, stretching all the received bladder cancer pathological images to the size of 512x512x3 in equal proportions, and then transmitting them to the starting layer;

[0083] Starting layer: consists of a two-dimensional convolution layer, a batch normalization layer, and a pooling layer; the two-dimensional convolution layer contains a {3×3×3×32} traditional convolution kernel, receives the [512×512×3] feature map output by the output layer and outputs a [512×512×32] feature map, which is used to extract shallow edge textures and increase the number of channels to 32; the batch normalization layer receives the [512×512×32] feature map output by the two-dimensional convolution layer, calculates the mean and variance independently for each channel, and outputs a normalized [512×512×32] feature map through standardization and affine transformation to suppress internal covariate shift, and then introduces nonlinearity through the ReLU activation function; the pooling layer uses a {2×2} two-dimensional maximum pooling, receives the [512×512×32] feature map output by the batch normalization layer, and then outputs a [256×256×32] feature map to reduce the spatial size by 2 times;

[0084] Continuous residual separation convolution layer composed of four serial residual separation convolution blocks: Each residual separation convolution block includes a main path, a residual branch and a compression excitation enhancement module, wherein the main route is composed of depth-separable convolution, batch normalization and depth-separable convolution; depth-separable convolution can be split into depth-wise convolution (3×3×N×1, channel-by-channel convolution, N is the number of input channels) and point-by-point convolution (1×1×N×N', point-by-point convolution, N' is the number of output channels). When deepening the same model depth, compared with traditional convolution, depth-wise separable convolution can greatly reduce parameters and computational complexity; residual branch separation There are two cases: identity mapping and projection mapping. When the number of input channels is the same as the number of output channels, the identity mapping directly adds the original feature map to the main output without any additional operations. It only jumps and adds it to the main output to ensure lossless information flow. When the number of input channels is different from the number of output channels, the projection mapping first uses traditional convolution [1×1×N, step size = 2] and batch normalization to align the number of channels and match the spatial size, and then adds it to the main output. In the compression excitation enhancement module, the spatial information of each channel is first compressed into a scalar through global average pooling to obtain the channel description vector [Z1, Z2,…, Z N ],Z N Represents the global average pooling value of the Nth channel, and then inputs the channel description vector into a two-layer fully connected network, ReLU activation function, Sigmoid activation function to finally obtain the channel weight [S1, S2, ..., S N ], S N Represents the importance weight of the Nth channel. The channel weight is returned and multiplied with the feature map channel by channel to ensure that the output feature map size is exactly the same as the input feature map size. All channels are weighted, important channels are amplified, and unimportant channels are suppressed.

[0085] The main path in the first residual separable convolution block receives the [256x256x32] feature map output by the starting layer, and outputs a [256x256x32] feature map after a first depth separable convolution {3x3x32x1, stride=1}, batch normalization, a ReLU activation function, a second depth separable convolution {3x3x32x1, stride=1}, and batch normalization; the residual branch is an identity mapping, and the input and output are both [256x256x32] feature maps; after being added element by element and channel by channel, a [256x256x32] feature map after residual fusion is obtained; the compression excitation reinforcement module receives the [256x256x32] feature map after residual fusion, obtains a channel description vector [Z1, Z2, …, Z 32 ] through global average pooling, and finally obtains a channel weight [S1, S2, …, S 32 ] after processing by two fully connected networks, a ReLU activation function, and a Sigmoid activation function; the channel weight is fed back and multiplied with the feature map channel by channel to obtain a [256x256x32] feature map output by the first residual separable convolution block.

[0086] The main path in the second residual separable convolution block receives the [256x256x32] feature map output by the first residual separable convolution block, and outputs a [128x128x64] feature map after a first depth separable convolution {3x3x32x1, stride=2}, batch normalization, a ReLU activation function, a second depth separable convolution {3x3x64x1, stride=1}, and batch normalization; the residual branch is a projection mapping that adjusts the input [256x256x32] to [128x128x64] through a 1x1 convolution {1x1x32x64, stride=2}, and after batch normalization, is added element by element and channel by channel with the main path to obtain a [128x128x64] feature map after residual fusion; the compression excitation reinforcement module receives the [128x128x64] feature map after residual fusion, obtains a channel description vector [Z1, Z2, …, Z 64 ] through global average pooling, and finally obtains a channel weight [S1, S2, …, S 64 ] after processing by two fully connected networks, a ReLU activation function, and a Sigmoid activation function; the channel weight is fed back and multiplied with the feature map channel by channel to obtain a [128x128x64] feature map output by the second residual separable convolution block.

[0087] The main path in the third residual separable convolution block receives the [128×128×64] feature map output by the second residual separable convolution block, and outputs a [128×128×64] feature map after a first depth separable convolution {3×3×64×1, stride=1}, batch normalization, a ReLU activation function, a second depth separable convolution {3×3×64×1, stride=1}, and batch normalization; the residual branch is an identity mapping, and the input and output are both [128×128×64], which are added to each other channel by channel and element by element, to obtain a [128×128×64] feature map after residual fusion; the compression and excitation reinforcement module receives the [128×128×64] feature map after residual fusion, obtains a channel description vector [Z1, Z2, …, Z 64 ] through global average pooling, and obtains a channel weight [S1, S2, …, S 64 ] after processing by two fully connected networks, a ReLU activation function, and a Sigmoid activation function, and multiplies the channel weight back to the feature map channel by channel to obtain a [128×128×64] feature map output by the third residual separable convolution block;

[0088] The main path in the fourth residual separable convolution block receives the [128×128×64] feature map output by the third residual separable convolution block, and outputs a [64×64×128] feature map after a first depth separable convolution {3×3×64×1, stride=2}, batch normalization, a ReLU activation function, a second depth separable convolution {3×3×128×1, stride=1}, and batch normalization; the residual branch is a projection mapping that adjusts the input [128×128×64] to [64×64×128] through a 1×1 convolution {1×1×64×128, stride=2}, and is added to the main path channel by channel and element by element after batch normalization, to obtain a [64×64×128] feature map output by the fourth residual separable convolution block;

[0089] The global average pooling layer receives the [64×64×128] feature map output by the fourth residual separable convolution block, and obtains a channel description vector [F1, F2, …, F 128 ] of 128 dimensions after global average pooling;

[0090] The fully connected layer is composed of a feature dimension increasing layer, a random inactivation layer, and a classification fully connected layer; the 128-dimensional vector output by the global average pooling layer is input into the feature dimension increasing layer, converted by a weight matrix {256×128} and a bias {256}, processed by batch normalization and a ReLU activation function, and then output as a

[256] -dimensional feature vector; then input into the random inactivation layer, in which the neurons are randomly shielded at a ratio of 0.5, and the output is kept at

[256] dimensions; then input into the classification fully connected layer, and a class probability vector [P1, P2, …, Pclass_num ], and finally output [class_num] -dimensional classification results, class_num: total number of classes, [class_num] -dimension: dimension of the output vector is equal to the total number of classes class_num.

[0091] In some embodiments, the present embodiment provides a computer device, comprising at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the automatic quantification method of lightweight intratumoral tumor infiltrating lymphocytes.

[0092] In some embodiments, the present embodiment provides a computer readable storage medium having stored thereon a computer program, and the computer program is executed by a processor to implement the automatic quantification method of lightweight intratumoral tumor infiltrating lymphocytes.

[0093] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for automatic quantification of lightweight intratumoral tumor infiltrating lymphocytes, characterized by: The method comprises the following steps: Step S1: obtaining and preprocessing the bladder cancer staining pathological image, and obtaining effective color blocks, the effective color blocks including tumor color blocks, non-tumor color blocks, lymphocyte aggregation color blocks and non-lymphocyte aggregation color blocks; After the effective color blocks are labeled, a training set and a test set of the effective color blocks are constructed; Two independent lightweight deep learning models are constructed, the lightweight deep learning model being composed of an input layer, a starting layer, four residual separated convolution blocks, a global average pooling layer and a full connection layer, the lightweight deep learning model being trained based on the training set and the test set of the effective color blocks, and two independent lightweight deep learning models respectively identifying tumor color blocks and lymphocyte aggregation color blocks being obtained; The lightweight deep learning model is composed of an input layer, a starting layer, four residual separated convolution blocks, a global average pooling layer and a full connection layer, and specifically comprises: The input layer is used for receiving the bladder cancer pathological image and transmitting it to the starting layer; The starting layer is provided with a two-dimensional convolution layer containing 32 3*3 convolution kernels, shallow edge texture is quickly extracted through ordinary convolution, the channel is upgraded to 32, internal covariate shift is inhibited through batch normalization, nonlinearity is introduced through a ReLU activation function, and the spatial size is reduced by 2 through 2*2 two-dimensional maximum pooling; Each residual separated convolution block includes a main path, a residual branch and a compression excitation reinforcement module; the main path sequentially passes through a 3*3 separable convolution, batch normalization, a ReLU activation function and a 3*3 separable convolution for feature extraction; the residual branch adds a residual error after adjusting the channel through a 1*1 convolution and batch normalization; the compression excitation reinforcement module generates a channel weight through global average pooling and multiplies it with the original feature; and finally a 64*64*128 feature matrix is outputted; The global average pooling layer converts the 64*64*128 feature matrix into a 128-dimensional vector through two-dimensional global average pooling; The full connection layer is composed of a feature dimension increasing layer, a random inactivation layer and a classification full connection layer, and a classification result is obtained through a Softmax activation function; Step S2: inputting the pathological image of the patient with bladder cancer, using the two independent lightweight deep learning models respectively identifying tumor color blocks and lymphocyte aggregation color blocks to identify the tumor color blocks and lymphocyte aggregation color blocks of the pathological image of the patient with bladder cancer; and calculating the proportion of the overlapping area of the tumor color blocks and the lymphocyte aggregation color blocks to the total tumor color block area according to the spatial coordinates of the tumor color blocks and the lymphocyte aggregation color blocks of the pathological image of the patient with bladder cancer; Step S3: evaluating the score of tumor-infiltrating lymphocytes in the tumor of the patient with bladder cancer according to the proportion of the overlapping area of the tumor color blocks and the lymphocyte aggregation color blocks to the total tumor color block area.

2. The method of claim 1, wherein the method is for automatic quantification of light-weight intratumoral tumor infiltrating lymphocytes. In step S1, the training set and the test set of the effective color blocks include the training set and the test set of the tumor color blocks and the non-tumor color blocks, and the training set and the test set of the lymphocyte aggregation color blocks and the non-lymphocyte aggregation color blocks.

3. The method of claim 2, wherein the method is for automatic quantification of light-weight intratumoral tumor infiltrating lymphocytes. Step S1: training a lightweight deep learning model based on the training set and test set of effective color blocks, obtaining two independent lightweight deep learning models for identifying tumor color blocks and lymphocyte aggregation color blocks respectively, specifically: Step S101: training two initial lightweight deep learning models based on the training set and test set of tumor color blocks and non-tumor color blocks, and the training set and test set of lymphocyte aggregation color blocks and non-lymphocyte aggregation color blocks, to obtain a lightweight deep learning model for identifying tumor color blocks and a lightweight deep learning model for identifying lymphocyte aggregation color blocks; Step S102: visualizing the weight regions of the lightweight deep learning model for identifying tumor color blocks and the lightweight deep learning model for identifying lymphocyte aggregation color blocks through gradient-weighted class activation mapping; Step S103: collecting known bladder cancer pathological images as an external validation set to verify the identification efficiency of the lightweight deep learning model for identifying tumor color blocks and the lightweight deep learning model for identifying lymphocyte aggregation color blocks on tumor color blocks and lymphocyte aggregation color blocks respectively.

4. The method of claim 3, wherein the method is for automatic quantification of light-weight intratumoral tumor infiltrating lymphocytes. The bladder cancer pathological image received by the input layer is a 512x512 pixel, 3-channel RGB image.

5. The method of automatic quantification of light-weight intratumoral tumor infiltrating lymphocytes according to claim 4, wherein: In each residual separable convolution block, the main path first passes through a 3x3 separable convolution, which first performs channel-wise convolution and then 1x1 convolution; then batch normalization is performed, nonlinearity is introduced through a ReLU activation function, 3x3 separable convolution is performed again for feature extraction; the residual branch adjusts the number of channels through 1x1 convolution to match the number of channels of the output features of the main path, and after batch normalization, the residual branch is added to the output features of the main path; the compression excitation reinforcement module generates channel weights through global average pooling and multiplies them with the original features; The four residual separable convolution blocks are connected in series, continuously extracting, optimizing and strengthening the features, and finally outputting a 64x64x128 feature matrix.

6. The method of claim 5, wherein the method is for automatic quantification of light-weighted intratumoral tumor infiltrating lymphocytes. In step S3, the proportion of the overlapping area of the tumor color block and the lymphocyte aggregation color block to the total tumor color block area is positively correlated with the fraction of tumor-infiltrating lymphocytes in the tumor of the bladder cancer patient.

7. A computer device, characterized by: The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

8. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the method of any one of claims 1 to 6.

Images