Method for evaluating maturity of three-level lymphatic structure of gastric cancer based on deep learning

Abstract

The invention discloses a gastric cancer three-level lymphatic structure maturity evaluation method based on deep learning, and the method comprises the steps: taking a pathological full-slice image as input, carrying out the segmentation of a tumor region and a surrounding region through a U-Net network, carrying out the rough positioning of TLSs through a YOLOv5X detection network, and cutting a local image, and performing fine segmentation on the TLSs and the GC in the TLSs by using a segmentation network based on the SAM architecture. On the basis of the segmentation result, the GTR value of the GC in each TLSs is calculated, the T score of the tumor area and the P score of the surrounding area are further obtained, finally, the M score of the maturity of the TLSs is calculated through Cox regression weighted fusion, and quantitative evaluation of the maturity of the TLSs is achieved. According to the method, efficiency and precision can be taken into consideration, comprehensive coverage of the TLSs candidate region is ensured, a reliable GTR value is obtained through fine segmentation, and the method has the advantages of automation, objectiveness and high efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and medical image analysis, and particularly relates to a tertiary lymphoid structure maturity evaluation method for gastric cancer based on deep learning. BACKGROUND

[0002] Tertiary lymphoid structures (TLSs) are widely present in tumor microenvironments and have important influences on tumor immune responses and patient prognosis. In the field of medical pathology image analysis, pathological whole-slide images are a key diagnostic tool for observing and evaluating the morphological features and maturity of TLSs. The maturity of TLSs is usually determined by whether a germinal center (GC) is formed, and the presence of a GC is considered a core indicator of TLS maturity.

[0003] Traditional TLS maturity evaluation methods mainly rely on the experience and intuitive judgment of pathologists, observing pathological whole-slide images to determine whether a GC exists in TLSs. However, this method has certain limitations, such as strong subjectivity, insufficient quantification, susceptibility to selection bias, and difficulty in achieving efficient analysis on large-scale whole-slide images. With the development of deep learning technology, automated pathological image analysis methods have gradually become a research hotspot. These methods can automatically identify and segment pathological regions through neural networks, thereby improving the efficiency and objectivity of analysis. However, existing methods still have deficiencies in TLS maturity evaluation:

[0004] On the one hand, only a rough classification based on the presence or absence of a GC is possible, and there is a lack of quantification of the proportion of GCs in TLSs.

[0005] On the other hand, existing methods often ignore the spatial distribution differences of TLSs in the tumor interior region and the tumor peripheral region.

[0006] Therefore, there is an urgent need for a TLS maturity quantitative scoring method that is highly accurate, efficient, and repeatable. This method not only can automatically identify and segment TLSs and GCs in whole-slide images, but also can construct a quantitative scoring system and generate a more scientific TLS maturity score by considering spatial distribution characteristics, which can be used to assist in patient prognosis evaluation and clinical decision-making. SUMMARY

[0007] In order to overcome the deficiencies in the prior art, the application provides a gastric cancer tertiary lymphatic structure maturity evaluation method based on deep learning, which can automatically identify and segment TLSs and GC inside the TLSs in a pathological whole section image, calculate the GC area ratio GTR (GC-to-TLSs ratio, GC-to-TLSs area ratio) value in the TLSs, and construct a unified TLSs maturity scoring system combined with the distribution characteristics of the TLSs in the tumor internal region and the tumor peripheral region, thereby realizing efficient, accurate and repeatable patient prognosis evaluation.

[0008] In order to achieve the above application purposes and solve the technical problems, the technical solutions adopted are as follows:

[0009] A gastric cancer tertiary lymphatic structure maturity evaluation method based on deep learning comprises the following steps:

[0010] Step 1, image preprocessing: the pixel value normalization and size unification of the input gastric cancer pathological whole section image are performed, and the image is scaled to a preset standard resolution;

[0011] Step 2, tumor region segmentation: the U-Net network is used for semantic segmentation of the pathological whole section image to obtain the boundary of the tumor region T and the tumor peripheral region P;

[0012] Step 3, TLSs coarse positioning: the detection network YOLOv5X based on the recall rate priority training strategy is used to locate the potential TLSs in the whole section image, and the local TLSs image is obtained after the detected region boundary is extended by 30 pixels and cropped;

[0013] Step 4, TLSs and GC fine segmentation: the encoder-decoder segmentation network based on SAM is used to perform pixel-level segmentation on the local TLSs image obtained in step 3 to extract the TLSs region and the GC region inside the TLSs region;

[0014] Step 5, GTR value calculation: for each TLSs, the ratio of the GC area to the TLSs area is calculated to obtain the GTR value of the GC area to the area of the TLSs, wherein the GTR value corresponding to the TLSs without detected GC is zero;

[0015] Step 6, score generation: according to the results obtained in step 5, the average GTR value of all TLSs in the tumor region T is calculated as the T score, and the average GTR value of all TLSs in the tumor peripheral region P is calculated as the P score;

[0016] Step 7, TLSs maturity score: based on the weighted coefficients of multivariate Cox regression analysis, the T score and the P score are weighted and fused to obtain the TLSs maturity comprehensive score M score, which is used for patient prognosis evaluation.

[0017] Furthermore, in step 2, the semantic segmentation U-Net network uses the Dice coefficient as an evaluation index for model segmentation performance.

[0018] Furthermore, in step 3, the coarse localization network for TLSs adopts a recall-first training strategy, that is, during the model training process, by reducing the prediction confidence threshold, adjusting the weight of the loss function, and optimizing the non-maximum suppression parameter, in order to ensure the comprehensive detection of TLSs candidate regions.

[0019] Furthermore, in step 3, the TLSs coarse localization network uses the mean box accuracy (bbox mAP) and recall as performance evaluation metrics for model detection.

[0020] Furthermore, in step 4, the TLSs and GC fine segmentation network use a VisionTransformer pre-trained with SAM as the network encoder.

[0021] Furthermore, in step 4, the TLSs and GC fine-grained segmentation network employ a lightweight convolutional neural network as the decoder. The decoder consists of two depthwise separable convolutional modules and a 1×1 convolutional layer, mathematically represented as follows:

[0022] X o =Conv 1×1 (Conv DBR (U(Conv DBR (X i ))))

[0023] Among them, variable X i Represents the input feature map, Conv DBR (·) indicates a depthwise separable convolutional module, containing convolution operations, followed by batch normalization and ReLU activation. U(·) indicates an upsampling operation. Conv 1×1 Represents a 1×1 convolution, X o This indicates the output feature map.

[0024] Furthermore, in step 4, the SAM-based segmentation network uses the Dice coefficient as an evaluation index for model segmentation performance.

[0025] Furthermore, in step 5, the GTR value, which is the ratio of the GC area of ​​a single TLS to the area of ​​the TLSs it belongs to, is mathematically expressed as follows:

[0026]

[0027] Wherein, variable S GC Represents the total area of ​​pixels belonging to the germinal centers within the tertiary lymphoid structure; variable STLS total area of pixels representing tertiary lymphoid structures.

[0028] Further, in step 7, the single TLSs maturity comprehensive score M score is mathematically expressed as follows:

[0029] M score = a1T score + a2P score

[0030] Wherein, the variable T score represents the average GTR value of all TLSs in the tumor area T region, P score represents the average GTR value of all TLSs in the tumor peripheral area P region, and variables a1 and a2 represent the corresponding regression coefficients of T score and P score in Cox regression respectively.

[0031] Compared with the prior art, the present application has the following advantages and positive effects:

[0032] 1. The present application can simultaneously recognize and accurately outline the tertiary lymphoid structures and their internal germinal centers by performing multi-scale feature extraction and automatic segmentation on pathological whole slide images through a deep learning model, overcoming the low efficiency and strong subjectivity of traditional manual interpretation.

[0033] 2. The present application adopts a phased process of U-Net segmentation, YOLOv5X coarse positioning and fine segmentation based on a SAM architecture model, realizing efficient and accurate identification and segmentation of TLSs and their internal GCs.

[0034] 3. The present application introduces a recall rate priority strategy in YOLOv5X training, effectively avoiding TLSs missed detection, ensuring comprehensive coverage of candidate regions and reliability of scoring.

[0035] 4. The automatic scoring process of the quantitative index based on GC / TLSs area ratio GTR proposed by the present application simplifies manual operation, reduces the workload of pathologists, and improves analysis efficiency and consistency. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. In the drawings:

[0037] Figure 1is a flowchart of a deep learning-based gastric cancer tertiary lymphoid structure maturity evaluation method according to an embodiment of the present application;

[0038] Figure 2 is a model network architecture diagram according to an embodiment of the present application;

[0039] Figure 3 is a tumor region segmentation result schematic diagram according to an embodiment of the present application;

[0040] Figure 4 is a tertiary lymphoid structure preliminary detection result schematic diagram according to an embodiment of the present application;

[0041] Figure 5 is a tertiary lymphoid structure and germinal center fine segmentation result schematic diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0043] As shown in Figures 1-5 The present embodiment discloses a deep learning-based gastric cancer tertiary lymphoid structure maturity evaluation method, comprising the following steps:

[0044] Step 1, image preprocessing: the pixel value normalization and size unification processing are performed on the input gastric cancer pathological whole section image, and it is scaled to a preset standard resolution of 2048x2048;

[0045] Step 2, tumor region segmentation: a U-Net (U-shaped Convolutional Neural Network) network is used to perform semantic segmentation on the pathological whole section image to obtain the boundary of the intratumoral region T and the peritumoral region P;

[0046] Specifically, in step 2, the semantic segmentation U-Net network uses the Dice (Dice Similarity Coefficient) coefficient as the model segmentation performance evaluation index.

[0047] Step 3, TLSs rough positioning: the detection network of YOLOv5X based on the recall priority training strategy is used to locate potential TLSs in the whole slice image, and after extending 30 pixel range outside the detected region boundary, the local TLSs image is cropped;

[0048] Specifically, in step 3, the TLSs rough positioning network adopts a recall priority training strategy, that is, in the model training process, by reducing the prediction confidence threshold, adjusting the loss function weight and optimizing the non-maximum suppression parameter method, to ensure the comprehensive detection of TLSs candidate regions.

[0049] Further, in step 3, the TLSs rough positioning network adopts bounding boxes mean Average Precision (bbox mAP) and Recall as model detection performance evaluation indexes.

[0050] Step 4, TLSs and GC fine segmentation: the local TLSs image obtained in step 3 is subjected to pixel-level segmentation by using an encoder-decoder segmentation network based on SAM (Segment Anything Model), and the TLSs region and the GC region inside it are extracted respectively;

[0051] Specifically, in step 4, the TLSs and GC fine segmentation network adopts a Vision Transformer (ViT) pre-trained by SAM as a network encoder.

[0052] Further, in step 4, the TLSs and GC fine segmentation network adopts a lightweight convolutional neural network (CNN) as a decoder, which is composed of two deep separable convolution modules and a 1×1 convolution layer, and is mathematically represented as follows:

[0053] X o =Conv 1×1 (Conv DBR (U(Conv DBR (X i ))))

[0054] Wherein, the variable X i represents the input feature map, Conv DBR (·) represents a deep separable convolution module, which contains convolution operation, followed by batch normalization and RELU activation, U(·) represents an upsampling operation, Conv 1×1 represents a 1×1 convolution, and X o represents the output feature map.

[0055] Further, in step 4, the SAM-based segmentation network adopts the Dice coefficient as the model segmentation performance evaluation index.

[0056] Step 5, GTR value calculation: for each TLSs, calculate the ratio of GC area to TLSs area, and obtain the GC area to TLSs area ratio GTR value, wherein the TLSs corresponding to the undetected GC has a GTR value of zero;

[0057] Specifically, in step 5, the GC area to TLSs area ratio GTR value of the single TLSs is mathematically expressed as follows:

[0058]

[0059] wherein the variable S GC represents the total area of pixels belonging to the germinal center within the tertiary lymphoid structure; the variable S TLS represents the total area of pixels of the tertiary lymphoid structure.

[0060] Step 6, score generation: according to the results obtained in step 5, calculate the average GTR value of all TLSs in the tumor region T as the T score, and calculate the average GTR value of all TLSs in the tumor peripheral region P as the P score;

[0061] Step 7, TLSs maturity score: based on the weighted coefficients of multivariate Cox (Cox Proportional Hazards Regression, Cox proportional hazards regression model) regression analysis, the T score and the P score are weighted and fused to obtain the TLSs maturity comprehensive score M score (maturity score), which is used for patient prognosis evaluation.

[0062] Specifically, in step 7, the single TLSs maturity comprehensive score M score is mathematically expressed as follows:

[0063] M score = α1T score + α2P score

[0064] wherein the variable T score represents the average GTR value of all TLSs in the tumor region T, P score represents the average GTR value of all TLSs in the tumor peripheral region P, and the variables α1 and α2 represent the regression coefficients corresponding to T score and P score in the Cox regression, respectively.

[0065] Figure 2is a model network architecture diagram of an embodiment of the present application The architecture. First, the U-Net network is used to realize the automatic segmentation of the tumor area and the surrounding area, then the YOLOv5X detection network is used to coarsely locate the three-level lymphatic structure in the whole slice image, and finally the SAM-based segmentation network is combined to finely segment the TLSs and the internal germinal center, so as to obtain the TLSs maturity score for subsequent GTR calculation and.

[0066] Figure 3 is a U-Net network tumor area segmentation result diagram of an embodiment of the present application The dark red area in the image is the tumor area, and the pink area outside the boundary is the tumor peripheral area.

[0067] Figure 4 is a three-level lymphatic structure preliminary detection result diagram of an embodiment of the present application Through the detection network based on YOLOv5X, multiple potential three-level lymphatic structure candidate regions are automatically identified and located in the pathological whole slice image, and the detection result is marked with a red frame.

[0068] Figure 5 is a three-level lymphatic structure and germinal center fine segmentation result diagram of an embodiment of the present application Through the segmentation network based on SAM, the three-level lymphatic structure area (green boundary inside area) and the internal germinal center area (red boundary inside area) can be accurately segmented.

[0069] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A deep learning-based method for assessing the maturity of three-tiered lymphatic structures in gastric cancer, characterized in that, Includes the following steps: Step 1, Image Preprocessing: Normalize the pixel values ​​and unify the size of the input gastric cancer pathological whole slice image, and scale it to the preset standard resolution; Step 2, Tumor region segmentation: Use the U-Net network to perform semantic segmentation on the whole pathological slice image to obtain the boundary between the tumor region T and the surrounding tumor region P. Step 3, Coarse localization of TLSs: The detection network of YOLOv5X, which uses a recall-first training strategy, locates potential TLSs in the full slice image, and then crops the image after extending the detected region by 30 pixels outside the boundary to obtain a local TLSs image; Step 4, Fine-grained segmentation of TLSs and GC: The local TLSs image obtained in Step 3 is segmented at the pixel level using an encoder-decoder segmentation network based on SAM, and the TLSs region and its internal GC region are extracted respectively. Step 5, GTR value calculation: For each TLSs, calculate the ratio of its GC area to the area of ​​the TLSs, and obtain the GTR value, which is the ratio of the GC area to the area of ​​the TLSs in which it is located. The GTR value corresponding to the TLSs for which no GC was detected is zero. Step 6, score generation: Based on the results obtained in Step 5, calculate the average GTR value of all TLSs in the T region of the tumor area as the T score, and the average GTR value of all TLSs in the P region of the surrounding tumor area as the P score. Step 7, TLSs Maturity Score: Based on the weighted coefficients of multivariate Cox regression analysis, the T score and P score are weighted and fused to obtain the comprehensive TLSs maturity score M score, which is used to achieve patient prognostic assessment.

2. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 2, the semantic segmentation U-Net network uses the Dice coefficient as an evaluation index for model segmentation performance.

3. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 3, the coarse localization network for TLSs adopts a recall-first training strategy, which means that during model training, the prediction confidence threshold is lowered, the weights of the loss function are adjusted, and the non-maximum suppression parameters are optimized to ensure the comprehensive detection of TLSs candidate regions.

4. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 3, the TLSs coarse localization network uses the mean box accuracy (bbox mAP) and recall as performance evaluation metrics for model detection.

5. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 4, the TLSs and GC fine segmentation network use a Vision Transformer pre-trained with SAM as the network encoder.

6. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 4, the TLSs and GC fine-grained segmentation network uses a lightweight convolutional neural network as the decoder. The decoder consists of two depthwise separable convolutional modules and a 1×1 convolutional layer, which can be mathematically represented as follows: X o =Conv 1×1 (Conv DBR (U(Conv DBR (X i )))) Among them, variable X i Represents the input feature map, Conv DBR (·) indicates a depthwise separable convolutional module, containing convolution operations, followed by batch normalization and ReLU activation. U(·) indicates an upsampling operation. Conv 1×1 Represents a 1×1 convolution, X o This indicates the output feature map.

7. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 4, the SAM-based segmentation network uses the Dice coefficient as the performance evaluation index for model segmentation.

8. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 5, the GTR value, which is the ratio of the GC area of ​​a single TLS to the area of ​​the TLSs it belongs to, is mathematically expressed as follows: Wherein, variable S GC Represents the total area of ​​pixels belonging to the germinal centers within the tertiary lymphoid structure; variable S TLS This represents the total pixel area of ​​the tertiary lymphoid structure.

9. The method for assessing the maturity of three-tiered lymphatic structures in gastric cancer based on deep learning according to claim 1, characterized in that, In step 7, the comprehensive maturity score M for a single TLS is mathematically represented as follows: M score =α1T score +α2P score Among them, variable T score P represents the average GTR value of all TLSs within the T region of the tumor. score The mean GTR value represents all TLSs within the P region surrounding the tumor, and variables α1 and α2 represent the T values ​​in the Cox regression, respectively. score and P score The corresponding regression coefficients.

Images