Method and system for predicting three-level lymphatic structure based on HE staining pathological image
Through the TLS-DSNet model, combined with CNN and Transformer branches, two-stage training strategy and loss function optimization are adopted to solve the data imbalance and receptive field problems in the detection of tertiary lymph structures in the existing technology, and achieve the super-large goal Effective detection and accurate prediction.
Patent Information
- Application Number
- CN202410914633.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-11-12
AI Technical Summary
In the detection of tertiary lymphatic structure targets at the pathology level, there are problems such as data imbalance, small receptive field, difficulty in capturing global image information, and task order and information contribution balance in multi-task learning, resulting in the model being Poor performance is performed when identifying super-large-sized tertiary lymphatic structures.
Using the TLS-DSNet model based on HE stained pathological images, combining the CNN branch with U-shaped structure and the 12-layer Transformer branch, through two-stage training strategy and loss function optimization, the CNN branch is used for semantic segmentation and scale prediction, and the Transformer branch performs global features. Extract and target detection is performed through the YOLO model to achieve effective prediction of tertiary lymphatic structure.
Enhance the adaptability and generalization ability of the model, break through the limitations of the receptive field range, realize effective detection of super-large targets, reduce calculation needs, avoid potential misleading of the auxiliary tasks on the main tasks, and improve prediction accuracy and recall rate.
Smart Images

Figure CN118941916A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for predicting tertiary lymphatic structure based on HE-stained pathological images. Background Art
[0002] In recent years, more and more clinical practices have chosen to use tissue biopsy to molecularly characterize tumors, that is, to quantitatively analyze the morphological information and tissue characteristics in histological whole slide images (WSI). Among the numerous histopathological structural markers, tertiary lymphoid structures (TLS) refer to organized ectopic lymphoid structures formed by the aggregation of lymphocytes in non-lymphoid organs under pathological and physiological conditions. They are commonly found in multiple tumor types and are important sites for the interaction between tumor cells and immune cells, and are closely related to the patient's prognosis. Mature TLS include B cell areas and germinal centers surrounded by T cells. Inducing the formation of TLS in tumor tissues allows T cells and B cells to develop and differentiate into effector cells and memory cells that can recognize and kill tumor cells, which can mediate anti-tumor immune responses and have important clinical significance.
[0003] In recent years, the rise and popularity of deep learning technology has introduced revolutionary tools and new ways of thinking in the field of medical image processing. Computer-aided diagnosis technology based on deep learning has become the key to improving the efficiency and accuracy of pathological image analysis. Convolutional neural networks are an important part of deep learning. They have been proven to automatically extract features that human observers cannot detect and can complete visual tasks without the support of expert knowledge and rich experience in related fields. At the same time, multi-task learning can improve the training efficiency of the model and enhance its generalization ability under limited data resources by transferring knowledge between tasks.
[0004] However, although existing deep learning algorithms have made some progress in the field of pathological image analysis, this technology still has the following defects in the target detection task of TLS at the pathological level:
[0005] First, due to the diverse sizes of TLS in tumor tissues, while most bounding boxes are concentrated in smaller sizes, there are still some significantly larger outlier targets. As a result, during the learning process, the model tends to generate prediction results for smaller anchor points due to data imbalance, resulting in prediction preference and ignoring the few large outlier targets in the data.
[0006] Second, the convolution kernels used in the CNN-based backbone network have a small receptive field and lack the ability to process the global representation of the image, making it difficult to capture a wider range of visual information, which limits the model's ability to recognize large targets, especially in the face of ultra-large-scale TLS detection.
[0007] Third, faced with the differences in the fine granularity of different annotations and the final optimization goals, how to balance the positive guidance and potential misleading of each auxiliary task to the main task, and reasonably plan the training order and information contribution of different task branches in model training, is a major challenge in multi-task learning. Summary of the invention
[0008] The purpose of the present invention is to provide a method and system for predicting tertiary lymphatic structure based on HE-stained pathological images.
[0009] The purpose of the present invention can be achieved by the following technical solutions:
[0010] A method for predicting tertiary lymphatic structures based on HE-stained pathological images comprises the following steps:
[0011] Obtain HE-stained pathological images, perform preprocessing and image annotation, and construct an image dataset;
[0012] Based on the image dataset, a two-stage training strategy was adopted to train the TLS-DSNet model, which included a U-shaped CNN branch and a 12-layer Transformer branch. HE-stained pathological images were used as model input, and the results of the two branches were obtained and then fused to predict the tertiary lymphatic structure.
[0013] The HE-stained pathological images to be predicted are preprocessed and then input into the trained TLS-DSNet model to output the prediction results of the three-level lymphatic structure.
[0014] When the HE-stained pathological image is input into the TLS-DSNet model, it is first processed by a main convolutional layer to reduce the image size to one-fourth of the original size, generate an initial feature map, and then send the initial feature map to the CNN branch and the Transformer branch respectively.
[0015] In the CNN branch, the initial feature map is first processed through three convolutional coding layers in sequence to obtain the encoding result; in the decoding process, the encoded feature map is gradually restored to its size through convolutional decoding layers, skip connections, and upsampling operations to obtain the semantic segmentation prediction results of the three-level lymphatic structure, and the three-level lymphatic structure scale prediction results are output through two fully connected layers.
[0016] In the Transformer branch, the initial feature map is mapped to a 768-dimensional feature space by a linear layer through linear projection, and position encoding is added to form an embedded sequence; the embedded sequence is sent to a 12-layer Transformer encoder for processing to obtain an encoded embedded sequence, in which each layer contains a multi-head self-attention mechanism and a feedforward neural network; and the encoded embedded sequence is converted to the same size as the feature map of the last layer of the CNN branch through deformation and upsampling operations, and superimposed on the CNN feature map; the feature maps of different scales obtained by the superimposed feature map through 3 residual convolution layers are sent to the detection head of the YOLO model to obtain the target detection results of the three-level lymphatic structure.
[0017] The two-stage training strategy is specifically as follows: the first stage trains the semantic segmentation and scale prediction tasks in the CNN branch of the TLS-DSNet model, and the second stage fixes some parameters of the model pre-trained in the first stage to learn the target detection task.
[0018] In the first stage, the parameters of the residual convolution layer and the YOLO model detection head used in the Transformer branch of the TLS-DSNet model are fixed, and the gradient feedback obtained by calculating the loss is only used to update the backbone network of the model and the segmentation and scale prediction head of the CNN branch.
[0019] The total loss of the first stage is expressed as:
[0020] L1=α1*L sc +α2*L seg
[0021] Among them, α1 and α2 are to balance the two losses L sc , L seg The hyperparameter weights set, L sc is the scale prediction loss, L seg Predict loss for segmentation;
[0022] The scale prediction loss is evaluated using cross entropy loss, expressed as:
[0023]
[0024] Where N is the total number of samples; C is the total number of categories; y ij is an indicator variable, that is, 1 when the i-th sample belongs to category j, otherwise it is 0; p ij is the probability that the i-th sample belongs to category j;
[0025] The segmentation prediction loss is evaluated using Dice Loss, expressed as:
[0026]
[0027] Among them, p i represents the probability of each pixel prediction, g i represents the true label, and M represents the total pixels of the image.
[0028] In the second stage, the weights of the TLS-DSNet model pre-trained in the first stage are used in the learning of the second stage target detection task, and the parameters of the decoding convolution layer and the two prediction heads in the CNN branch are updated. The loss gradient is passed back to update the parameters in the backbone network, Transformer branch and YOLO detection head of the model.
[0029] The total loss of the second stage is expressed as:
[0030] L2=α3*L bbox +α4*L conf
[0031] Among them, α3 and α4 are to balance the two losses L bbox , L conf The hyperparameter weights set, L bbox is the bounding box regression loss, L conf is the confidence loss;
[0032] The confidence loss is a binary cross entropy loss, expressed as:
[0033]
[0034] Where N is the total number of samples; C is the total number of categories; y ij is an indicator variable, that is, 1 when the i-th sample belongs to category j, otherwise it is 0; p ij is the probability that the i-th sample belongs to category j;
[0035] The loss function of the bounding box regression is CIoU loss, expressed as:
[0036]
[0037] Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ρ 2 (b p ,b t ) is the prediction box b p With the real center b t The Euclidean distance of , c is the diagonal length of the smallest box containing two boxes, and μ is the consistency term for evaluating the aspect ratio prediction results, expressed as:
[0038]
[0039] Among them, w and h are the height and width of the real box, respectively, and w′ and h′ are the height and width of the predicted box, respectively.
[0040] A system for predicting tertiary lymphatic structures based on HE-stained pathological images, comprising:
[0041] Dataset construction module: used to obtain HE-stained pathological images and perform preprocessing and image annotation to construct image datasets;
[0042] TLS-DSNet model training module: used to train the TLS-DSNet model based on the image dataset using a two-stage training strategy. The TLS-DSNet model includes a U-shaped CNN branch and a 12-layer Transformer branch. HE-stained pathological images are used as model inputs. The results of the two branches are obtained and then fused to predict the tertiary lymphatic structure.
[0043] Prediction module: It is used to pre-process the HE-stained pathological image to be predicted and input it into the trained TLS-DSNet model to output the prediction results of the three-level lymphatic structure.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] First, the model constructed by the present invention utilizes the flexible attention allocation introduced by the Transformer architecture to dynamically adjust the focus point in the image according to the data requirements, thereby enhancing the adaptability and generalization ability of the model. At the same time, the model is allowed to pre-learn the scale information of the image and the semantic segmentation information of the target in the first stage, providing prior information related to the distribution of the data set for the training of the second stage model, and dynamically adjusting the anchor point generation preference to avoid the prediction bias caused by the unbalanced distribution of the original labels in the data set.
[0046] Second, this application proposes a CNN-Transformer dual-branch hybrid architecture, which, based on the extracted local feature information, combines the global attention mechanism brought by the Transformer architecture to capture the global representation of the image. By fusing two complementary information representations, it breaks through the limitations of existing methods in the range of the receptive field and achieves effective detection of ultra-large targets.
[0047] Third, the model constructed in this application uses a two-stage training strategy and a dual-branch architecture to perform loss calculations and parameter updates on its corresponding branches in different tasks and training stages, thereby optimizing its forward guidance for the final target detection task while reducing the computational requirements of model training, avoiding potential misleading of the model's main task by different loss optimization targets for each auxiliary task, and directly inheriting key information related to the main task from multiple auxiliary tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is a flow chart of the method of the present invention;
[0049] Figure 2 It is the framework diagram of the TLS-DSNet model of the present invention;
[0050] Figure 3 It is a schematic diagram of the two-stage training strategy of the present invention;
[0051] Figure 4 It is a flow chart of the visualization platform of the present invention;
[0052] Figure 5 This is a comparison chart of PR curves of different models when IoU=0.5 in an embodiment. DETAILED DESCRIPTION
[0053] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0054] This embodiment provides a method for predicting tertiary lymphatic structures based on HE-stained pathological images. Figure 1 As shown, the following steps are included:
[0055] S1, obtain HE stained pathological images and perform preprocessing and image annotation to build an image dataset.
[0056] S2, based on the image dataset, adopts a two-stage training strategy to train the TLS-DSNet model.
[0057] like Figure 2 As shown in the figure, the TLS-DSNet model includes a U-shaped CNN branch and a 12-layer Transformer branch. The HE-stained pathological image is used as the model input. The results of the two branches are obtained separately and then fused to predict the tertiary lymphatic structure.
[0058] When the HE-stained pathological image is input into the TLS-DSNet model, it is first processed by a main convolutional layer to reduce the image size to one-fourth of the original size, generate an initial feature map, and then send the initial feature map to the CNN branch and the Transformer branch respectively.
[0059] In the CNN branch, the initial feature map is first processed through three convolutional coding layers in sequence to obtain the encoding result; in the decoding process, the encoded feature map is gradually restored to its size through convolutional decoding layers, skip connections, and upsampling operations to obtain the semantic segmentation prediction results of the three-level lymphatic structure, and the three-level lymphatic structure scale prediction results are output through two fully connected layers.
[0060] In the Transformer branch, the initial feature map is mapped to a 768-dimensional feature space by a linear layer through linear projection, and position encoding is added to form an embedding sequence; the embedded sequence is sent to a 12-layer Transformer encoder for processing to obtain an encoded embedded sequence, in which each layer contains a multi-head self-attention mechanism and a feedforward neural network; and the encoded embedded sequence is converted to the same size as the feature map of the last layer of the CNN branch through deformation (Reshape) and upsampling operations, and superimposed on the CNN feature map; the feature maps of different scales obtained by the superimposed feature map through 3 residual convolution layers are sent to the detection head of the YOLO model to obtain the target detection results of the three-level lymphatic structure.
[0061] In order to fuse the information of different characteristics obtained by the two branches, the patch embedding sequence output by the Transformer branch is deformed and upsampled to align it with the feature map output by the CNN branch in terms of dimension, and superimposed on the CNN feature map of the corresponding layer for subsequent convolutional layer processing. The feature map output by the CNN branch is combined with the patch embedding sequence of the Transformer branch by average pooling and deformation to provide input for the next encoding layer.
[0062] S3, the HE-stained pathological image to be predicted is preprocessed and then input into the trained TLS-DSNet model to output the three-level lymphatic structure prediction result.
[0063] like Figure 3 As shown in the figure, the two-stage training strategy is as follows: the first stage trains the two auxiliary tasks of semantic segmentation and scale prediction in the CNN branch of the TLS-DSNet model; the second stage fixes some parameters of the model pre-trained in the first stage and performs the learning of the target detection task.
[0064] In the first stage (Stage I), the parameters of the residual convolution layer used by the Transformer branch in the TLS-DSNet model and the YOLO model detection head are fixed, and the gradient feedback obtained by calculating the loss is only used to update the backbone network of the model and the segmentation and scale prediction head of the CNN branch.
[0065] In Stage I, the cross entropy loss is used to evaluate the scale prediction accuracy of the model, and the scale prediction loss L is obtained. sc It can be expressed mathematically as:
[0066]
[0067] Where N is the total number of samples, C is the total number of categories, and y ij is an indicator variable, that is, 1 when the i-th sample belongs to category j, otherwise it is 0. ij The probability that the i-th sample belongs to class j.
[0068] In addition, Dice Loss is used to evaluate the learning effect of the model in the segmentation task, and the loss of segmentation prediction L is obtained. seg can be expressed as:
[0069]
[0070] Among them, p i represents the probability of each pixel prediction, g i represents the true label, and N represents the total pixels of the image. Thus, the total loss L1 in the first stage can be described as:
[0071] L1=α1*L sc +α2*L seg (3)
[0072] Among them, α1 and α2 are hyperparameter weights set to balance the two losses. In subsequent experiments, this embodiment sets these two hyperparameters to α1=0.25 and α2=0.75 respectively.
[0073] After completing the model learning of Stage I, the model weights of the pre-trained TLS-DSNet are used in the learning of the second stage (Stage II) target detection task. In Stage II, the parameters of the decoding convolution layer (Mask Decoder Conv Block) and the two prediction heads in the CNN branch are updated with a very small learning rate (lr = 1e-6). The loss gradient feedback is mainly used to update the parameters in the backbone network, Transformer branch and YOLO Detection Head of the model. This embodiment uses the loss function proposed by YOLO v5 to evaluate the prediction effect of the model in the target detection task, including the bounding box regression loss L bbox and confidence loss L conf There are two parts. Among them, the confidence loss is the binary cross entropy loss, and the calculation is the same as formula (1). The loss function of bounding box regression is CIoU loss (Complete Intersection over Union Loss), and its mathematical expression is:
[0074]
[0075] Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ρ 2 (b p ,b t ) is the prediction box b p With the real center b t The Euclidean distance of , c is the diagonal length of the smallest box containing two boxes, and μ is the consistency term for evaluating the aspect ratio prediction results, which is mathematically expressed as:
[0076]
[0077] Among them, w and h are the height and width of the real box, respectively, and w′ and h′ are the height and width of the predicted box, respectively. Therefore, the total loss L2 of target detection in Stage II can be described as:
[0078] L2=α3*L bbox +α4*L conf (6)
[0079] Among them, α3 and α4 are hyperparameter weights set to balance the two losses. In subsequent experiments, this embodiment sets these two hyperparameters to α3=0.05 and α4=1 respectively.
[0080] In summary, the present invention has designed and developed for the first time a new dual-branch multi-task hybrid network architecture, named TLS-DSNet, which aims to identify and extract the feature information of TLS from tumor tissue pathology images. This architecture combines the local features and global information of the image, improves the performance of identifying TLS in tissue pathology images, and achieves the best TLS detection effect currently. In addition, the present invention also designs a two-stage training process and parameter update strategy, which improves the learning effect of the model on the main task of target detection by optimizing the way the model inherits and integrates various annotation information from auxiliary tasks. At the same time, it has been successfully applied to other mainstream backbone networks, showing its good generalizability.
[0081] The TLS-DSNet and other advanced detection models mentioned in the present invention can be integrated into a TLS detection visualization platform to achieve rapid localization of tertiary lymphatic structures and quantification of feature information from WSI images. Figure 4 As shown in the figure, the visualization platform can receive the original WSI input by the user, then perform data preprocessing and patch cutting, output the detection results through the trained detection model saved in the platform, and count the target information in TLS, so as to realize the effects of visualizing the MIV distribution in WSI, displaying the statistical results of various TLS features, and analyzing TLS features and clinical information, providing medical workers with an efficient interactive tool.
[0082] In order to verify the performance of the present invention, experiments were conducted on the TCGA BRCA breast cancer dataset in this embodiment, and a series of backbone networks including YOLO v5 and ResNet series were analyzed and compared. The experiments were trained and tested according to the experimental regulations of the dataset.
[0083] Step 1: Construct a training dataset at the patch level. Randomly divide the data into training and test sets at the WSI level in a ratio of 8:2. After that, save the WSI at three different magnifications: 2X, 4X, and 10X. Use Python's PIL package to construct a square image of 640*640 pixels, and generate the data annotation format required by the YOLO model through the existing annotations to obtain the bounding box coordinates of each target. Each annotation is saved as a five-element array according to the center position coordinates, height, width, and category label. Each image contains three types of annotations: scale label, semantic segmentation label, and target detection box.
[0084] Step 2: Construct a deep learning dataset. The deep learning algorithm model takes out a batch (batch_size is customizable, 10 is recommended) of random images each time, and the images are randomly rotated 90-360 degrees, or randomly flipped up and down or left and right. Gaussian noise is randomly added to the three HSV channels of each image, where the mean parameter of the Gaussian noise is 20. Each image is randomly translated in a certain direction, where the maximum displacement is 100 pixels.
[0085] Step 3: Construct the TLS-DSNet model framework according to Figure 3 Execute the pre-training learning process of the first stage (Stage I). In Stage I, the model is pre-trained for segmentation and scale prediction tasks, using scale labels and semantic segmentation labels. Cross Entropy Loss is used to evaluate the scale prediction accuracy of the model, and Dice Loss is used to evaluate the learning effect of the model in the segmentation task. The parameters of the residual convolution layer and the subsequent YOLO Detection Head used in the Transformer branch of TLS-DSNet are fixed. The gradient feedback obtained by calculating the loss for each input image is only used to update the backbone network of the model and the segmentation and scale prediction heads of the CNN branch. After 100 epochs, the pre-trained model is obtained.
[0086] Step 4: Follow Figure 3 Execute the second stage (Stage II) training process. In Stage II, the model uses CIoU loss as the bounding box regression loss and binary cross entropy loss as the confidence loss to evaluate the prediction effect of the model in the target detection task. At the same time, the parameters of the decoding convolution layer (Mask Decoder Conv Block) and the two prediction heads in the CNN branch are updated with a very small learning rate (lr = 1e-6). The loss gradient is mainly used to update the parameters in the backbone network, Transformer branch and YOLO Detection Head of the model. After 100 epochs, the trained model is obtained.
[0087] Step 5: Predict model indicators on the test set. Draw the F1-confidence curve to confirm the predicted confidence parameters, fix the IoU parameters, and predict the indicators of the network on the test set to compare the mean and standard deviation of the performance of other methods with this method in indicators including Recall, Precision, mAP, etc., and draw visual detection results and PR curves and other indicators to obtain horizontal comparison results of the overall performance of the model.
[0088] Table 1 Quantitative evaluation results of comparative experiments
[0089]
[0090]
[0091] As shown in Table 1, the TLS-DSNet proposed in this paper shows the best performance in all indicators except Precision. Specifically, compared with other models, the proposed method improves the F1-score, Recall, mAP@0.50 and mAP@0.50:0.95 by 3.5%, 4.7%, 4.1% and 2.3% respectively. In addition, Figure 5 The PR curve shows that TLS-DSNet exhibits high precision at most recall levels, especially in the medium to high recall range, where its curve is significantly higher than other models, indicating that it can still effectively control false positives while increasing the coverage of positive samples and has better overall prediction performance.
[0092] Table 2 shows the results of the ablation experiment to demonstrate the necessity of the two-stage training strategy and multi-task architecture proposed in the present invention. After pre-training, all models showed performance improvements in most indicators, especially in the Recall and mAP indicators that are more concerned in clinical tasks, which increased by an average of 5.1%, 3.3% and 2.6% respectively compared with the models that have not been pre-trained. This shows that the training strategy designed by the present invention can help the model directly inherit key information related to the main task from multiple auxiliary tasks to improve the overall learning effect of the model on the main task.
[0093] Table 2 Quantitative evaluation results of ablation experiments
[0094]
[0095] This embodiment also provides a system for predicting tertiary lymphatic structures based on HE-stained pathological images, including:
[0096] Dataset construction module: used to obtain HE-stained pathological images and perform preprocessing and image annotation to construct image datasets;
[0097] TLS-DSNet model training module: used to train the TLS-DSNet model based on the image dataset using a two-stage training strategy. The TLS-DSNet model includes a U-shaped CNN branch and a 12-layer Transformer branch. HE-stained pathological images are used as model inputs. The results of the two branches are obtained and then fused to predict the tertiary lymphatic structure.
[0098] Prediction module: It is used to pre-process the HE-stained pathological image to be predicted and input it into the trained TLS-DSNet model to output the prediction results of the three-level lymphatic structure.
[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0100] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A method for predicting tertiary lymphatic structure based on HE-stained pathological images, characterized in that: The following steps are involved: Obtain HE-stained pathological images, perform preprocessing and image annotation, and construct an image dataset; Based on the image dataset, a two-stage training strategy was adopted to train the TLS-DSNet model, which included a U-shaped CNN branch and a 12-layer Transformer branch. HE-stained pathological images were used as model input, and the results of the two branches were obtained and then fused to predict the tertiary lymphatic structure. The HE-stained pathological images to be predicted are preprocessed and then input into the trained TLS-DSNet model to output the prediction results of the three-level lymphatic structure.
2. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 1, characterized in that: When the HE-stained pathological image is input into the TLS-DSNet model, it is first processed by a main convolutional layer to reduce the image size to one-fourth of the original size, generate an initial feature map, and then send the initial feature map to the CNN branch and the Transformer branch respectively.
3. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 1, characterized in that: In the CNN branch, the initial feature map is first processed through three convolutional coding layers in sequence to obtain the encoding result; in the decoding process, the encoded feature map is gradually restored to its size through convolutional decoding layers, skip connections, and upsampling operations to obtain the semantic segmentation prediction results of the three-level lymphatic structure, and the three-level lymphatic structure scale prediction results are output through two fully connected layers.
4. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 3, characterized in that: In the Transformer branch, the initial feature map is mapped to a 768-dimensional feature space by a linear layer through linear projection, and position encoding is added to form an embedded sequence; the embedded sequence is sent to a 12-layer Transformer encoder for processing to obtain an encoded embedded sequence, in which each layer contains a multi-head self-attention mechanism and a feedforward neural network; and the encoded embedded sequence is converted to the same size as the feature map of the last layer of the CNN branch through deformation and upsampling operations, and superimposed on the CNN feature map; the feature maps of different scales obtained by the superimposed feature map through 3 residual convolution layers are sent to the detection head of the YOLO model to obtain the target detection results of the three-level lymphatic structure.
5. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 4, characterized in that: The two-stage training strategy is specifically as follows: the first stage trains the semantic segmentation and scale prediction tasks in the CNN branch of the TLS-DSNet model, and the second stage fixes some parameters of the model pre-trained in the first stage to learn the target detection task.
6. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 5, characterized in that: In the first stage, the parameters of the residual convolution layer and the YOLO model detection head used in the Transformer branch of the TLS-DSNet model are fixed, and the gradient feedback obtained by calculating the loss is only used to update the backbone network of the model and the segmentation and scale prediction head of the CNN branch.
7. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 6, characterized in that: The total loss of the first stage is expressed as: L1=α1*L sc +α2*L seg Among them, α1 and α2 are to balance the two losses L sc , L seg The hyperparameter weights set, L sc is the scale prediction loss, L seg Predict loss for segmentation; The scale prediction loss is evaluated using cross entropy loss, expressed as: Where N is the total number of samples; C is the total number of categories; y ij is an indicator variable, that is, 1 when the i-th sample belongs to category j, otherwise it is 0; p ij is the probability that the i-th sample belongs to category j; The segmentation prediction loss is evaluated using Dice Loss, expressed as: Among them, p i represents the probability of each pixel prediction, g i represents the true label, and M represents the total pixels of the image.
8. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 5, characterized in that: In the second stage, the weights of the TLS-DSNet model pre-trained in the first stage are used in the learning of the second stage target detection task, and the parameters of the decoding convolution layer and the two prediction heads in the CNN branch are updated. The loss gradient is passed back to update the parameters in the backbone network, Transformer branch and YOLO detection head of the model.
9. The method for predicting tertiary lymphatic structure based on HE-stained pathological images according to claim 8, characterized in that: The total loss of the second stage is expressed as: L2=α3*L bbox +α4*L conf Among them, α3 and α4 are to balance the two losses L bbox , L conf The hyperparameter weights set, L bbox is the bounding box regression loss, L conf is the confidence loss; The confidence loss is a binary cross entropy loss, expressed as: Where N is the total number of samples; C is the total number of categories; y ij is an indicator variable, that is, 1 when the i-th sample belongs to category j, otherwise it is 0; p ij is the probability that the i-th sample belongs to category j; The loss function of the bounding box regression is CIoU loss, expressed as: Among them, IoU represents the intersection-over-union ratio between the predicted box and the real box, ρ 2 (b p ,b t ) is the prediction box b p With the real center b t The Euclidean distance of , c is the diagonal length of the smallest box containing two boxes, and μ is the consistency term for evaluating the aspect ratio prediction results, expressed as: Among them, w and h are the height and width of the real box, respectively, and w′ and h′ are the height and width of the predicted box, respectively.
10. A system for predicting tertiary lymphatic structure based on HE-stained pathological images, characterized in that: include: Dataset construction module: used to obtain HE-stained pathological images and perform preprocessing and image annotation to construct image datasets; TLS-DSNet model training module: used to train the TLS-DSNet model based on the image dataset using a two-stage training strategy. The TLS-DSNet model includes a U-shaped CNN branch and a 12-layer Transformer branch. HE-stained pathological images are used as model inputs. The results of the two branches are obtained and then fused to predict the tertiary lymphatic structure. Prediction module: It is used to pre-process the HE-stained pathological image to be predicted and input it into the trained TLS-DSNet model to output the prediction results of the three-level lymphatic structure.
Citation Information
Patent Citations
Remote sensing image road extraction method based on multi-task neural network
CN115953699A
Semi-supervised underwater image semantic segmentation method based on generative adversarial learning
CN116051840A
Pathological image classification method and device, electronic equipment and storage medium
CN117409233A
Image processing method and apparatus, model training method and apparatus, and terminal device
WO2024092590A1
Cited By
Method and device for interpreting cell pathology slide, computer equipment and medium
CN119942541A
Breast cancer HE slice three-level lymph structure density detection method based on deep learning
CN120634950A
Gene expression prediction method and system based on HE staining pathological image
CN120895106A
Gene expression prediction method and system based on h&e staining pathological images
CN120895106B
Multi-scale feature integrated lymph node identification system and method
CN121010555A
