Lung cancer STAS auxiliary diagnosis method and device based on scale perception multi-instance learning

Abstract

The invention provides a lung cancer STAS auxiliary diagnosis method and device based on scale perception multi-instance learning. The method comprises the following steps: acquiring a slice image of lung tissue of a patient, and preprocessing the slice image to obtain a preprocessed pathological tissue image; segmenting the preprocessed pathological tissue image to obtain a first patch set comprising a plurality of patches, and performing image enhancement on the first patch set to obtain a second patch set; extracting features of each patch in the second patch set to obtain a vectorized feature matrix; the feature matrix is input into a trained SMILE model to obtain a prediction result, whether the patient is lung cancer STAS or not is diagnosed in an auxiliary mode based on the prediction result, and the SMILE model comprises a zoom perception attention mechanism. According to the method, the attention mechanism of zoom perception is introduced into the SMILE model, the attention on the patch with the low weight value is increased, and the STAS recognition accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical imaging diagnosis, and specifically relates to a STAS-assisted diagnosis method and device for lung cancer based on scale-aware multi-instance learning. Background Art

[0002] Spread through air spaces (STAS) is a new invasion pattern of lung cancer, which is defined by the World Health Organization as the spread of tumor cells (such as micropapillary clusters, solid nests or single cells) into air spaces beyond the leading edge of the primary tumor.

[0003] Domestic and international studies have shown that STAS of lung cancer is more common in lung cancer patients with pleural invasion, vascular invasion, larger tumor volume, and higher pathological stage, and is closely associated with a poor prognosis. Currently, histopathological images are considered the gold standard for STAS of lung cancer, but the current diagnostic process of STAS of lung cancer relies heavily on pathologists to manually identify and examine pathological sections. This traditional method is highly subjective, and its accuracy relies heavily on the experience and expertise of pathologists, making it susceptible to human bias. In addition, when faced with large-scale pathological data, pathologists must interpret a large number of samples, which often results in a heavy workload, thereby limiting the efficiency and accuracy of each individual diagnosis. This burden not only prolongs the turnaround time, but also increases the risk of misdiagnosis and missed diagnosis. Summary of the Invention

[0004] The purpose of the present invention is to introduce a zoom-aware attention mechanism into the multi-instance learning framework model, increase the attention to patches with lower weight values ​​(partial areas outside the main tumor area), reduce the model's excessive dependence on high-attention patches, thereby enhancing the SMILE model's ability to capture sparse heterogeneous features and improving the STAS recognition accuracy.

[0005] In a first aspect, the present invention provides a lung cancer STAS auxiliary diagnosis method based on scale-aware multi-instance learning. The auxiliary diagnosis method includes the following steps: step S10, obtaining a slice image of the patient's lung tissue, and preprocessing the slice image to obtain a preprocessed pathological tissue image, wherein the preprocessing includes digitizing the whole slide image, segmenting the tissue area, and detecting the background and blurred area; step S20, segmenting the preprocessed pathological tissue image to obtain a first patch set including a plurality of patches with uniform size and uniform shape, and performing image enhancement on the first patch set to obtain a second patch set; step S30, extracting features of each patch in the second patch set to obtain a vectorized feature matrix, wherein each element of the feature matrix is ​​used to represent the extracted features of a patch; step S40, inputting the feature matrix into a trained SMILE model to obtain a prediction result, and based on the prediction result, auxiliary diagnosis is made as to whether the patient has lung cancer STAS or not, wherein the SMILE model uses PyTorch as a framework and includes a scale-aware attention mechanism.

[0006] In a specific embodiment, in step S40, the process of the SMILE model outputting the prediction result includes: step (1), obtaining the attention score corresponding to each patch based on the feature matrix and the attention score calculation formula, wherein the attention score calculation formula is:

[0007] A=W a (tanh(V a H)☉o(U a H))={a1,a2,...,a n};

[0008] represents the unregularized attention score, and are three initialized weight feature matrices, tanh represents the hyperbolic tangent function, σ represents the sigmoid activation function, and a irepresents the first attention score corresponding to the i-th patch, e is the attention mechanism parameter, and d is the feature compression dimension; step (2), each first attention score is subjected to Max_Min normalization processing respectively, and a plurality of second attention scores corresponding to each patch are obtained; step (3), each second attention score is compared with a preset threshold, and the second attention score greater than the preset threshold is scaled based on a preset scaling factor, and the second attention score corresponding to the patch is updated with the scaled attention score; step (4), the feature of each patch and the second attention score corresponding to the patch are weightedly summed to obtain a fused feature; step (5), using the fused feature as input, a prediction result is obtained based on a linear layer with a Sigmoid activation function, and the prediction result is 1 or 0, where 1 represents STAS and 0 represents non-STAS.

[0009] In a specific embodiment, adjusting the preset threshold and the preset scaling factor can dynamically adjust the area focused on by the attention mechanism, the preset threshold is 0.5, and the preset scaling factor is 0.5 to 0.6.

[0010] In a specific embodiment, in step S30, the step of extracting each patch includes offline acquiring initialization features based on a hybrid model that integrates convolutional neural network and Transformer architecture, and fine-tuning the initialization features based on the network structure of the online stage, wherein the network structure of the online stage includes three layers: BatchNorm1d, Linear and ReLU.

[0011] In a specific embodiment, the training method of the SMILE model includes: step (a), obtaining a lung cancer histopathology image set consisting of multiple WSI images that have completed STAS and non-STAS identification; step (b), segmenting each WSI image in the lung cancer histopathology image set, detecting the tissue area, detecting the background, the blurred area, performing segmentation processing, and image enhancement to obtain multiple packages, each package including multiple instances; step (c), forming a data set from the multiple packages and the labels corresponding to the packages, and splitting the data set into a training set and a validation set, training the SMILE model based on five-fold cross-validation to obtain the trained SMILE model, wherein the label corresponding to the package is 1 or 0, and when the label value is 1, it indicates that it is identified as STAS, and when the label value is 0, it indicates that it is identified as non-STAS.

[0012] In a specific embodiment, the lung cancer histopathology image set is selected from one of the STAS_TCGA image set, the STAS_CPTAC image set and the STAS_CSU image set.

[0013] In a specific embodiment, the SMILE model is trained using the Ranger optimizer to adjust the model parameters, with a learning rate of 2×e -4 , the weight decay is 1×e -5 , the number of training rounds is more than 100, the batch size is 12, the threshold is set to 0.5~0.8, the scaling factor is set to 0.3~0.8, the feature dimension l is 768, the feature compression dimension d is 256, and the attention mechanism parameter e is 64.

[0014] In a specific embodiment, in step S20, the patch is a square patch with a size of 256×256 pixels.

[0015] In a specific implementation, in step S20, a GAN model is used to perform image enhancement.

[0016] In a second aspect, the present invention provides a lung cancer STAS auxiliary diagnosis device based on scale-aware multi-instance learning, the auxiliary diagnosis device comprising: an acquisition module for acquiring a slice image of a patient's lung tissue, and preprocessing the slice image to obtain a preprocessed pathological tissue image, wherein the preprocessing comprises digitizing the whole slide image, segmenting tissue regions, and detecting background and blurred regions; an image enhancement module for segmenting the preprocessed pathological tissue image to obtain a first patch set comprising a plurality of patches of uniform size and shape, and performing image enhancement on the first patch set to obtain a second patch set; a feature extraction module for extracting features of each patch in the second patch set to obtain a vectorized feature matrix, wherein each element of the feature matrix is ​​used to represent a feature extracted from a patch; a diagnosis module for inputting the feature matrix into a trained SMILE model to obtain a prediction result, and based on the prediction result, assisting in diagnosing whether the patient has lung cancer STAS or not, wherein the SMILE model comprises a scale-aware attention mechanism.

[0017] The beneficial effects of the present invention include at least:

[0018] The present invention provides a STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning, which comprises the following steps: step S10, obtaining a slice image of the patient's lung tissue, and preprocessing the slice image to obtain a preprocessed pathological tissue image, wherein the preprocessing includes digitizing the whole slide image, segmenting the tissue area, and detecting the background and blurred area; step S20, segmenting the preprocessed pathological tissue image to obtain a first patch set including a plurality of patches with uniform size and uniform shape, and performing image enhancement on the first patch set to obtain a second patch set; step S30, extracting features of each patch in the second patch set to obtain a feature matrix represented by vectorization, wherein the feature matrix Each element of the matrix is ​​used to represent a feature extracted from a patch; in step S40, the feature matrix is ​​input into a trained SMILE model to obtain a prediction result, and based on the prediction result, an auxiliary diagnosis is made as to whether the patient has lung cancer STAS or not. The SMILE model uses a PyTorch framework and includes a scale-aware attention mechanism. The present invention improves the existing SMILE model by introducing a scale-aware attention mechanism into the SMILE model, thereby increasing attention to patches with lower weight values ​​(partial areas outside the main tumor area), reducing the SMILE model's excessive reliance on high-attention patches, thereby enhancing the model's ability to capture sparse heterogeneous features and improving the STAS recognition accuracy.

[0019] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic flow chart of the steps of a STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning provided by one embodiment of the present invention;

[0021] Figure 2 This is a flowchart of a STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning provided by an embodiment of the present invention, wherein: Figure 2 a in the figure is the overall workflow diagram of the auxiliary diagnosis method. Figure 2 b in Fig. 2 is the flowchart for extracting patch features;

[0022] Figure 3 This is a module diagram of a STAS-assisted diagnosis device for lung cancer based on scale-aware multi-instance learning provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0024] In the present invention, the full English name of SMILE is Scale-Aware Multi-Instance Learning, and the Chinese name of the SMILE model is Scale-Aware Multi-Instance Learning Model.

[0025] The full English name of WSI is Whole Slide Imaging, and the Chinese name of WSI image is whole slide image.

[0026] See also Figure 1 and Figure 2 According to a first aspect of the present invention, the present invention provides a STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning, comprising the following steps:

[0027] Step S10: Acquire a slice image of the patient's lung tissue, and preprocess the slice image to obtain a preprocessed pathological tissue image. The preprocessing includes digitizing the whole slide image, segmenting tissue areas, and detecting background and blurred areas.

[0028] In the present invention, the number of the slice image is one, which may be a frozen section (FS) slice, a paraffin section (PS) slice, or an immunohistochemistry (IHC) slice.

[0029] In the present invention, whole slide image digitization processing, namely WSI processing, refers to the generation of panoramic digital images by digitizing traditional glass pathology slides (such as HE-stained slides and immunohistochemistry slides) using a high-resolution scanner.

[0030] In an optional embodiment, the OTSU algorithm is used to segment the tissue region.

[0031] In the present invention, the digital processing of the whole slide image, the segmentation of the tissue area using the OTSU algorithm, and the detection of the background and the blurred area are all processed using conventional methods provided by the prior art, and the present invention does not make any improvements thereto.

[0032] Step S20 : ​​segmenting the preprocessed pathological tissue image to obtain a first patch set including a plurality of patches with uniform size and uniform shape, and performing image enhancement on the first patch set to obtain a second patch set.

[0033] In an optional embodiment, the patch is a square patch with a size of 256×256 pixels.

[0034] It is understandable that when the size of the pre-processed pathological tissue image is small, the pre-processed pathological tissue image may be enlarged before the segmentation operation is performed.

[0035] In an optional embodiment, a GAN model is used for image enhancement.

[0036] Step S30 : extracting features from each patch in the second patch set to obtain a feature matrix represented by vectorization, where each element of the feature matrix is ​​used to represent a feature extracted from a patch.

[0037] In the present invention, the second patch set can be expressed as:

[0038] Among them, x i represents the i-th patch, n represents the number of patches, R 256×256×3 Represents a three-dimensional real tensor with dimensions of 256×256×3, where 256 is the size and 3 is the number of channels.

[0039] Please refer to Figure 2 , Figure 2 The package a in is the second patch set, which includes 9 patches. In the present invention, patches can also be described as instances, that is, Figure 2 The package a in can be understood as the second patch set including 9 patches, or can be understood as the package including 9 instances.

[0040] In the present invention, the obtained feature matrix is ​​expressed as Among them, h i is the feature of the i-th patch.

[0041] In the present invention, Where n represents the number of patches and d represents the feature compression dimension.

[0042] In an optional embodiment, the step of extracting features of each patch includes offline acquiring initialization features based on a hybrid model that integrates convolutional neural networks and Transformer architectures, and fine-tuning the initialization features based on the network structure of the online stage, wherein the network structure of the online stage includes three layers: BatchNorm1d, Linear, and ReLU.

[0043] Specifically, the patch feature extraction process is divided into two stages: offline and online. In the offline stage, the backbone network adopts a hybrid model that combines a convolutional neural network (CNN) and a Transformer architecture, named CTransPath. After pre-training based on semantic contrastive learning, the backbone network's weights are fixed and no longer participate in subsequent network training. Specifically, the output of the last layer of CTransPath is extracted as the initialization of the patch features. This process is shown in Equation (1).

[0044] T = CTransPath(X)(1), where n represents the number of patches and l represents the feature dimension.

[0045] The online network structure consists of three layers: a BatchNorm1d layer, a Linear layer, and a ReLU layer. The parameters of these layers are continuously adjusted as the model is optimized to fine-tune the patch feature training process. The online network process is shown in Equations (2) and (3).

[0046] R = BatchNormld(T)(2), where

[0047] H=ReLU(Linear(R))(3)

[0048] Please refer to the patch feature extraction process Figure 2 The b in.

[0049] Step S40: Input the feature matrix into the trained SMILE model to obtain a prediction result, and based on the prediction result, assist in diagnosing whether the patient has lung cancer STAS or not. The SMILE model uses a PyTorch framework and includes a scale-aware attention mechanism.

[0050] In an optional embodiment, the process of the SMILE model outputting a prediction result includes:

[0051] Step (1): Based on the feature matrix and the attention score calculation formula, a plurality of first attention scores corresponding to each patch are obtained, wherein the attention score calculation formula is:

[0052] A=W a (tanh(V a H)☉σ(U a H))={a1,a2,...,a n}.

[0053] represents the unregularized attention score, and are three initialized weight feature matrices, tanh represents the hyperbolic tangent function, σ represents the sigmoid activation function, and a i represents the first attention score corresponding to the i-th patch, e is the attention mechanism parameter, and d is the feature compression dimension.

[0054] It should be noted that the unregularized attention score means that no processing has been performed, such as using a penalty function for optimization.

[0055] In the present invention, the three initialized weight feature matrices can be set based on experience or existing technology.

[0056] Step (2): Each first attention score is normalized by Max_Min to obtain multiple second attention scores corresponding to each patch.

[0057] The formula for Max_Min normalization is as follows:

[0058]

[0059] Step (3): compare each second attention score with a preset threshold, scale the second attention scores greater than the preset threshold based on a preset scaling factor, and update the second attention score corresponding to the patch with the scaled weight value.

[0060] The formulas for obtaining the second attention score corresponding to each patch based on the multiple first attention scores corresponding to each patch are shown in formulas (4) to (6):

[0061]

[0062] Where A is the first attention score obtained in step (1), E is the matrix with all features set to 1, S is the feature matrix after processing, Softmax is the activation function, Factor is the preset scaling factor, Threshold is the preset threshold, Max_Min is the normalization processing function, ⊙ is the point-by-point multiplication, is point-by-point subtraction, is scalar multiplication.

[0063] Please refer to Figure 2 a in Figure 2 The above processing is shown in a.

[0064] Step (4) performs weighted summation on the features of each patch and the second attention score corresponding to the patch to obtain a fused feature.

[0065]

[0066] Among them, sa i is the second attention score corresponding to the i-th patch, h i is the feature of the i-th patch, and Z is the fused feature.

[0067] Step (5): using the fused features as input, obtaining a prediction result based on a linear layer with a Sigmoid activation function, wherein the prediction result is 1 or 0, where 1 represents STAS and 0 represents non-STAS.

[0068]

[0069] Among them, σ represents the sigmoid activation function.

[0070] In the present invention, the prediction result It is 1 or 0, where 1 represents STAS and 0 represents non-STAS. When the prediction result is 1, the probability of predicting the patient as STAS is high, and when the prediction result is 0, the probability of predicting the patient as non-STAS is high.

[0071] In the present invention, adjusting the preset threshold and the preset scaling factor can dynamically adjust the area focused on by the attention mechanism.

[0072] Preferably, the preset threshold is 0.5, and the preset scaling factor is 0.5-0.6.

[0073] In the present invention, the first attention score corresponding to each patch reflects the relative importance of each patch in the STAS prediction decision. In order to enable the model to learn more generalized feature representations, a scaling-aware attention mechanism is designed. Based on the first attention score A, the first attention score is first normalized to obtain the second attention score, and then the second attention score that exceeds the preset threshold is scaled to reduce the model's dependence on some high-attention patches (instances).

[0074] In an optional embodiment, the training method of the SMILE model includes:

[0075] Step (a): obtaining a lung cancer histopathology image set consisting of multiple WSI images that have completed STAS and non-STAS identification.

[0076] In the present invention, the lung cancer histopathology image set is selected from one of the STAS_TCGA image set, the STAS_CPTAC image set and the STAS_CSU image set.

[0077] In the present invention, the STAS_TCGA image set includes multiple lung cancer histopathology images downloaded from The Cancer Genome Atlas (TCGA), the STAS_CPTAC image set includes multiple lung cancer histopathology images downloaded from the Clinical Proteomic Tumor Analysis Consortium (CPTAC) project, and the STAS_CSU image set includes the collected and collated histopathology images of STAS lung cancer patients from the Second Xiangya Hospital of Central South University.

[0078] Specifically:

[0079] STAS_TCGA Image Collection: Relevant LUAD WSIs were downloaded from the TCGA website. Based on inclusion and exclusion criteria, 541 WSIs from PS (with an unknown number of patients) were collected. All WSIs were cross-diagnosed by three experienced pathologists to determine STAS status, type of disseminated lesions, and tumor type. In this study, the STAS TCGA dataset includes 155 STASWSIs and 269 non-STAS WSIs, along with corresponding patient survival time and status. 117 WSIs were excluded.

[0080] STAS_CPTAC image set: 1,139 WSIs were obtained from CPTAC4. Based on the inclusion and exclusion criteria, 304 PS WSIs were labeled as STAS, 191 were labeled as non-STAS, and 640 WSIs were excluded. Included image data were cross-diagnosed by three pathologists to determine pathology type, including STAS status.

[0081] STAS_CSU Image Collection: From April 2020 to December 2023, 356 patients diagnosed with lung cancer (specifically STAS) who underwent pulmonary nodule resection at Xiangya Second Hospital were selected to form a retrospective lung cancer cohort. Comprehensive clinical and pathological data for each patient were collected, including age, tumor size, lymph node metastasis, distant metastasis, clinical stage, time to recurrence, and survival time and status. Two experienced pathologists independently reviewed each patient's pathological data, including frozen and paraffin-embedded hematoxylin and eosin (H&E)-stained sections and immunohistochemistry (IHC) sections, to confirm the presence of STAS, the specific pathological subtype of any disseminated lesions, the detailed histological subtype of lung cancer, and the expression of key proteins (PD-L1, TP53, Ki-67, and ALK). In this cohort, there were 150 non-STAS patients and 206 STAS patients. Each patient's tumor specimen was cut into multiple paraffin blocks by a pathologist, and each paraffin block corresponded to multiple H&E-stained sections. A total of 1,290 frozen sections (FS) and permanent sections (PS), as well as 1,436 IHC sections, were collected and scanned. Of these, 247 FS sections contained 81 STAS and 158 non-STAS FSs, while 1,043 PS sections contained 585 STAS and 436 non-STAS PSs. All FS and PS sections were digitized into whole slide images (WSIs).

[0082] In the present invention, the inclusion criteria included: (1) diagnosis of LUAD, (2) availability of conventional pathological sections containing the primary tumor and adjacent non-tumor tissues, (3) detailed TNM staging, (4) high-quality sections without bending, wrinkling, blurring or color distortion, and (5) exclusion of sections containing only the primary tumor without adjacent non-tumor tissues.

[0083] Step (b) segmenting tissue regions, detecting background and blurring regions, performing segmentation processing, and image enhancement on each WSI image in the lung cancer histopathology image set to obtain multiple packages, each of which includes multiple instances.

[0084] In the present invention, the OTSU algorithm is used to segment tissue regions.

[0085] In the present invention, the segmentation process is to segment each WIS image into a plurality of image blocks of uniform size and shape, and the size of each image block is 256×256 pixels.

[0086] In the present invention, a WIS image divided into multiple image blocks is named as a package, and each image block is an instance, which is the same as the patch mentioned above.

[0087] In the present invention, the GAN model is used for image enhancement.

[0088] In the present invention, the techniques of segmenting tissue regions, detecting background, blurring regions, segmentation processing and image enhancement are all existing technologies and are not improved in the present invention.

[0089] Step (c) comprises a plurality of packages and labels corresponding to the packages into a data set, and splits the data set into a training set and a validation set, and trains the SMILE model based on five-fold cross-validation to obtain the trained SMILE model. The label corresponding to the package is 1 or 0, and when the label value is 1, it indicates that it is identified as STAS, and when the label value is 0, it indicates that it is identified as non-STAS.

[0090] When the SMILE model is trained based on five-fold cross validation, the process of outputting the prediction results is the same as the steps (1) to (5) included in the SMILE model outputting the prediction results described above, and will not be repeated here.

[0091] In the present invention, each instance in a packet is marked as 1 (STAS) or 0 (non-STAS). A packet with a tag value of 1 has at least one instance marked as 1, and a packet with a tag value of 0 has all instances marked as 0.

[0092] In this study, all histopathology images were reviewed by three experienced pathologists who performed microscopic examinations to determine the STAS label for each WSI. This double-blind protocol employed a cross-diagnosis approach to obtain accurate labeling, reduce pathologist bias, and minimize the risk of missed or misdiagnosis.

[0093] In the present invention, the ratio of the training set to the validation set is 4:1.

[0094] In the present invention, the parameters of the SMILE model are optimized using a two-class simplified LCE loss function, where the simplified LCE loss function formula is:

[0095] Among them, y is the true value, is the predicted value.

[0096] In the present invention, the training of the SMILE model uses the Ranger optimizer to adjust the model parameters, and the learning rate is 2×e -4 , the weight decay is 1×e -5 , the number of training rounds is more than 100, the batch size is 12, the threshold is set to 0.5~0.8, the scaling factor is 0.3~0.8, the feature dimension l is 768, the feature compression dimension d is 256, and the attention mechanism parameter e is 64.

[0097] The training termination conditions for the SMILE model training provided by the present invention can be reaching the maximum number of training rounds (epochs), monitoring the lack of improvement in the validation set indicators (Early Stopping), the gradient norm being less than a threshold, or the loss decrease being less than a threshold.

[0098] It should be noted that in the present invention, the SMILE model is trained using two NVIDIA RTX4090 GPUs on the PyTorch framework.

[0099] It should be noted that the present invention uses conventional technology to train the SMILE model. The difference from the existing technology is that the model is improved and a zoom-aware attention mechanism is introduced. At the same time, the dataset for model training is of high quality and provided by multiple institutions.

[0100] Example 1

[0101] Evaluating the performance of model diagnosis STAS

[0102] The performance of the SMILE model provided by the present invention and the existing 10 STAS diagnostic models in diagnosing STAS was compared on three STAS_TCGA image sets, the STAS_CPTAC image set, and the STAS_CSU image set. The evaluation indicators (accuracy and area under the curve AUC) of each STAS diagnostic model are detailed in Table 1.

[0103] Table 1 Evaluation indicators of various STAS diagnostic models

[0104]

[0105] As shown in Table 1, our method performs best on the CPTAC dataset and also performs well on the TCGA and CSU datasets, surpassing only a few other methods. On the TCGA dataset, our method achieves an accuracy of 0.6064 and an AUC of 0.5736, outperforming most other methods and surpassing only CLAM-SB (accuracy of 0.6017 and AUC of 0.5542) and DGMIL (accuracy of 0.5621 and AUC of 0.6084). On the CPTAC dataset, our method achieves an accuracy of 0.6450 and an AUC of 0.6517, the highest accuracy and AUC values ​​of all methods. On the CSU dataset, our method achieves an accuracy of 0.5655 and an AUC of 0.5979. Although the performance on this dataset is not as good as that on the CPTAC dataset, it still outperforms most other methods, second only to Maxpooling (accuracy 0.5751, AUC 0.6055) and DGMIL (accuracy 0.5116, AUC 0.5854).

[0106] Example 2

[0107] The effects of threshold settings and scaling factors on the evaluation indicators of the SMILE model were investigated through ablation experiments. Based on different threshold settings and scaling factors, the diagnostic STAS performance of the SMILE model provided by the present invention was compared in the STAS_TCGA image set and the STAS_CPTAC image set. The results are detailed in Table 2.

[0108] Table 2 Effects of different thresholds and scaling factors on the performance of the SMILE model

[0109]

[0110] It should be noted that "w / o" in Table 2 means that no threshold or scaling factor adjustment is set.

[0111] On the TCGA and CPTAC datasets, accuracy, AUC, and F1 scores generally showed an upward trend as the threshold and scaling factor decreased. The model proposed in this paper reached a local optimum on CPTAC with a threshold of 0.5 and a scaling factor of 0.5, and on TCGA with a threshold of 0.5 and a scaling factor of 0.6.

[0112] Please refer to Figure 3 According to a second aspect of the present invention, the present invention further provides a lung cancer STAS auxiliary diagnosis device 100 based on scale-aware multi-instance learning, the auxiliary diagnosis device 100 comprising:

[0113] The acquisition module 101 is used to acquire a slice image of the patient's lung tissue and preprocess the slice image to obtain a preprocessed pathological tissue image. The preprocessing includes digitizing the whole slide image, segmenting the tissue area using the OTSU algorithm, and detecting the background and blurred area.

[0114] The image enhancement module 102 is configured to segment the preprocessed pathological tissue image to obtain a first patch set including a plurality of patches with uniform size and shape, and perform image enhancement on the first patch set to obtain a second patch set.

[0115] The feature extraction module 103 is configured to extract features from each patch in the second patch set and obtain a feature matrix represented by vectorization, where each element of the feature matrix represents a feature extracted from a patch.

[0116] The diagnosis module 104 is used to input the feature matrix into the trained SMILE model to obtain a prediction result, and to assist in diagnosing whether the patient has lung cancer STAS or not based on the prediction result. The SMILE model uses PyTorch as a framework and includes a scale-aware attention mechanism.

[0117] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the working process of the auxiliary diagnosis device 100 described above can refer to the contents of the aforementioned method embodiment and will not be repeated here.

[0118] The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, several simple deductions and substitutions can be made without departing from the concept of the present invention, and all of these should be considered to fall within the scope of protection of the present invention.

Claims

1. A STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning, characterized by: The auxiliary diagnosis method comprises the following steps: Step S10: Acquire a slice image of the patient's lung tissue, and preprocess the slice image to obtain a preprocessed pathological tissue image, wherein the preprocessing includes digitizing the whole slide image, segmenting the tissue area, and detecting the background and blurred area; Step S20: segmenting the preprocessed pathological tissue image to obtain a first patch set including a plurality of patches with uniform size and shape, and performing image enhancement on the first patch set to obtain a second patch set; Step S30: extract features from each patch in the second patch set to obtain a feature matrix represented by vectorization, where each element of the feature matrix is ​​used to represent a feature extracted from a patch; Step S40: Input the feature matrix into the trained SMILE model to obtain a prediction result, and based on the prediction result, assist in diagnosing whether the patient has lung cancer STAS or not. The SMILE model uses PyTorch as a framework and includes a scale-aware attention mechanism.

2. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 1 is characterized in that: In step S40, the process of the SMILE model outputting the prediction result includes: Step (1): Based on the feature matrix and the attention score calculation formula, obtain a first attention score corresponding to each patch, wherein the attention score calculation formula is: represents the unregularized attention score, and are three initialized weight feature matrices, tanh represents the hyperbolic tangent function, σ represents the sigmoid activation function, and a i represents the first attention score corresponding to the i-th patch, e is the attention mechanism parameter, and d is the feature compression dimension; Step (2): Each first attention score is normalized by Max_Min to obtain multiple second attention scores corresponding to each patch. Step (3), comparing each second attention score with a preset threshold, scaling the second attention scores that are greater than the preset threshold based on a preset scaling factor, and updating the second attention score corresponding to the patch with the scaled attention score; Step (4), performing weighted summation on the features of each patch and the second attention score corresponding to the patch to obtain a fused feature; Step (5): using the fused features as input, obtaining a prediction result based on a linear layer with a Sigmoid activation function, wherein the prediction result is 1 or 0, where 1 represents STAS and 0 represents non-STAS.

3. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 2 is characterized in that: Adjusting the preset threshold and the preset scaling factor can dynamically adjust the area focused on by the attention mechanism, where the preset threshold is 0.5 and the preset scaling factor is 0.5 to 0.

6.

4. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to any one of claims 1 to 3, characterized in that: In step S30, the step of extracting each patch includes offline acquiring initialization features based on a hybrid model that integrates convolutional neural networks and Transformer architecture, and fine-tuning the initialization features based on the network structure of the online stage, wherein the network structure of the online stage includes three layers: BatchNorm1d, Linear, and ReLU.

5. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 1 is characterized in that: The training method of the SMILE model includes: Step (a), obtaining a lung cancer histopathology image set consisting of multiple WSI images that have completed STAS and non-STAS identification; Step (b), performing tissue region segmentation, background detection, blurred region segmentation, segmentation processing, and image enhancement on each WSI image in the lung cancer histopathology image set, to obtain multiple packages, each package including multiple instances; Step (c) comprises a plurality of packages and labels corresponding to the packages into a data set, and splits the data set into a training set and a validation set, and trains the SMILE model based on five-fold cross-validation to obtain the trained SMILE model. The label corresponding to the package is 1 or 0, and when the label value is 1, it indicates that it is identified as STAS, and when the label value is 0, it indicates that it is identified as non-STAS.

6. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 5 is characterized in that: The lung cancer histopathology image set is selected from one of the STAS_TCGA image set, the STAS_CPTAC image set and the STAS_CSU image set.

7. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 5 is characterized in that: The SMILE model was trained using the Ranger optimizer to adjust the model parameters, with a learning rate of 2×e -4 , the weight decay is 1×e -5 , the number of training rounds is more than 100, the batch size is 12, the threshold is set to 0.5~0.8, the scaling factor is set to 0.3~0.8, the feature dimension l is 768, the feature compression dimension d is 256, and the attention mechanism parameter e is 64.

8. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 1 is characterized in that: In step S20 , the patch is a square patch with a size of 256×256 pixels.

9. The STAS-assisted diagnosis method for lung cancer based on scale-aware multi-instance learning according to claim 1, characterized in that: In step S20, the GAN model is used to perform image enhancement.

10. A lung cancer STAS-assisted diagnosis device based on scale-aware multi-instance learning, characterized in that: The auxiliary diagnostic device comprises: An acquisition module is used to acquire slice images of the patient's lung tissue and preprocess the slice images to obtain preprocessed pathological tissue images, wherein the preprocessing includes digitizing the whole slide image, segmenting the tissue area, and detecting the background and blurred area; an image enhancement module, configured to segment the preprocessed pathological tissue image to obtain a first patch set including a plurality of patches having a uniform size and shape, and perform image enhancement on the first patch set to obtain a second patch set; a feature extraction module, configured to extract features from each patch in the second patch set and obtain a feature matrix represented by vectorization, wherein each element of the feature matrix is ​​used to represent a feature extracted from a patch; A diagnosis module is configured to input the feature matrix into a trained SMILE model to obtain a prediction result, and to assist in diagnosing whether the patient has lung cancer STAS or not based on the prediction result. The SMILE model is based on a PyTorch framework and includes a scale-aware attention mechanism.

Images