Subpleural lung lesion ultrasound image analysis method based on multi-task deep learning

Abstract

The application discloses a subpleural lung lesion ultrasonic image analysis method based on multi-task deep learning, which comprises the following steps: obtaining subpleural lung ultrasonic image data to be analyzed and inputting the data into a multi-task deep learning model based on unlabeled data self-supervised contrast learning pre-training; segmenting the ultrasonic image by using a segmentation branch in the model to obtain a lesion segmentation mask, wherein the segmentation branch adopts a U-Net architecture with a skip connection and is provided with a residual semantic module between an encoder and a decoder; extracting global context features corresponding to standardized data by using a classification branch, extracting multi-scale convolution features across branches from different levels of the U-Net encoder and an output end of the residual semantic module; and fusing and classifying the multi-scale convolution features and the global context features to output corresponding analysis results. The application realizes collaborative learning of lesion segmentation and benign and malignant classification, and improves the robustness and cross-center generalization capability of image analysis in a complex ultrasonic scene.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning. Background Technology

[0002] Lung cancer is the leading cause of cancer-related death worldwide. Because lung cancer is often discovered at an advanced stage, early identification of malignant lesions is crucial for improving prognosis and guiding individualized treatment. With the widespread use of chest imaging and the advancement of screening programs, peripheral lung lesions, especially subpleural lung lesions (SPLs), are being detected more and more frequently in clinical practice. Unlike central lung lesions, SPLs are directly adjacent to the pleura, avoiding the obstruction of sound beam propagation by normal inflated lung tissue. They offer advantages such as no ionizing radiation, real-time imaging, bedside accessibility, portability, and lower cost, making them particularly suitable for initial assessment, dynamic monitoring, and repeated follow-up, and possessing significant clinical research and application value.

[0003] However, current ultrasound diagnosis still heavily relies on operator experience and subjective judgment. Image acquisition and result interpretation exhibit significant individual differences, and consistency among different observers is often severely limited. Furthermore, some ultrasound and contrast-enhanced ultrasound features overlap between benign and malignant lesions, making accurate differentiation solely based on human experience extremely challenging. While deep learning has shown certain advantages in medical imaging tasks, current research on lung AI primarily focuses on CT and chest X-rays. The limited number of existing lung ultrasound AIs mainly focus on tasks such as B-line recognition and pneumonia assessment; systematic research on deep learning systems for differentiating benign and malignant SPLs is still lacking. More seriously, traditional fully supervised learning heavily relies on a large amount of high-quality manual segmentation and annotation, resulting in extremely high data acquisition costs and strong subjectivity. Simultaneously, existing pure classification networks are easily affected by background tissue and irrelevant echoes when processing ultrasound images and cannot provide a basis for model decision-making, thus acting as "black box classifiers" that are difficult for clinicians to verify and trust.

[0004] Therefore, it is necessary to propose an interactive method for ultrasound image lesion analysis and interpretability based on self-supervised multi-task learning to solve the problems of insufficient accuracy in automatic lesion identification, limited feature representation ability, weak cross-scene adaptability, and difficulty in interpreting analysis results in the existing technology. Summary of the Invention

[0005] In view of this, the present invention provides a method and system for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning, in order to solve the technical problems in the clinical diagnosis of subpleural lung lesions by ultrasound, such as high dependence on human experience, strong subjective inconsistency, and the scarcity of high-quality annotations and the weak generalization of deep learning.

[0006] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning, comprising:

[0008] Acquire subpleural lung ultrasound image data to be analyzed;

[0009] The ultrasound image data is preprocessed to generate standardized ultrasound image data;

[0010] The standardized ultrasound image data is input into a pre-trained multi-task deep learning model; the pre-training process of the multi-task deep learning model is completed through self-supervised contrastive learning based on unlabeled training image data.

[0011] The lesions in the standardized ultrasound image data are segmented using the segmentation branch in the multi-task deep learning model to obtain a segmentation mask; wherein, the segmentation branch adopts a U-Net architecture with skip connections, and a residual semantic module is provided between the encoder and the decoder;

[0012] The global context features corresponding to the standardized ultrasound image data are extracted using the classification branch in the multi-task deep learning model, and multi-scale convolutional features are extracted from different levels of the encoder in the U-Net architecture and the output of the residual semantic module.

[0013] The multi-scale convolutional features and the global context features are fused, and classification prediction is performed based on the fused features to output the image analysis results of the subpleural lung lesions corresponding to the ultrasound image data.

[0014] Furthermore, the process of training based on unlabeled training image data through self-supervised contrastive learning includes:

[0015] For each unlabeled training image in the input, random data augmentation is performed to generate a first augmented view and a second augmented view;

[0016] The first enhanced view and the second enhanced view are input into the core feature extraction network of the multi-task deep learning model to extract feature vectors respectively.

[0017] The core feature extraction network is trained using a contrastive loss function based on NT-Xent to learn discriminative representations relevant to downstream tasks;

[0018] The mathematical expression for the contrastive loss function is as follows:

[0019] ;

[0020] In the formula, This represents the cosine similarity metric function. This represents the adjustment coefficient. Indicates the training batch size. For indicator functions, when The value is 1 when the time is right, and 0 otherwise. and These are the feature vectors corresponding to the first and second enhanced views, respectively.

[0021] Furthermore, the classification branch includes a Swing Transformer backbone network, a multi-scale feature extraction module, and a fusion classification module;

[0022] The step involves extracting global context features corresponding to the standardized ultrasound image data using the classification branch of the multi-task deep learning model, and extracting multi-scale convolutional features from different levels of the encoder and the output of the residual semantic module in the U-Net architecture, including:

[0023] The Swin Transformer backbone network is used to model long-distance pixel dependencies in the standardized ultrasound image data to extract global context features;

[0024] The multi-scale feature extraction module is used to extract corresponding hierarchical features from different levels of the encoder and the output of the residual semantic module in the U-Net architecture to obtain multi-scale convolutional features.

[0025] The fusion classification module fuses multi-scale convolutional features with global context features to obtain fused features, performs linear mapping and normalized probability calculation on the fused features, and outputs image analysis results.

[0026] Furthermore, the Swin Transformer backbone network is used to model long-range pixel dependencies in the standardized ultrasound image data, and global context features are extracted, including:

[0027] The standardized ultrasound image data is resampled to obtain a resampled image of a preset size;

[0028] The resampled image is input into the Swing Transformer backbone network, and its hierarchical window attention mechanism is used to extract long-range pixel dependencies to obtain the global context feature matrix.

[0029] Furthermore, multi-scale convolutional features are extracted from different levels of the encoder in the U-Net architecture and from the output of the residual semantic module, including:

[0030] Extract hierarchical feature maps with different spatial resolutions from the first to the fourth layers of the encoder in the U-Net architecture;

[0031] Extract the deep, high-dimensional features from the residual semantic module output between the encoder and decoder;

[0032] The hierarchical feature maps with different spatial resolutions and the deep high-dimensional features are subjected to global average pooling to form a multi-scale representation.

[0033] Furthermore, the training process of the multi-task deep learning model also includes supervised fine-tuning, using a multi-task joint loss function for collaborative optimization;

[0034] The multi-task joint loss function is expressed as follows:

[0035] ;

[0036] In the formula, For the total loss function, and To adjust the weighting coefficients for balancing multiple tasks; This is the segmentation loss function used to calculate the true lesion label. With predictive lesion segmentation mask The differences between them; This is the classification loss function used to calculate the true benign / malignant label. Compared with classification prediction results Cross-entropy between them.

[0037] Furthermore, the segmentation loss function It is a multi-constraint loss function composed of Dice loss and binary cross-entropy loss with equal weights or a preset weighting ratio.

[0038] Furthermore, after outputting the image analysis results of the lesions corresponding to the ultrasound image data, the method also includes: constructing interactive visualization auxiliary information based on the intermediate state and prediction output of the multi-task deep learning model;

[0039] The interactive visualization auxiliary information includes: the lesion segmentation mask, the benign and malignant prediction probability, the Grad-CAM heatmap of the image pixel interest area that contributes the most to the model discrimination, and the relative gain contribution map of each feature branch, which are synchronously rendered and presented on the auxiliary diagnostic terminal interface.

[0040] Secondly, the present invention also discloses a subpleural lung lesion ultrasound image analysis system based on multi-task deep learning, used to perform the method described in the above technical solution, comprising:

[0041] The data acquisition module is used to acquire subpleural lung ultrasound image data to be analyzed.

[0042] The preprocessing module is used to preprocess the ultrasound image data to generate standardized ultrasound image data;

[0043] The self-supervised learning module is used to input the standardized ultrasound image data into a pre-trained multi-task deep learning model; the pre-training process of the multi-task deep learning model is completed through self-supervised contrastive learning based on unlabeled training image data.

[0044] The segmentation module is used to segment lesions in the standardized ultrasound image data using the segmentation branch in the multi-task deep learning model to obtain a segmentation mask; wherein the segmentation branch adopts a U-Net architecture with skip connections, and a residual semantic module is provided between the encoder and the decoder.

[0045] The feature extraction module is used to extract global context features corresponding to the standardized ultrasound image data using the classification branch in the multi-task deep learning model, and to extract multi-scale convolutional features from different levels of the encoder in the U-Net architecture and the output of the residual semantic module.

[0046] The output module is used to fuse the multi-scale convolutional features and the global context features, perform classification prediction based on the fused features, and output the image analysis results of the subpleural lung lesions corresponding to the ultrasound image data.

[0047] Thirdly, the present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the above technical solution.

[0048] Compared with existing technologies, the ultrasound image analysis method for subpleural lung lesions based on multi-task deep learning proposed in this invention has the following advantages:

[0049] (1) This method introduces a self-supervised contrastive learning pre-training mechanism, enabling the core feature extraction network to spontaneously learn robust low-level and high-level representations with high discriminative power against ultrasound artifacts, noise, and attenuation using only unlabeled raw ultrasound image data before supervised fine-tuning. This solves the pain points of scarce clinical ultrasound labeled data and poor cross-center generalization ability, greatly reducing the absolute dependence of the algorithm model on expert-annotated samples, and substantially enhancing the system's cross-center generalization performance and noise resistance when facing data from different ultrasound instruments and medical institutions.

[0050] (2) Unlike the traditional dual-path independent network that separates segmentation and classification, this invention adopts a multi-task joint learning framework with a shared encoder. The classification branch directly reuses the hierarchical features of the segmentation branch (U-Net) encoder and the output of multi-scale lesion features from the residual semantic module, and fuses them with the global context features extracted by the classification backbone network. This provides spatial prior information of lesion regions for the classification task, thereby enhancing the discrimination ability of lesion regions, improving model training efficiency and generalization ability across center data.

[0051] (3) Unlike traditional deep learning black-box classification models, this invention outputs pixel-level lesion segmentation masks, model-focused region heatmaps, and feature branch contribution visualization results simultaneously with the predicted probabilities of benign and malignant lesions, thus realizing a visual explanation of the model analysis process. Through multi-task joint supervision and multi-scale feature fusion, the interpretability of the model analysis process is improved, providing clinicians with intuitive and verifiable auxiliary diagnostic evidence, and helping to improve the consistency and stability of image analysis results. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the ultrasound image analysis method for subpleural lung lesions based on multi-task deep learning provided by the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of the multi-task deep learning model provided by the present invention;

[0054] Figure 3 Radar graph of ablation experimental results provided by the present invention;

[0055] Figure 4 This is a schematic diagram of the structure of the ultrasound image analysis system for subpleural lung lesions based on multi-task deep learning provided by the present invention. Detailed Implementation

[0056] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0057] Please see Figure 1 This embodiment provides a method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning, including:

[0058] Step S101: Acquire the subpleural lung ultrasound image data to be analyzed;

[0059] Step S102: Preprocess the ultrasound image data to generate standardized ultrasound image data;

[0060] Step S103: Input the standardized ultrasound image data into a pre-trained multi-task deep learning model with a shared encoder structure; the pre-training process of the multi-task deep learning model is completed by self-supervised contrastive learning based on open-source unlabeled ultrasound training image data, and the pre-trained parameters obtained after learning are loaded after completion.

[0061] Step S104: Extract lesion features using a shared encoder, and segment the lesions in the standardized ultrasound image data using the segmentation branch of the multi-task deep learning model to obtain a segmentation mask; wherein, the segmentation branch adopts a U-Net architecture with skip connections, and a residual semantic module is provided between the encoder and the decoder.

[0062] Step S105: Utilize the classification branch in the multi-task deep learning model to reuse the lesion features generated by the shared encoder at different levels, and combine them with the output of the residual semantic module to construct multi-scale lesion features, while extracting the global context features corresponding to the standardized ultrasound image data.

[0063] Step S106: The multi-scale lesion features output by the shared encoder are fused with the global context features, and classification prediction is performed based on the fused features to output the image analysis results of the subpleural lung lesions corresponding to the ultrasound image data.

[0064] The proposed method for ultrasound image analysis of subpleural lung lesions based on multi-task deep learning in this invention first acquires the ultrasound image data of the subpleural lung lesions to be analyzed and performs normalization preprocessing to generate standardized ultrasound image data. This standardized data is then input into a multi-task deep learning model pre-trained based on self-supervised contrastive learning of unlabeled original images. Second, using the segmentation branch of the model, a U-Net architecture with skip connections and residual semantic modules is used to perform high-precision segmentation of lesions in the standardized images to output a lesion segmentation mask. Simultaneously, a classification branch is used to extract global contextual features, and multi-scale convolutional features from different levels of the U-Net encoder and the residual semantic module are extracted across branches. Finally, a fusion classification mechanism is used to perform channel-level concatenation and fusion of the multi-scale convolutional features with lesion spatial localization constraints and the long-distance global contextual features, ultimately outputting the final analysis result of the benign or malignant nature of the subpleural lung lesions. This scheme achieves high-precision lesion image analysis and accurate classification of benign and malignant lesions through deep multi-task collaboration, significantly improving diagnostic robustness in complex ultrasound scenarios.

[0065] As a specific embodiment, the ultrasound image data to be analyzed is preprocessed to generate standardized ultrasound image data, specifically including:

[0066] Region of Interest Extraction: Identify the grid text region and the boundary of the fan-shaped or rectangular acoustic window in the ultrasound image data to be analyzed, remove medical text, background noise and black borders outside the window, and crop to obtain the region of interest image of the core lesion;

[0067] Spatial size standardization: Using bilinear interpolation or bicubic interpolation algorithms, the image of the region of interest of the core lesion is resampled to a preset uniform pixel size to generate a reference size image;

[0068] Image grayscale normalization: The reference size image is subjected to channel conversion and brightness alignment to eliminate brightness deviation caused by different ultrasound equipment gain and dynamic range, and the standardized ultrasound image data is generated.

[0069] Since raw ultrasound images (DICOM or JPG format) typically contain character information such as patient name, examination time, probe frequency, and mechanical index (MI), directly inputting them into the network can lead to severe overfitting or misjudgment by the model. This embodiment first uses a boundary detection algorithm or a preset mask matrix to automatically identify the effective area (fan-shaped or rectangular acoustic window) of the ultrasound beam emission. The black invalid background and text noise outside the window are physically cropped, retaining only the region of interest (ROI) image containing the pleural line and subpleural lung tissue.

[0070] Furthermore, due to differences in the number of probe array elements and sampling rate of ultrasound instruments from different manufacturers (such as GE, Philips, and Mindray), the output image resolution varies (e.g., or This embodiment employs a bicubic interpolation algorithm to uniformly scale or fill the cropped ROI image to a fixed resolution while maintaining the aspect ratio of the lesion's anatomical structure, in order to adapt to the input dimension requirements of the subsequent neural network.

[0071] In a preferred embodiment, the process of training based on unlabeled training image data through self-supervised contrastive learning includes:

[0072] For each unlabeled training image in the input, random data augmentation is performed to generate a first augmented view and a second augmented view;

[0073] The first enhanced view and the second enhanced view are input into the core feature extraction network of the multi-task deep learning model to extract feature vectors respectively.

[0074] The core feature extraction network is trained using a contrastive loss function based on NT-Xent to learn discriminative representations relevant to downstream tasks;

[0075] The mathematical expression for the contrastive loss function is as follows:

[0076] ;

[0077] In the formula, This represents the cosine similarity metric function. This represents the adjustment coefficient. Indicates the training batch size. For indicator functions, when The value is 1 when the time is right, and 0 otherwise. and These are the feature vectors corresponding to the first and second enhanced views, respectively.

[0078] As a specific implementation, the pre-training data came from the publicly available Lung UltrasoundDataset (figshare, DOI: 10.6084 / m9.figshare.30093577), containing 2756 B-mode lung ultrasound images. For each input image, two different views were generated through random data augmentation, and a contrastive learning strategy based on NT-Xentloss was used for training to learn discriminative representations relevant to downstream tasks. The weights obtained after pre-training were used to initialize the backbone networks for downstream segmentation and classification tasks.

[0079] Introducing self-supervised learning (SSL) SPLs for discrimination is key to improving the robustness and external generalization ability of the model in this application. Medical ultrasound images are typically characterized by blurred textures, significant noise interference, and large differences in equipment, while high-quality annotation relies heavily on experienced physicians to delineate and confirm each case. Therefore, in actual modeling, we often face the challenges of "limited annotations" and "heterogeneous distribution." In this context, the value of SSL lies not only in improving initial classification performance, but also in learning more transferable representations through unlabeled data. Azizi et al. showed in a large-scale medical image classification study that self-supervised pre-training combined with domain-specific unlabeled data can improve classification accuracy and robustness, and exhibit better stability in distribution-biased scenarios; Sowrirajan et al. also found in a chest X-ray study that MoCo pre-training can improve model representation ability and cross-dataset transferability. Consistent with these studies, our ablation experiments show that after removing SSL, the model's AUC on external test set 1 decreased from 0.794 to 0.705, and on external test set 2 from 0.761 to 0.662, with accuracy also decreasing from 0.799 and 0.715 to 0.750 and 0.684, respectively. This performance degradation is particularly pronounced on external data, suggesting that the representations learned by SSL do not only improve the training set fit but may also enhance the model's adaptability to cross-center data distributions. Notably, previous work in ultrasound scenarios also supports this trend. Jiao et al. demonstrated in ultrasound video representation learning that self-supervised learning can provide more effective initialization for downstream standard section recognition; Droste et al. further showed that self-supervised representation learning based on ultrasound physician visual attention patterns can also provide a meaningful feature base for subsequent tasks. In summary, in tasks like SPLs, which have high annotation costs, large image heterogeneity, and high external generalization requirements, SSL is not merely a technical add-on but a crucial foundation supporting model robustness and cross-center transferability.

[0080] In some embodiments, the training process of the multi-task deep learning model further includes supervised fine-tuning, using a multi-task joint loss function for collaborative optimization;

[0081] The multi-task joint loss function is expressed as follows:

[0082] ;

[0083] In the formula, For the total loss function, and To adjust the weighting coefficients for balancing multiple tasks; This is the segmentation loss function used to calculate the true lesion label. With predictive lesion segmentation mask The differences between them; This is the classification loss function used to calculate the true benign / malignant label. Compared with classification prediction results Cross-entropy between them. By jointly optimizing the shared feature extraction network and each task branch, the overall performance of lesion segmentation and benign / malignant classification, as well as the model's generalization ability, are improved.

[0084] As a preferred embodiment, the segmentation loss function It is a multi-constraint loss function composed of Dice loss and binary cross-entropy loss with equal weights or a preset weighting ratio.

[0085] As a specific implementation, a U-Net architecture with skip connections is used for automatic segmentation of SPL lesions in the segmentation branch. The encoder extracts local texture and boundary information layer by layer, while the decoder gradually restores the spatial resolution and outputs a lesion mask. To further enhance the network's expressive power and training stability, a residual block is introduced between the encoder and decoder, allowing information to flow deeper within the network, alleviating the gradient vanishing problem, and helping the network learn more complex features. The final segmentation mask serves as the lesion localization result, while the features output from each layer of the encoder are further used in subsequent classification branches.

[0086] In a preferred embodiment, the classification branch includes a Swing Transformer backbone network, a multi-scale feature extraction module, and a fusion classification module;

[0087] The step involves extracting global context features corresponding to the standardized ultrasound image data using the classification branch of the multi-task deep learning model, and extracting multi-scale convolutional features from different levels of the encoder and the output of the residual semantic module in the U-Net architecture, including:

[0088] The Swin Transformer backbone network is used to model long-distance pixel dependencies in the standardized ultrasound image data to extract global context features;

[0089] The multi-scale feature extraction module is used to extract corresponding hierarchical features from different levels of the encoder and the output of the residual semantic module in the U-Net architecture to obtain multi-scale convolutional features.

[0090] The fusion classification module fuses multi-scale convolutional features with global context features to obtain fused features, performs linear mapping and normalized probability calculation on the fused features, and outputs image analysis results.

[0091] Furthermore, the Swin Transformer backbone network is used to model long-range pixel dependencies in the standardized ultrasound image data, and global context features are extracted, including:

[0092] The standardized ultrasound image data is resampled to obtain a resampled image of a preset size;

[0093] The resampled image is input into the Swing Transformer backbone network, and its hierarchical window attention mechanism is used to extract long-range pixel dependencies to obtain the global context feature matrix.

[0094] As a specific example, in the classification branch, the original ultrasound image is first resampled to 224 × 224 and input into the Swin Transformer backbone network initialized by self-supervised pre-training to extract global contextual features (Swin features).

[0095] Simultaneously, multi-scale feature maps are extracted from different layers of the U-Net encoder, including R1, R2, R3, R4, and residual semantic block (RS) output features. These features are then processed by global average pooling (GAP) to form multi-scale representations. These multi-scale features are fused with the global features output by the Swin Transformer and input into the classification head, outputting an analysis result indicating whether the lesion is benign or malignant. In this invention, the classification branch reuses the multi-scale lesion features output by the segmentation encoder, avoiding repeated learning of lesion localization information. Furthermore, it combines global contextual features for joint classification, achieving feature sharing between localization and classification tasks, thus improving model training efficiency and cross-center generalization ability.

[0096] Specifically, the mathematical expression for feature fusion in the fusion classification module is as follows:

[0097]

[0098] In the formula, Indicates fusion characteristics, Concatenation operators indicating channel direction. This represents the global average pooling function. to This represents the feature maps extracted from different levels of the encoder in the U-Net architecture. This represents the features extracted from the output of the residual semantic module. This represents the global context feature.

[0099] The formulas for linear mapping and normalized probability calculation in the fusion classification module are as follows:

[0100]

[0101] In the formula, This represents the output probability matrix predicting the benign or malignant nature of subpleural lung lesions. This is the weight matrix. For bias vectors, This represents the normalized exponential activation function.

[0102] The above fusion strategy preserves both the local structural information based on the convolutional encoder and the global contextual dependency based on the Transformer, which helps to improve the analysis capabilities of complex lung lesion images.

[0103] like Figure 2 As shown, Figure 2 A schematic diagram of the multi-task deep learning model architecture provided by this invention is shown.

[0104] As a preferred embodiment, after outputting the benign or malignant analysis results of the subpleural lung lesions corresponding to the ultrasound image data, the method further includes: generating interactive visual auxiliary information based on the intermediate feature representation and prediction output of the multi-task deep learning model;

[0105] The interactive visualization auxiliary information includes: the lesion segmentation mask, the benign and malignant prediction probability, the Grad-CAM heatmap of the image pixel interest area that contributes the most to the model discrimination, and the relative gain contribution map of each feature branch, which are synchronously rendered and presented on the auxiliary diagnostic terminal interface.

[0106] To assess the contribution of each key component of the model to the segmentation and classification performance, this application designed a series of ablation experiments and compared them on the validation set (Val), internal test set (Test), and external test set (Ext_Test). All ablation models employed the same data partitioning, training strategy, and evaluation process as the full model to ensure comparability of results. Specifically, the following model variants were constructed:

[0107] (1) SLLA-UNet, which is the complete model used in this invention, includes self-supervised pre-training, a Swin Transformer branch and a multi-scale feature fusion module; (2) non-SSL, removes the self-supervised pre-training module and uses random initialization for the classification backbone; (3) non-MS, removes the multi-scale feature branch and retains the global features extracted by the Swin Transformer and the RS layer features for classification; (4) non-Swin, removes the Swin Transformer branch and uses only the multi-scale features from the U-Net encoder for classification; (5) non-MS-Swin, removes both the multi-scale features and the Swin Transformer branch and uses only the RS layer features for classification, used to evaluate the overall gain of the complete feature fusion strategy.

[0108] Among them, non-SSL is used to evaluate the contribution of self-supervised representation learning to the model's generalization ability, non-MS is used to verify the role of multi-scale convolutional features in lesion structure representation, non-Swin is used to evaluate the gain of global context semantics of Transformer, and non-MS-Swin is used to examine the change in model performance after the absence of two key feature branches.

[0109] A total of 1183 patients with subpleural lung lesions were ultimately included. 868 patients from centers I-III constituted the model development cohort and were randomly assigned in an 8:1:1 ratio to form the training set (n = 694), internal validation set (n = 87), and internal test set (n = 87). 245 patients from center IV and 70 patients from center V served as external test set 1 and external test set 2, respectively. Baseline clinical characteristics of each cohort are shown in Table 1. Among the seven baseline variables, there were statistically significant differences in age, BMI, maximum lesion diameter, lesion location, and the occurrence of pleural effusion among the different cohorts (all P < 0.05), while there were no statistically significant differences in gender and smoking history (all P > 0.05). Overall, the BMI and maximum lesion diameter in both external test cohorts were higher than in the development cohort, and the proportion of patients with pleural effusion in external test set 2 was significantly lower, suggesting a certain degree of heterogeneity in case composition between the external validation and development cohorts.

[0110] Table 1. Baseline characteristics of patients in the training set, internal test set, and external validation set.

[0111]

[0112] Validated results show that SLLA-UNet exhibits good joint segmentation and classification performance on the internal validation set, internal test set, external test set 1, and external test set 2. In the classification task, the model's AUC on the four datasets are 0.813, 0.764, 0.794, and 0.761, respectively, with external test sets 1 and 2 maintaining similar discriminative capabilities to the development stage. Apart from AUC, the model's accuracy, sensitivity, specificity, precision, and F1-score show limited overall variation across different datasets, with sensitivity generally high and specificity relatively low and somewhat fluctuating. In the segmentation task, SLLA-UNet maintains high levels of mDice and mIoU on all four datasets. Calibration curves show that the model's predicted probabilities are generally close to the ideal reference line compared to the actual observed probabilities; decision curves show that within a wide threshold probability range, SLLA-UNet's net gain is higher than the "treat-all" and "treat-none" strategies.

[0113] To further evaluate the impact of key modules in SLLA-UNet on model performance, this study compares the performance of the complete model and its four ablation variants (non-SSL, non-MS, non-Swin, and non-MS-Swin) on the validation set, internal test set, external test set 1, and external test set 2 (e.g., ...). Figure 3 (As shown). Overall, the complete model exhibits more balanced performance across different datasets.

[0114] In classification tasks, the performance degradation was most significant after removing self-supervised pre-training. Compared to the full model, the AUC of non-SSL on the internal test set, external test set 1, and external test set 2 decreased from 0.764, 0.794, and 0.761 to 0.659, 0.705, and 0.662, respectively; the corresponding accuracies decreased from 0.784, 0.799, and 0.715 to 0.643, 0.750, and 0.684, respectively. Simultaneously, the segmentation performance of non-SSL on external test set 2 also showed a significant decline, with mDice and mIoU decreasing from 0.806 and 0.682 to 0.748 and 0.605, respectively. These results indicate that self-supervised pre-training plays a crucial role in the model's classification and segmentation stability under cross-dataset conditions.

[0115] After removing the multi-scale feature branch (non-MS), the model's accuracy and sensitivity on the internal test set decreased from 0.7844 and 0.8716 to 0.7383 and 0.8057, respectively. On external test set 1, the AUC decreased from 0.794 to 0.776, and the sensitivity decreased from 0.897 to 0.869, although the specificity increased slightly (0.493 vs 0.510). On external test set 2, the AUC of non-MS was 0.772, with overall performance comparable to the full model and not significantly better. After removing the Swin branch (non-Swin), the model's AUC on the internal and external test sets 1 decreased to 0.726 and 0.750, respectively, lower than the full model's 0.764 and 0.794; the corresponding accuracies also decreased from 0.784 and 0.799 to 0.715 and 0.731. When multi-scale features and the Swin branch are removed simultaneously (non-MS-Swin), the model performance further deteriorates, with AUCs of 0.707 and 0.724 on external test set 1 and external test set 2, respectively, both lower than the full model. Meanwhile, this variant's F1-score on external test set 1 is only 0.805, lower than the full model's 0.871. The full model demonstrates a more balanced overall performance across multiple dimensions, including AUC, sensitivity, and segmentation metrics (mDice and mIoU), while all ablation models exhibit varying degrees of performance degradation, and their results on the prospective cohort (external test set 2) are sufficiently stable.

[0116] Therefore, the above results demonstrate that the organic combination of self-supervised pre-training, multi-scale convolutional features, and global context modeling constitutes a crucial foundation for the overall performance improvement and stability of the model. The model in this embodiment effectively alleviates the pain point of the extreme scarcity of high-quality labeled tags in the field of medical ultrasound through a self-supervised pre-training mechanism. Furthermore, by leveraging a unique cross-task feature fusion architecture, it deeply decouples and collaboratively optimizes local strong spatial boundary constraints with macroscopic long-distance semantic context in the underlying feature space. This breaks through the performance bottleneck of single-task networks in complex ultrasound artifact scenarios, significantly improving the accuracy, cross-center generalization, and clinical interpretability of the auxiliary diagnosis of subpleural pulmonary space-occupying lesions.

[0117] like Figure 4 As shown, this embodiment of the invention also provides a subpleural lung lesion ultrasound image analysis system 400 based on multi-task deep learning, used to execute the method described in the above technical solution, including:

[0118] The data acquisition module 401 is used to acquire subpleural lung ultrasound image data to be analyzed;

[0119] Preprocessing module 402 is used to preprocess the ultrasound image data to generate standardized ultrasound image data;

[0120] The self-supervised learning module 403 is used to input the standardized ultrasound image data into a pre-trained multi-task deep learning model with a shared encoder structure; the pre-training process of the multi-task deep learning model is completed through self-supervised contrastive learning based on unlabeled training image data.

[0121] The segmentation module 404 is used to extract lesion features using a shared encoder and segment the lesions in the standardized ultrasound image data through the segmentation branch of the multi-task deep learning model to obtain a segmentation mask; wherein, the segmentation branch adopts a U-Net architecture with skip connections and a residual semantic module is provided between the encoder and the decoder.

[0122] The feature extraction module 405 is used to reuse the lesion features of different levels generated by the shared encoder in the classification branch of the multi-task deep learning model, and to construct multi-scale lesion features in combination with the output of the residual semantic module, while extracting the global context features corresponding to the standardized ultrasound image data.

[0123] The output module 406 is used to fuse the multi-scale lesion features output by the shared encoder with the global context features, perform classification prediction based on the fused features, and output the image analysis results of the subpleural lung lesions corresponding to the ultrasound image data.

[0124] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the above technical solutions.

[0125] The computer-readable storage medium and computing device provided in the above embodiments of the present invention can be implemented with reference to the content specifically described above regarding the implementation of the ultrasound image analysis of subpleural lung lesions based on multi-task deep learning, and have similar beneficial effects to the ultrasound image analysis method of subpleural lung lesions based on multi-task deep learning described above, which will not be repeated here.

[0126] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning, characterized in that, include: Acquire subpleural lung ultrasound image data to be analyzed; The ultrasound image data is preprocessed to generate standardized ultrasound image data; The standardized ultrasound image data is input into a pre-trained multi-task deep learning model with a shared encoder structure; the pre-training process of the multi-task deep learning model is completed through self-supervised contrastive learning based on unlabeled training image data. The lesion features are extracted using a shared encoder, and the lesions in the standardized ultrasound image data are segmented through the segmentation branch of the multi-task deep learning model to obtain a segmentation mask; wherein, the segmentation branch adopts a U-Net architecture with skip connections, and a residual semantic module is provided between the encoder and the decoder; The classification branch in the multi-task deep learning model is used to reuse the lesion features at different levels generated by the shared encoder, and combined with the output of the residual semantic module to construct multi-scale lesion features. At the same time, the global context features corresponding to the standardized ultrasound image data are extracted. The multi-scale lesion features output by the shared encoder are fused with the global context features, and classification prediction is performed based on the fused features to output the image analysis results of the subpleural lung lesions corresponding to the ultrasound image data.

2. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 1, characterized in that, The process of training based on unlabeled training image data through self-supervised contrastive learning includes: For each unlabeled training image in the input, random data augmentation is performed to generate a first augmented view and a second augmented view; The first and second enhanced views are input into the core feature extraction network of the multi-task deep learning model to extract feature vectors respectively; the core feature extraction network is trained using the NT-Xent-based contrastive loss function to learn discriminative representations related to downstream tasks; The mathematical expression for the contrastive loss function is as follows: ； In the formula, This represents the cosine similarity metric function. This represents the adjustment coefficient. Indicates the training batch size. For indicator functions, when The value is 1 when the time is right, and 0 otherwise. and These are the feature vectors corresponding to the first and second enhanced views, respectively. After completing the self-supervised contrastive learning training, model initialization parameters suitable for subpleural lung ultrasound image analysis tasks are obtained and used to initialize the shared core feature extraction network to support the joint training of subsequent lesion segmentation and benign / malignant classification.

3. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 1, characterized in that, The classification branch includes a multi-scale lesion feature extraction module, a global context feature extraction module, and a fusion classification module, wherein; The method utilizes the classification branch in the multi-task deep learning model to reuse the lesion features generated by the shared encoder at different levels, and combines them with the output of the residual semantic module to construct multi-scale lesion features. Simultaneously, it extracts the global context features corresponding to the standardized ultrasound image data, including: The multi-scale lesion feature extraction module extracts corresponding hierarchical features from different levels of the encoder and the output of the residual semantic module in the U-Net architecture to obtain multi-scale lesion features. The global context feature extraction module extracts global context features, and the global context feature extraction module adopts the Swing Transformer network. The fusion classification module fuses multi-scale lesion features with global context features to obtain fused features, performs linear mapping and normalized probability calculation on the fused features, and outputs image analysis results.

4. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 3, characterized in that, The global context feature extraction module extracts global context features, including: The standardized ultrasound image data is resampled to obtain a resampled image of a preset size; The resampled image is input into the Swing Transformer backbone network, and its hierarchical window attention mechanism is used to extract long-range pixel dependencies to obtain the global context feature matrix.

5. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 1, characterized in that, The classification branch in the multi-task deep learning model reuses the lesion features at different levels generated by the shared encoder, including: Extract hierarchical feature maps with different spatial resolutions from the first to the fourth layers of the encoder in the U-Net architecture; Extract the deep, high-dimensional features from the residual semantic module output between the encoder and decoder; The hierarchical feature maps with different spatial resolutions and the deep high-dimensional features are subjected to global average pooling to form a multi-scale representation.

6. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 1, characterized in that, The method further includes: After completing self-supervised pre-training, the multi-task deep learning model is jointly optimized using supervised fine-tuning, and a multi-task joint loss function is used for collaborative training of lesion segmentation and benign / malignant classification tasks. The multi-task joint loss function is expressed as follows: ； In the formula, For the total loss function, and To adjust the weighting coefficients for balancing multiple tasks; This is the segmentation loss function used to calculate the true lesion label. With predictive lesion segmentation mask The differences between them; This is the classification loss function used to calculate the true benign / malignant label. Compared with classification prediction results Cross-entropy between them.

7. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 6, characterized in that, The segmentation loss function It is a multi-constraint loss function composed of Dice loss and binary cross-entropy loss with equal weights or a preset weighting ratio.

8. The method for analyzing ultrasound images of subpleural lung lesions based on multi-task deep learning according to claim 1, characterized in that, After outputting the image analysis results of the lesion corresponding to the ultrasound image data, the method further includes: constructing interactive visualization auxiliary information based on the intermediate state and prediction output of the multi-task deep learning model; The interactive visualization auxiliary information includes: the lesion segmentation mask, the probability of benign or malignant prediction, the heatmap of the model's area of ​​interest, and the visualization results of each feature branch, which are synchronously rendered and presented on the auxiliary diagnostic terminal interface.

9. A subpleural lung lesion ultrasound image analysis system based on multi-task deep learning, characterized in that, For performing the method according to any one of claims 1-8, comprising: The data acquisition module is used to acquire subpleural lung ultrasound image data to be analyzed. The preprocessing module is used to preprocess the ultrasound image data to generate standardized ultrasound image data; The self-supervised learning module is used to input the standardized ultrasound image data into a pre-trained multi-task deep learning model with a shared encoder structure; the pre-training process of the multi-task deep learning model is completed through self-supervised contrastive learning based on unlabeled training image data. The segmentation module is used to extract lesion features using a shared encoder and segment the lesions in the standardized ultrasound image data through the segmentation branch of the multi-task deep learning model to obtain a segmentation mask; wherein, the segmentation branch adopts a U-Net architecture with skip connections and a residual semantic module is provided between the encoder and the decoder. The feature extraction module is used to reuse the lesion features of different levels generated by the shared encoder in the classification branch of the multi-task deep learning model, and combine them with the output of the residual semantic module to construct multi-scale lesion features, while extracting the global context features corresponding to the standardized ultrasound image data. The output module is used to fuse the multi-scale lesion features output by the shared encoder with the global context features, perform classification prediction based on the fused features, and output the image analysis results of the subpleural lung lesions corresponding to the ultrasound image data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images