A bronchial lumen image recognition method based on a MSGNet model
Patent Information
- Application Number
- CN202411323128.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2026-10-13
- Estimated Expiration
- 2044-09-23
AI Technical Summary
[0003]最近，卷积神经网络(Convolutional Neural Networks，CNN)已被广泛应用于胃肠内窥镜图像的分析，然而只有少数专注于对支气管镜图像进行分类的研究将CNN应用于支气管镜检查，也证明了卷积神经网络区分未发生病变的支气管管腔解剖位置的可行性，但在之前的研究中，从支气管镜图像中对异常支气管管腔的分类仍未得到探索
[0028]本发明提出了一个改进ResNet变体MSGNet(Multi-Scale Gated NeuralNetwork,多尺度门控神经网络)模型，该模型的特征提取模块采用了一系列卷积操作和池化操作，能够高效地提取图像的边缘、纹理和形状等特征，残差学习模块中通过引入带MSG跳跃连接的第二残差块，能够在不增加计算复杂度的情况下，显著提高模型的学习能力和泛化能力，配合全局信息整合模块和分类输出模块，可以预测待分类的支气管腔图像类别。
Smart Images
Figure CN119229195B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image data processing and analysis technology, and in particular to a method for bronchial lumen image recognition based on the MSGNet model. Background Technology
[0002] Compared to X-rays and computed tomography (CT), invasive bronchoscopy offers the most realistic and direct visualization of the trachea and bronchi. It has become a vital tool for the early screening, diagnosis, and treatment of respiratory diseases, including lung cancer, tuberculosis, and pneumonia. Following a complex and time-consuming bronchoscopy, endoscopists must write a postoperative report, serving as an important guide for subsequent diagnosis, treatment, and patient follow-up. However, current instruments do not record the corresponding bronchial lumen locations when bronchoscopic images are taken during the procedure. This asynchrony between bronchoscopy and postoperative report writing often requires endoscopists to spend extra time annotating important images, especially for abnormal bronchial branches. Furthermore, the bronchial tree, from the main bronchus to the lobar bronchi and then to the segmental bronchi, can be subdivided into 18 or more bronchial branches. This extremely complex structure makes correctly identifying bronchial branches from recorded images a task with a high learning curve. Therefore, the identification of bronchial lumen locations relies entirely on experienced endoscopists, undoubtedly increasing their workload. In summary, manually marking bronchial lumens from bronchoscopic images in postoperative reports is a labor-intensive and challenging task. Therefore, developing an advanced automated image classification system could effectively improve this situation.
[0003] Recently, Convolutional Neural Networks (CNNs) have been widely used in the analysis of gastrointestinal endoscopic images. However, only a few studies focusing on the classification of bronchoscopic images have applied CNNs to bronchoscopy, demonstrating the feasibility of using CNNs to distinguish the anatomical location of bronchial lumens without lesions. Previous studies, however, have not explored the classification of abnormal bronchial lumens from bronchoscopic images. Bronchial images captured during bronchoscopy often originate from abnormal branches and have diagnostic value; however, bronchoscopic images of diseased lumens typically show airway narrowing or obstruction, which existing CNN models cannot effectively identify and classify. Summary of the Invention
[0004] To address the aforementioned problems and technical requirements, the inventors have proposed a bronchial lumen image recognition method based on the MSGNet model. The MSGNet model constructed by this method can help endoscopists automatically annotate the location information of bronchial lumen images captured during surgery. The technical solution of this invention is as follows:
[0005] A method for bronchial lumen image recognition based on the MSGNet model includes the following steps:
[0006] Acquire bronchoscope images with labeled information, preprocess the images, and divide them into training, validation, and test sets according to proportions; the label information is the bronchial lumen category in the bronchoscope image, and the category is classified according to the location of the bronchial lumen;
[0007] A multi-scale gated neural network model MSGNet was built based on the ResNet model as the basic network architecture. The model includes a feature extraction module, a residual learning module, a global information integration module, and a classification output module connected in sequence. Multi-scale gated convolutional layers were used in the feature extraction module and the residual learning module to enable the model to adaptively focus on the region with the most information in the input image.
[0008] The model is trained using the training and validation sets to obtain the trained MSGNet model;
[0009] The trained MSGNet model is tested on the test set to obtain the category corresponding to each image in the test set.
[0010] The further technical solution is that the feature extraction module includes a multi-scale gated convolutional layer, a batch normalization layer, a ReLU activation function and a max pooling layer connected in sequence, which is used to extract multi-scale shallow image features from the preprocessed image data.
[0011] A further technical solution is that the residual learning module is used to extract deep image features from the shallow image features output by the feature extraction module, while avoiding gradient vanishing.
[0012] The residual learning module includes multiple first residual layers and second residual blocks, with each pair of first residual layers connected by a second residual block;
[0013] The first residual layer includes a plurality of first residual blocks connected in sequence;
[0014] For the first residual block, the input features are sequentially passed through the first convolution, the ReLU activation function, and the second convolution. They are then concatenated with the input features themselves and passed through the ReLU activation function again for output, which is then used as the input features for the next residual block.
[0015] For the second residual block, the input features pass through the first convolution, the ReLU activation function and the second convolution in one path, and through the multi-scale gate convolutional layer in the other path. The outputs of the two paths are concatenated and then passed through the ReLU activation function to output the output, which is used as the input features of the next first residual layer.
[0016] In this process, batch normalization layers are added after the first convolution, the second convolution, and the multi-scale gate convolution layers.
[0017] A further technical solution is that the global information integration module includes a global average pooling layer, which is used to integrate feature information from different channels.
[0018] The further technical solution is that the classification output module includes a fully connected layer and a Softmax layer, which are used to classify the feature information after feature extraction and integration, and generate prediction results of bronchial lumen categories.
[0019] Its further technical solution is that the multi-scale gated convolutional layer includes a feature extraction part and a feature selection part;
[0020] In the feature extraction part, three convolutional kernels of different sizes are used to convert the input features into three dual-channel spatial features. These three dual-channel spatial features are then concatenated and multi-scale fusion features are generated through pointwise convolution.
[0021] In the feature selection part, the multi-scale fusion feature is divided into two parts: one part is used to represent the extracted features, and the other part is used to generate a gate. The generated gate is dynamically weighted by the Sigmoid activation function to obtain the optimized final output feature.
[0022] A further technical solution involves preprocessing the image, including:
[0023] Images that are unclear, contain artifacts, are poorly lit, have completely closed or blocked bronchial lumens, are abnormal, or are not within the scope of the study were removed from the original image set.
[0024] The filtered images were resized to a uniform pixel size before data augmentation was performed to increase data diversity and model robustness.
[0025] The further technical solution is to use the Adam optimizer to optimize the learning process during model training, use ExponentialLR to adaptively adjust the learning rate, and add an early stopping mechanism to avoid model overfitting.
[0026] A further technical solution is to acquire bronchoscope images with labeled information, including images of normal bronchial cavities and images of bronchial cavities with lesions, and to include both types of images in each of the divided datasets.
[0027] The beneficial technical effects of this invention are:
[0028] This invention proposes an improved ResNet variant, MSGNet (Multi-Scale Gated Neural Network) model. The feature extraction module of this model employs a series of convolution and pooling operations, which can efficiently extract features such as edges, textures, and shapes from images. By introducing a second residual block with MSG skip connections in the residual learning module, the learning and generalization abilities of the model can be significantly improved without increasing computational complexity. Combined with the global information integration module and the classification output module, it can predict the category of the bronchial lumen image to be classified.
[0029] The MSGNet model uses an improved multi-scale gated convolution, MSGConv, instead of traditional convolution, enabling the model to adaptively focus on the most informative regions in the input image. MSGConv introduces multi-scale convolution to integrate features at different scales, allowing the model to focus not only on the bronchial lumen outline but also on detailed bronchial information; and through the fusion gating mechanism, the model can dynamically select the effective features extracted.
[0030] The MSGNet model is trained using a dataset composed of images of normal and diseased bronchi, ensuring excellent performance in identifying abnormal bronchi. Testing the model on a constructed test set demonstrates that the proposed model outperforms existing methods in both accuracy and robustness in bronchi location identification, exhibiting high clinical value. Attached Figure Description
[0031] Figure 1 This is a flowchart of the bronchial lumen image recognition method based on the MSGNet model provided in this application.
[0032] Figure 2 This is a schematic diagram of the MSGNet model structure provided in this application.
[0033] Figure 3 This is a schematic diagram of the MSGConv multiscale gated convolutional layer structure provided in this application. Detailed Implementation
[0034] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.
[0035] Please refer to Figure 1 As shown, this application provides a method for bronchial lumen image recognition based on the MSGNet model, which specifically includes the following steps:
[0036] Step 1: Acquire bronchoscopic images with labeled information, preprocess the images, and divide them into training, validation, and test sets according to proportions.
[0037] To address the problem that existing methods deliberately ignore diseased bronchial lumens, resulting in poor robustness and inapplicability to real-world clinical bronchial image recognition, this embodiment collects images of normal and diseased bronchial lumens from patients undergoing white light bronchoscopy in hospitals, along with corresponding label information. The label information identifies the bronchial lumen category in the bronchoscopic images, categorized according to their location: carina, left main bronchus, right main bronchus, left upper lobe bronchus, left lower lobe bronchus, right upper lobe bronchus, right intermediate bronchus, right middle lobe bronchus, and right lower lobe bronchus. The collected images were first screened by three experienced physicians. The inclusion criteria were: (1) the images must have good clarity and should not have severe blurring, noise, defocusing, or artifacts; (2) the lighting conditions of the images should be relatively good and there should be no excessively dark or bright images, strong shadows, or reflections; (3) the bronchial lumens in the images should not be completely closed, blocked, or completely unidentifiable due to sputum, blood, or other reasons; (4) the images should not contain abnormal images caused by bronchoscope equipment failure, external interference, or other non-bronchial lumen structures; (5) the bronchial lumens in the images should be included in the above 9 types of bronchial lumens. In the end, a total of 5807 bronchoscopic images from 9 locations were collected, including 4296 images of normal bronchial lumens and 1511 images of lesioned bronchial lumens.
[0038] The filtered images were resized to 224×224×3 pixels, and then data augmentation operations such as flipping and rotating were performed to increase data diversity and model robustness, providing a better data foundation for subsequent feature extraction and classification. Subsequently, images of normal and diseased bronchial lumens were extracted to construct a dataset, which was divided into training, validation, and test sets in a 70:15:15 ratio. Table 1 shows the distribution of each dataset.
[0039] Table 1. Image distribution at different locations in each dataset
[0040]
[0041] Step 2: Build a multi-scale gated neural network model MSGNet based on the ResNet model as the basic network architecture.
[0042] Due to its effectiveness in training deep neural networks and powerful feature extraction, ResNet and its variants have been widely used in downstream medical image analysis. Therefore, the MSGNet model proposed in this application uses ResNet-34 as its backbone. Figure 2The specific structure of the MSGNet model is shown. The input image first passes through a feature extraction module consisting of a multi-scale gated convolutional layer MSGConv, a batch normalization layer, a ReLU activation function, and a max pooling layer to obtain shallow image features. Then, it passes through a residual learning module to obtain deeper and more refined image features. Finally, it passes through an average pooling and classification output layer of a global information integration module to obtain the specific category corresponding to the image.
[0043] In this embodiment, the residual learning module includes four first residual layers and three second residual blocks, with each pair of first residual layers connected by a second residual block. First residual layer I and first residual layer II each include three sequentially connected first residual blocks, first residual layer III includes five sequentially connected first residual blocks, and first residual layer IV includes two sequentially connected first residual blocks. Both the first and second residual blocks include two connections, one of which includes a sequentially connected first convolution, a ReLU activation function, and a second convolution, with both the first and second convolution kernels set to 3×3. For the first residual block, the input features are concatenated with the input features of the other connection path, then processed by the ReLU activation function, and the output is used as the input features for the next residual block, such as... Figure 2 The structure of the first residual block is shown in the diagram. The difference between the second and first residual blocks lies in the introduction of a multi-scale gated convolutional layer (MSGConv) in another path. Specifically, for the second residual block, the input features pass through the sequentially connected path and the MSGConv in the other path, respectively. The two outputs are concatenated and then passed through a ReLU activation function, serving as the input features for the next first residual layer, as shown in the diagram. Figure 2 The structure of the second residual block is shown in the diagram. Designing the aforementioned residual learning module can avoid the vanishing gradient problem while maintaining network depth, thereby improving the training effect and model accuracy of deep networks. Adding batch normalization after each convolutional operation (including the first convolution, second convolution, and MSGConv) helps normalize the feature distribution, while the ReLU activation function introduces non-linearity, both of which contribute to improving model training and convergence.
[0044] Furthermore, to enable the model to adaptively focus on the most informative regions in the input image and improve recognition performance, this invention designs a multi-scale gated convolutional layer (MSGConv) to replace the traditional convolutional layer. For example... Figure 3As shown, MSGConv consists of two parts: feature extraction and feature selection. The feature extraction part first uses three convolutional kernels of different sizes to transform the input features into three dual-channel spatial features. These three dual-channel spatial features are then concatenated and multi-scale fused features are generated through pointwise convolution. The feature selection part incorporates a gating mechanism. The `torch.split` function is used to divide the multi-scale fused features into two parts: one part represents the extracted features, and the other part generates a gate. The generated gate dynamically weights the important extracted features using a Sigmoid activation function, allowing the model to automatically adjust the attention given to each feature, thus obtaining the optimized final output features.
[0045] The global information integration module integrates feature information from different channels to further enhance the model's ability to recognize complex structures, thereby improving the accuracy of bronchial lumen location identification. The classification output module classifies the extracted and integrated feature information to generate prediction results for bronchial lumen categories. This module includes fully connected layers and softmax layers to map feature vectors to specific category labels.
[0046] Step 3: Train the model using the training and validation sets to obtain the trained MSGNet model. This invention implements the MSGNet model in the PyTorch framework and trains it on an NVIDIA A100 GPU equipped with 40GB of video memory. The initial learning rate is 0.0001, the batch input for each training iteration is 32, and the cross-entropy loss function is used. During model training, the Adam optimizer is used to optimize the learning process, ExponentialLR is used to adaptively adjust the learning rate, and an early stopping mechanism is added to avoid overfitting. Finally, the MSGNet model constructed in Step 2 is trained using pre-labeled training and validation set data to obtain the trained MSGNet.
[0047] Step 4: Test the trained MSGNet model on the test set to obtain the category corresponding to each image in the test set, thereby providing reference information for bronchoscopy navigation or diagnostic assistance.
[0048] This embodiment uses metrics such as recognition accuracy, precision, recall, and F1-Score to evaluate the model's performance. The trained MSGNet model is then compared with several existing state-of-the-art models, including ResNet-34, EfficientNet, VGG-16, GoogleNet, and SwinTransformer. Table 2 presents the specific evaluation results.
[0049] Table 2. Comparison of MSGNet model and state-of-the-art model (unit: %)
[0050]
[0051] As shown in the table, the MSGNet model achieved average recognition accuracies of 93.35% and 87.83% for normal and diseased bronchial lumen images, respectively, surpassing all other models. It also performed well in precision (92.69% and 89.79%, respectively), recall (92.58% and 86.57%, respectively), and F1 score (92.36% and 87.80%, respectively). Furthermore, the MSGNet model achieved similar scores in the normal and diseased groups, indicating its ability to eliminate noise in diseased bronchial lumen and demonstrating strong robustness and generalization ability.
[0052] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.
Claims
1. A method for bronchial lumen image recognition based on the MSGNet model, characterized in that, The method includes: A bronchoscopic image with labeled information is acquired. After preprocessing, the image is divided into a training set, a validation set, and a test set according to a certain ratio. The labeled information refers to the bronchial lumen category in the bronchoscopic image, and the category is classified according to the location of the bronchial lumen. A multi-scale gated neural network (MSGNet) model is established based on the ResNet model as the basic network architecture. The model includes a feature extraction module, a residual learning module, a global information integration module, and a classification output module connected in sequence. Multi-scale gated convolutional layers are used in the feature extraction module and the residual learning module to enable the model to adaptively focus on the region with the most information in the input image. The model is trained using the training set and the validation set to obtain a trained MSGNet model; The trained MSGNet model is tested on the test set to obtain the category corresponding to each image in the test set. The feature extraction module includes a multi-scale gated convolutional layer, a batch normalization layer, a ReLU activation function, and a max pooling layer connected in sequence, used to extract multi-scale shallow image features from the preprocessed image data. The residual learning module is used to extract deep image features from the shallow image features output by the feature extraction module, while avoiding gradient vanishing. The residual learning module includes multiple first residual layers and second residual blocks, with each pair of first residual layers connected by a second residual block; The first residual layer includes a plurality of first residual blocks connected in sequence; For the first residual block, the input features are sequentially passed through the first convolution, the ReLU activation function, and the second convolution. They are then concatenated with the input features themselves and passed through the ReLU activation function again for output, which is then used as the input features for the next residual block. For the second residual block, the input features pass through the first convolution, the ReLU activation function and the second convolution in one path, and through the multi-scale gate convolutional layer in the other path. The two outputs are concatenated and then passed through the ReLU activation function to output the output, which is used as the input features of the next first residual layer. In this configuration, a batch normalization layer is added after the first convolution, the second convolution, and the multi-scale gate convolutional layer. The multi-scale gated convolutional layer includes a feature extraction part and a feature selection part; In the feature extraction part, three convolutional kernels of different sizes are used to convert the input features into three dual-channel spatial features. These three dual-channel spatial features are then concatenated and multi-scale fusion features are generated through pointwise convolution. In the feature selection section, the multi-scale fusion feature is divided into two parts: one part is used to represent the extracted features, and the other part is used to generate a gate. The generated gate is dynamically weighted by the extracted important features through the Sigmoid activation function to obtain the optimized final output feature.
2. The bronchial lumen image recognition method based on the MSGNet model according to claim 1, characterized in that, The global information integration module includes a global average pooling layer, which is used to integrate feature information from different channels.
3. The bronchial lumen image recognition method based on the MSGNet model according to claim 1, characterized in that, The classification output module includes a fully connected layer and a Softmax layer, which are used to classify the extracted and integrated feature information and generate prediction results for the bronchial lumen category.
4. The bronchial lumen image recognition method based on the MSGNet model according to claim 1, characterized in that, Preprocessing the image includes: Images that are unclear, contain artifacts, are poorly lit, have completely closed or blocked bronchial lumens, are abnormal, or are not within the scope of the study were removed from the original image set. The filtered images were resized to a uniform pixel size before data augmentation was performed to increase data diversity and model robustness.
5. The bronchial lumen image recognition method based on the MSGNet model according to claim 1, characterized in that, During the model training process, the Adam optimizer is used to optimize the learning process, ExponentialLR is used to adaptively adjust the learning rate, and an early stopping mechanism is added to avoid model overfitting.
6. The bronchial lumen image recognition method based on the MSGNet model according to claim 1, characterized in that, The acquired bronchoscopic images with tagged information include images of normal bronchial cavities and images of bronchial cavities with lesions. Each dataset contains both types of images.