Bronchial cavity image recognition method based on MSGNet model

Abstract

The invention discloses a bronchial cavity image recognition method based on an MSGNet model, and relates to the technical field of medical image data processing and analysis, and the method comprises the steps: obtaining a bronchoscope image with label information, carrying out the preprocessing of the image, and dividing the image into a training set, a verification set and a test set according to the proportion; a ResNet model is used as a basic network architecture, a multi-scale gate convolutional layer is fused, and an MSGNet model is established; training the model by using the training set and the verification set to obtain a trained MSGNet model; and performing model testing on the test set by using the trained model to obtain a category corresponding to each image in the test set. The model integrates a multi-scale feature extraction and gate selection mechanism, so that the model can adaptively pay attention to an area with the maximum information amount in an input image, and can dynamically select the extracted effective features, thereby improving the accuracy and robustness of bronchial cavity position identification, and having a relatively high clinical practical value.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image data processing and analysis, and in particular to a bronchial cavity image recognition method based on a MSGNet model. Background Art

[0002] Compared with X-ray and computed tomography (CT), invasive bronchoscopy can provide the most realistic and direct observation of the trachea and bronchi. It has become an important tool for early screening, diagnosis, and treatment of respiratory diseases, including lung cancer, tuberculosis, and pneumonia. After a complex and time-consuming bronchoscopy, endoscopists must write a postoperative report as an important guide for subsequent diagnosis, treatment, and patient follow-up. However, when bronchoscopic images are taken during surgery, existing instruments do not record the corresponding bronchial lumen position. The asynchrony between bronchoscopy and postoperative report writing often requires endoscopists to spend extra time annotating important images, especially for abnormal bronchial branches. On the other hand, the bronchial tree can be subdivided into 18 or more bronchial branches, from the main bronchi to the lobar bronchi to the segmental bronchi. This extremely complex structure makes it a high-learning-cost task to correctly identify bronchial branches from recorded images. Therefore, the identification of the bronchial lumen position is completely dependent on experienced endoscopists, which undoubtedly increases their workload. In conclusion, manual labeling of bronchial lumen from bronchoscopic images in postoperative reports is a labor-intensive and challenging task. Therefore, the development of an advanced automatic image classification system may effectively improve this situation.

[0003] Recently, Convolutional Neural Networks (CNN) have been widely used in the analysis of gastrointestinal endoscopic images, but only a few studies have focused on the classification of bronchoscopic images. The feasibility of using CNN to distinguish the anatomical location of the bronchial lumen without lesions has been demonstrated, but the classification of abnormal bronchial lumens from bronchoscopic images has not been explored in previous studies. Lumen images captured during bronchoscopy usually come from abnormal branches and have diagnostic value, but bronchoscopic images of diseased lumens usually show airway stenosis or obstruction, which cannot be effectively identified and classified by existing CNN models. Summary of the invention

[0004] In view of the above problems and technical requirements, the inventors have proposed a bronchial lumen image recognition method based on the MSGNet model. The MSGNet model constructed by this method can help endoscopists automatically annotate the position information of bronchial lumen images captured during surgery. The technical solution of the present invention is as follows:

[0005] A bronchial lumen image recognition method based on the MSGNet model comprises the following steps:

[0006] Obtain bronchoscopic images with label information, pre-process the images, and then divide them into a training set, a validation set, and a test set in proportion; the label information is the category of the bronchial cavity in the bronchoscopic image, and the category is classified according to the location of the bronchial cavity;

[0007] A multi-scale gated neural network MSGNet model is established with the ResNet model as the basic network architecture. The model includes a feature extraction module, a residual learning module, a global information integration module, and a classification output module connected in sequence. Multi-scale gated convolutional layers are used in the feature extraction module and the residual learning module to enable the model to adaptively focus on the area with the largest amount of information in the input image.

[0008] Use the training set and validation set to train the model and obtain the trained MSGNet model;

[0009] The trained MSGNet model is used to test the model on the test set to obtain the category corresponding to each image in the test set.

[0010] Its further technical solution is that the feature extraction module includes a multi-scale gated convolution layer, a batch normalization layer, a ReLU activation function and a maximum pooling layer connected in sequence, which is used to extract multi-scale shallow image features from the preprocessed image data.

[0011] A further technical solution is that the residual learning module is used to extract deep image features from shallow image features output by the feature extraction module while avoiding gradient vanishing;

[0012] The residual learning module includes a plurality of first residual layers and second residual blocks, and every two first residual layers are connected by a second residual block;

[0013] The first residual layer includes a plurality of first residual blocks connected in sequence;

[0014] For the first residual block, the input feature is sequentially subjected to the first convolution, the ReLU activation function, and the second convolution, concatenated with the input feature itself, and then output through the ReLU activation function and used as the input feature of the next residual block;

[0015] For the second residual block, the input features are sequentially passed through the first convolution, ReLU activation function and second convolution, and the other way is passed through the multi-scale gate convolution layer. The two outputs are concatenated and then output through the ReLU activation function and used as the input features of the next first residual layer;

[0016] Among them, batch normalization layers are added after the first convolution, the second convolution, and the multi-scale gated convolution layers.

[0017] A further technical solution is that the global information integration module includes a global average pooling layer for integrating feature information of different channels.

[0018] Its further technical solution is that the classification output module includes a fully connected layer and a Softmax layer, which are used to classify the feature information after feature extraction and integration to generate a prediction result of the bronchial cavity category.

[0019] Its further technical solution is that the multi-scale gate convolution layer includes a feature extraction part and a feature selection part;

[0020] In the feature extraction part, three convolution kernels of different sizes are used to convert the input features into three dual-channel spatial features, and these three dual-channel spatial features are concatenated and multi-scale fusion features are generated through point-by-point convolution;

[0021] In the feature selection part, the multi-scale fusion features are divided into two parts, one part is used to represent the extracted features, and the other part is used to generate the gate value. The generated gate value dynamically weights the extracted important features through the Sigmoid activation function to obtain the optimized final output features.

[0022] A further technical solution is to pre-process the image, including:

[0023] The images that were unclear, had artifacts, had poor lighting conditions, had completely closed or blocked bronchial lumens, had abnormal images, and were not within the scope of the study were eliminated from the original image set;

[0024] The filtered images are resized to a uniform pixel size and then data augmentation is performed to increase data diversity and model robustness.

[0025] Its further technical solution is to use the Adam optimizer to optimize the learning process during model training, use ExponentialLR to adaptively adjust the learning rate, and add an early stopping mechanism to avoid model overfitting.

[0026] A further technical solution is to obtain bronchoscopic images with label information, including normal bronchial cavity images and diseased bronchial cavity images, and each divided data set includes these two types of images.

[0027] The beneficial technical effects of the present invention are:

[0028] The present invention proposes an improved ResNet variant MSGNet (Multi-Scale Gated Neural Network) model. The feature extraction module of the model adopts a series of convolution operations and pooling operations, which can efficiently extract image features such as edges, textures and shapes. By introducing a second residual block with MSG jump connection in the residual learning module, the learning ability and generalization ability of the model can be significantly improved without increasing the computational complexity. In conjunction with the global information integration module and the classification output module, the category of the bronchial cavity image to be classified can be predicted.

[0029] The MSGNet model uses an improved multi-scale gate convolution MSGConv to replace the traditional convolution, so that the model can adaptively focus on the area with the most information in the input image. MSGConv introduces multi-scale convolution to integrate features of different scales, so that the model not only focuses on the bronchial cavity contour but also on the bronchial cavity details; and through the fusion gating mechanism, the model can dynamically select the extracted effective features.

[0030] The MSGNet model is trained using a dataset constructed from a combination of normal bronchial lumen images and images of diseased bronchial lumens, which ensures the model's excellent performance in the task of abnormal bronchial lumen identification. By testing the model on the constructed test set, the proposed model outperforms existing methods in terms of accuracy and robustness in bronchial lumen location identification, and has high clinical practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the bronchial cavity image recognition method based on the MSGNet model provided in this application.

[0032] Figure 2 It is a schematic diagram of the MSGNet model structure provided in this application.

[0033] Figure 3 It is a schematic diagram of the structure of the multi-scale gated convolutional layer MSGConv provided in this application. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0035] Please refer to Figure 1 As shown, the present application provides a bronchial cavity image recognition method based on the MSGNet model, which specifically includes the following steps:

[0036] Step 1: Obtain bronchoscopic images with label information, preprocess the images, and divide them into training set, validation set, and test set in proportion.

[0037] In order to solve the problem that the existing methods deliberately ignore the diseased bronchial cavity, resulting in poor robustness of the recognition method and inability to apply to the real clinical environment bronchial cavity image recognition problem, this embodiment collects normal bronchial cavity images, diseased bronchial cavity images and label information corresponding to the images of patients undergoing white light bronchoscopy in the hospital. The label information is the category of bronchial cavity in the bronchoscope image, and the category is classified according to the location of the bronchial cavity, including: carina, left main bronchus, right main bronchus, left upper lobe bronchus, left lower lobe bronchus, right upper lobe bronchus, right intermediate bronchus, right middle lobe bronchus, and right lower lobe bronchus. The collected images were first screened by three experienced physicians. The inclusion criteria were as follows: (1) The images must have good clarity and must not have severe blur, noise, out-of-focus or artifacts; (2) The lighting conditions of the images should be relatively good, and the images should not be too dark or too bright, or have strong shadows or reflections; (3) The bronchial lumen in the image should not be completely closed or blocked, or the bronchial lumen structure should not be completely unrecognizable due to sputum, blood, etc.; (4) The images should not contain abnormal images caused by bronchoscope equipment failure, external interference or other non-bronchial lumen structures; (5) The bronchial lumen in the image should be included in the above 9 categories of bronchial lumens. Finally, a total of 5807 bronchoscopic images from 9 locations were collected, including 4296 images of normal bronchial lumens and 1511 images of bronchial lumens with lesions.

[0038] The screened images were resized to 224×224×3 pixels, and then data augmentation operations such as flipping and rotation were performed to increase data diversity and model robustness, providing a better data basis for subsequent feature extraction and classification. Subsequently, images of the normal and diseased bronchial lumen groups were extracted to construct a dataset, which was divided into a training set, a validation set, and a test set in a ratio of 70:15:15. Table 1 shows the distribution of each dataset.

[0039] Table 1. Distribution of images at different locations in each dataset

[0040]

[0041] Step 2: Establish a multi-scale gated neural network MSGNet model using the ResNet model as the basic network architecture.

[0042] Due to the effectiveness of ResNet models in training deep neural networks and powerful feature extraction, ResNet and its variants have been widely used in downstream medical image analysis. Therefore, the MSGNet model proposed in this application adopts ResNet-34 as its backbone. Figure 2The specific structure of the MSGNet model is demonstrated. The input image first passes through a feature extraction module consisting of a multi-scale gated convolution layer MSGConv, a batch normalization layer, a ReLU activation function, and a maximum pooling layer to obtain shallow image features, then passes through a residual learning module to obtain deeper and more detailed image features, and finally passes through the average pooling and classification output layer of a global information integration module to obtain the specific category corresponding to the image.

[0043] In this embodiment, the residual learning module includes four first residual layers and three second residual blocks, and every two first residual layers are connected by a second residual block. The first residual layer I and the first residual layer II each include three first residual blocks connected in sequence, the first residual layer III includes five first residual blocks connected in sequence, and the first residual layer IV includes two first residual blocks connected in sequence. The first and second residual blocks each include two connections, one of which includes a first convolution, a ReLU activation function, and a second convolution connected in sequence, and the sizes of the first and second convolution kernels are set to 3×3. For the first residual block, the input feature is concatenated with the input feature of another path after passing through the sequential connection path, and then output through the ReLU activation function, and used as the input feature of the next residual block, such as Figure 2 The difference between the second residual block and the first residual block is that the multi-scale gate convolution layer MSGConv is introduced in another channel, that is, for the second residual block, the input features are respectively connected through the sequential connection channel and MSGConv in another channel, and the two outputs are concatenated and then output through the ReLU activation function and used as the input features of the next first residual layer, as shown in Figure 2 The design of the above residual learning module can avoid the gradient vanishing problem while maintaining the network depth, thereby improving the training effect of deep networks and the accuracy of the model. Adding batch normalization after each convolution operation (including the first convolution, the second convolution, and MSGConv) helps normalize the feature distribution, and the ReLU activation function introduces nonlinearity, which helps improve model training and convergence.

[0044] In addition, in order to enable the model to adaptively focus on the area with the most information in the input image and improve the recognition performance, the present invention designs a multi-scale gate convolution layer MSGConv to replace the traditional convolution layer. Figure 3As shown in the figure, MSGConv includes two parts: feature extraction and feature selection. The feature extraction part first uses three convolution kernels of different sizes to convert the input features into three dual-channel spatial features, concatenates these three dual-channel spatial features and generates multi-scale fusion features through point-by-point convolution. The feature selection part integrates the gating mechanism and divides the multi-scale fusion features into two parts through the torch.split function. One part is used to represent the extracted features, and the other part is used to generate the gate value. The generated gate value dynamically weights the extracted important features through the Sigmoid activation function, so that the model can automatically adjust the attention to each feature, thereby obtaining the optimized final output features.

[0045] The global information integration module is used to integrate the feature information of different channels to further enhance the model's ability to recognize complex structures, thereby improving the accurate recognition of the bronchial lumen location. The classification output module is used to classify the feature information after feature extraction and integration, and generate the prediction results of the bronchial lumen category, which includes a fully connected layer and a Softmax layer, which are used to map the feature vector to a specific category label.

[0046] Step 3: Train the model using the training set and the validation set to obtain a trained MSGNet model. The present invention implements the MSGNet model under the Pytorch framework and trains it on an NVIDIA A100 GPU equipped with 40G video memory. The initial learning rate of the training is 0.0001, the batch input of each training is 32, and the loss function adopts the cross entropy loss function. During the model training process, the Adam optimizer is used to optimize the learning process, the ExponentialLR is used to adaptively adjust the learning rate, and an early stopping mechanism is added to avoid overfitting problems in the model. Finally, the MSGNet model constructed in step 2 is trained using pre-labeled training set data and validation set data to obtain a trained MSGNet.

[0047] Step 4: Use the trained MSGNet model to test the model on the test set to obtain the category corresponding to each image in the test set, thereby providing reference information for bronchoscopic navigation or diagnostic assistance.

[0048] This embodiment uses indicators such as recognition accuracy, precision, recall, and F1-Score to evaluate the performance of the model. The trained MSGNet model is then compared with some existing state-of-art models, including ResNet-34, EfficientNet, VGG-16, GoogleNet, and SwinTransformer. Table 2 gives the specific evaluation results.

[0049] Table 2. Comparison of MSGNet model with state-of-art model (unit: %)

[0050]

[0051] As can be seen from the table, the average recognition accuracy of the MSGNet model for normal and diseased bronchial lumen images is 93.35% and 87.83%, respectively, which is higher than all other models. It also performs well in precision (92.69% and 89.79%, respectively), recall (92.58% and 86.57%, respectively), and F1 score (92.36% and 87.80%, respectively). In addition, the MSGNet model has similar scores in the normal group and the diseased group, indicating that it can eliminate noise in the diseased bronchial lumen and show its strong robustness and generalization ability.

[0052] The above is only a preferred embodiment of the present application, and the present invention is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included in the protection scope of the present invention.

Claims

1. A bronchial lumen image recognition method based on the MSGNet model, characterized in that: The method comprises: Obtaining a bronchoscopic image with label information, preprocessing the image, and then dividing it into a training set, a validation set, and a test set in proportion; wherein the label information is the category of the bronchial cavity in the bronchoscopic image, and the category is classified according to the location of the bronchial cavity; A multi-scale gated neural network MSGNet model is established with the ResNet model as the basic network architecture, wherein the model includes a feature extraction module, a residual learning module, a global information integration module and a classification output module connected in sequence; wherein a multi-scale gated convolutional layer is used in the feature extraction module and the residual learning module to enable the model to adaptively focus on the area with the largest amount of information in the input image; The model is trained using the training set and the validation set to obtain a trained MSGNet model; The trained MSGNet model is used to perform model testing on the test set to obtain the category corresponding to each image in the test set.

2. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: The feature extraction module includes the multi-scale gated convolution layer, batch normalization layer, ReLU activation function and maximum pooling layer connected in sequence, which are used to extract multi-scale shallow image features from the preprocessed image data.

3. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: The residual learning module is used to extract deep image features from shallow image features output by the feature extraction module while avoiding gradient vanishing; The residual learning module includes a plurality of first residual layers and second residual blocks, and every two first residual layers are connected by a second residual block; The first residual layer includes a plurality of first residual blocks connected in sequence; For the first residual block, the input feature is sequentially subjected to the first convolution, the ReLU activation function, and the second convolution, concatenated with the input feature itself, and then outputted through the ReLU activation function and used as the input feature of the next residual block; For the second residual block, the input feature passes through the first convolution, ReLU activation function and second convolution in sequence, and the other way passes through the multi-scale gate convolution layer. The two outputs are concatenated and then output through the ReLU activation function and used as the input feature of the next first residual layer; Among them, a batch normalization layer is added after the first convolution, the second convolution and the multi-scale gate convolution layer.

4. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: The global information integration module includes a global average pooling layer, which is used to integrate feature information of different channels.

5. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: The classification output module includes a fully connected layer and a Softmax layer, which are used to classify the feature information after feature extraction and integration to generate a prediction result of the bronchial cavity category.

6. The bronchial cavity image recognition method based on the MSGNet model according to any one of claims 1 to 3, characterized in that: The multi-scale gated convolutional layer includes a feature extraction part and a feature selection part; In the feature extraction part, three convolution kernels of different sizes are used to convert the input features into three dual-channel spatial features, and the three dual-channel spatial features are concatenated and multi-scale fusion features are generated through point-by-point convolution; In the feature selection part, the multi-scale fusion feature is divided into two parts, one part is used to represent each extracted feature, and the other part is used to generate a gate value. The generated gate value dynamically weights the extracted important features through the Sigmoid activation function to obtain the optimized final output feature.

7. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: Preprocessing the image includes: The images that were unclear, had artifacts, had poor lighting conditions, had completely closed or blocked bronchial lumens, had abnormal images, and were not within the scope of the study were eliminated from the original image set; The filtered images are resized to a uniform pixel size and then data augmentation is performed to increase data diversity and model robustness.

8. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: During the model training process, the Adam optimizer is used to optimize the learning process, the ExponentialLR is used to adaptively adjust the learning rate, and an early stopping mechanism is added to avoid overfitting of the model.

9. The bronchial cavity image recognition method based on the MSGNet model according to claim 1, characterized in that: The bronchoscopic images with label information obtained include normal bronchial cavity images and bronchial cavity images with lesions, and each divided data set includes these two types of images.

Images