Magnetic control capsule endoscope image stomach anatomic structure recognition method based on deep learning

Abstract

The application discloses a method for identifying gastric anatomical structure based on magnetic control capsule endoscopy image of deep learning, screening the pictures output by the magnetic control capsule endoscopy detection, classifying according to the anatomical position, expanding the data set, and dividing into a training set and a test set; constructing, training and testing a multi-scale fusion pyramid visual Transformer network and a target detection network with an attention module, after the test, using the multi-scale fusion pyramid visual Transformer network to extract the global features of the image from the gastric magnetic control capsule endoscopy image, and classifying the magnetic control capsule endoscopy image; using the target detection network with the attention module to locate the classified pictures and identify the gastric anatomical structure. The application solves the problem of low resolution of the magnetic control capsule endoscopy examination pictures in the prior art, and can accurately identify the gastric anatomical position, further achieving the purpose of positioning the magnetic control capsule endoscopy in the stomach.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a method for recognizing gastric anatomical structure of magnetic capsule endoscopy image based on deep learning. BACKGROUND

[0002] Traditional upper gastrointestinal endoscopy is still the main method for diagnosing gastric diseases at present. However, it is an invasive examination, and patients will have various uncomfortable symptoms during the examination, and the tolerance is very poor. Although there is a painless examination method, it needs to be performed with general anesthesia in advance, and a part of patients with cardiovascular and cerebrovascular diseases and respiratory diseases cannot perform painless gastroscopy, and some patients will have uncomfortable symptoms after general anesthesia. Based on the above reasons, the traditional upper gastrointestinal endoscopy method is difficult to be applied and popularized in the screening of large-scale gastric diseases.

[0003] In 2006, Federico et al. proposed a gastrointestinal endoscopy capsule based on a magnetic shell, which can be manually controlled to move, called magnetic capsule endoscopy. It is a non-invasive, painless, comfortable and safe disposable endoscope, which can avoid cross infection during gastrointestinal examination and allow comprehensive observation of the gastric mucosa of the large gastric cavity and small intestine, overcoming the shortcomings that the traditional endoscope cannot examine the upper and lower digestive tract at the same time. In addition, doctors can also use external instruments and corresponding software to control the magnetic capsule endoscopy in the human stomach by using the magnetic field, so as to achieve the purpose of specialized examination of the whole stomach. Compared with the traditional endoscopy system, the magnetic capsule endoscopy will automatically generate a large number of picture data during examination. Usually, thousands of pictures are generated for a complete gastric detection, and tens of thousands of pictures are generated for a complete digestive tract detection, and the workload of doctors screening capsule endoscopy image data is very large. In addition, these pictures have lower resolution than traditional endoscopes, and are not easy to identify, which can easily cause misdiagnosis and missed diagnosis. Therefore, an efficient and accurate capsule endoscopy image intelligent identification system is needed to assist doctors to improve the diagnosis accuracy. Among them, accurately identifying the gastric anatomical site based on the capsule endoscopy image is an important problem that the capsule endoscopy image intelligent identification system needs to solve first, which can achieve the purpose of positioning the magnetic capsule endoscopy in the stomach. SUMMARY

[0004] The purpose of the present application is to provide a method for recognizing gastric anatomical structure of magnetic capsule endoscopy image based on deep learning, which can solve the problems of large number of pictures generated by magnetic capsule endoscopy examination, artificial screening workload, and low resolution and not easy to identify of these pictures in the prior art.

[0005] The present application solves the above problems by the following technical scheme:

[0006] A method for recognizing gastric anatomical structure of magnetic capsule endoscopy image based on deep learning, comprising:

[0007] Step S100, screening the pictures detected by the magnetic controlled capsule endoscope, classifying the pictures according to anatomical positions, taking the pictures as a capsule endoscope image data set, and expanding the capsule endoscope image data set by using affine transformation technology, and dividing the pictures of each category in the data set into a training set and a test set according to a preset ratio;

[0008] Step S200, respectively constructing a multi-scale fusion pyramid visual Transformer network and a target detection network with an attention module, training the multi-scale fusion pyramid visual Transformer network and the target detection network with the attention module by using the training set, and testing the trained multi-scale fusion pyramid visual Transformer network and the target detection network with the attention module by using the test set;

[0009] Step S300, after the test, using the multi-scale fusion pyramid visual Transformer network to extract the global features of the gastric magnetic controlled capsule endoscope images, classifying the magnetic controlled capsule endoscope images, and using the target detection network with the attention module to locate the classified pictures and identify the gastric anatomical structure.

[0010] The effective images are screened from the magnetic controlled capsule endoscope images, and then classified according to anatomical positions (for example, the fundus and the body of the stomach are the upper part of the stomach, the antrum, the pylorus and the gastric angle are the lower part of the stomach, and the small intestine is a separate category), and then the data set is expanded, and the training set and the test set are divided according to the same ratio. The multi-scale fusion pyramid visual Transformer model is constructed, and after training and testing, the multi-scale fusion pyramid visual Transformer model is used for classifying and identifying the upper part of the stomach, the lower part of the stomach and the small intestine in the magnetic controlled capsule endoscope images, the target detection network with the attention module is constructed, and after training and testing, the target detection network with the attention module is used for positioning the gastric angle and the pylorus in the lower part of the stomach in the magnetic controlled capsule endoscope images. The method proposed in the application can better extract global features of images and has better classification and identification performance in the case of large amount of data.

[0011] The construction method of the multi-scale fusion pyramid visual Transformer network is as follows:

[0012] The second generation PVT is used as a benchmark model, and then the average pooling and 1*1 convolution are used to add the feature information in the previous layer pyramid to the next layer pyramid, so as to add the feature information of the shallow layer to the deep layer feature and strengthen the global feature extraction capability.

[0013] The multi-scale fusion pyramid visual Transformer network is composed of a first input layer, an encoding layer, a decoding layer, a linear layer and a first feedback network, wherein:

[0014] The first input layer: as the RGB three channels of the magnetic controlled capsule endoscopy image, the image is expanded through a data enhancement algorithm, facilitating model training.

[0015] The encoding layer: used for encoding the input image of the input layer through repeatedly stacking the multi-head self-attention module of the Transformer eight times and the feedforward neural network;

[0016] The decoding layer: used for decoding the output of the encoding layer through repeatedly stacking the masked multi-head self-attention module of the Transformer eight times and the multi-head self-attention module, to obtain the feature vector extracted from the image;

[0017] The encoding layer and the decoding layer are stacked according to the number of 3, 4, 6 and 3 respectively, to obtain four layers of feature layers of the pyramid, the convolution operation is used to reduce the sampling between the feature layers, and the convolution and the pooling operation are used to add the input at the encoding layer of the previous layer feature layer and the next layer feature layer;

[0018] The linear layer: after completing the four times of the feature pyramid operation, the feature layer is normalized for probability prediction through the linear layer with the input category of 3, to obtain the probability of each category corresponding to each image, and the classification loss of the entire neural network is calculated through the cross-entropy loss function;

[0019] The first feedback network: the loss error is calculated using the cross-entropy loss function, then the gradient of each layer parameter in the feedback process is calculated through the loss error, and the parameters of each layer are adjusted, and the cross-entropy loss function is shown as formula (1):

[0020]

[0021] Wherein, n is the number of categories of classification; y i is the real label, taking the value of 0 and 1, that is, if the category is y i , then it is 1, otherwise it is 0; represents the probability that the prediction result is the corresponding category.

[0022] Since the differences between different anatomical parts of the stomach are small, relying on local pixel information cannot well extract the feature differences between the anatomical parts of the stomach. The second generation PVT is used as a benchmark model in the application, and the average pooling and 1x1 convolution are used to add the feature information in the previous layer pyramid to the next layer pyramid, to add the shallow feature information to the deep feature, to strengthen the global feature extraction capability of the model, and to construct a multi-scale fusion pyramid type visual Transformer network, to classify the magnetic controlled capsule endoscopy image of the stomach, to identify three large categories of the upper part of the stomach (the body and the bottom of the stomach), the lower part of the stomach (the angle of the stomach, the antrum of the stomach and the pylorus), and the small intestine, to realize efficient and accurate multi-classification.

[0023] The target detection network of the added attention module is obtained by adding a channel attention module in a YOLOv5 deep learning network backbone.

[0024] The target detection network of the added attention module is composed of a second input layer, a convolutional layer, a bottleneck layer and a second feedback network.

[0025] The second input layer is used to input the image itself, the labeled anchor box center point coordinates and the txt file of the length and width, and sequentially input the image and the corresponding txt file into the next layer; or used to input the image itself into the next layer.

[0026] The convolutional layer is used to perform two times of convolution including normalization and activation function, and then stacked with convolution including residual structure to prevent the problem of gradient disappearance, and after two times of convolution including residual structure, the channel attention module including horizontal and vertical axis information is used to strengthen the previously extracted features, and the above operation is repeated three times, and finally a fast spatial pyramid pooling operation is used to strengthen the multi-scale feature fusion, and then a channel attention module is used to strengthen the feature extraction again.

[0027] The bottleneck layer is used to perform an upsampling operation on the output of the convolutional layer, and then concatenate the output of the second last channel attention to the above output, and after a convolutional layer including residual structure, input the output into the upsampling process of the next layer.

[0028] After the upsampling process is completed, the output after the third last channel attention module is concatenated into the output after the upsampling process, and after a convolutional layer including residual structure, the first detection head is obtained to obtain the detection of large targets.

[0029] The above upsampling process is then down-sampled by convolution, and the input of the last upsampling is concatenated, and after a convolutional layer including residual structure, the detection head of medium targets is obtained.

[0030] Then, the output of the backbone network is concatenated to obtain the detection head of small targets.

[0031] The second feedback network includes three loss functions, including the prediction class, the class confidence and the prediction box position loss function based on the complete intersection over union CIoU, and the final loss function formula is shown in equations (2)-(6):

[0032] Loss=λ1L cls +λ2L obj +λ3L IoU #(2)

[0033]

[0034]

[0035]

[0036]

[0037] where λ i is a learnable weight, ρ 2 (b,b gt ) is the Euclidean distance between the center points of the real and predicted bounding boxes, b represents the real bounding box, b gt represents the predicted bounding box, c represents the diagonal distance of the minimum closed region that can contain both the predicted and real bounding boxes, w and h are the length and width of the predicted bounding box, w T and h T are the length and width of the real bounding box, v represents the difference between the length-width ratio of the real and predicted bounding boxes converted to an angle, and the purpose of α is to limit the value of v within the interval [0-1].

[0038] The application takes YOLOv5 as a benchmark target detection model, adds a CA module in the BackBone to enhance the feature extraction capability, so as to achieve the purpose of efficient and accurate gastric angle and pylorus detection and positioning.

[0039] Compared with the prior art, the application has the following advantages and beneficial effects:

[0040] (1) The application expands the data set, classifies the magnetically controlled capsule endoscopy images by using a classification network, and then locates the classified images by using a target detection model, thereby solving the problem of low resolution of magnetically controlled capsule endoscopy images in the prior art, which is not easy to identify, and accurately identifying the anatomical parts of the stomach, thereby further achieving the purpose of positioning the magnetically controlled capsule endoscopy in the stomach.

[0041] (2) The existing digestive tract endoscopy image anatomical structure identification algorithm mostly classifies according to three types of esophagus, stomach and intestine, and most of the algorithms are only for classifying traditional endoscopy images or traditional wireless capsule endoscopy images, and there is currently no method for identifying the anatomical parts of the stomach in magnetically controlled capsule endoscopy images. The application classifies the stomach region in the magnetically controlled capsule endoscopy image in detail, including the upper stomach (stomach body and fundus), the lower stomach (gastric antrum, pylorus and gastric angle), and further locates the positions of the gastric angle and the pylorus by using a target detection model.

[0042] (3) The application combines a multiscale fusion pyramid visual Transformer model and a target detection network with an attention module to classify and identify the anatomical parts of the stomach. Compared with the traditional feature extraction and classification identification method, the application can better extract global image features and has better classification and identification performance in the case of large data volume. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is a flowchart of the present application;

[0044] Figure 2 is a pyramid visual Transformer network structure diagram of multi-scale fusion;

[0045] Figure 3 is a target detection network structure diagram with an attention module added;

[0046] Figure 4 is a confusion matrix of the ConvNeXt model identifying the anatomical parts of the stomach;

[0047] Figure 5 is a confusion matrix of the PVT-V2-b2 model identifying the anatomical parts of the stomach;

[0048] Figure 6 is a confusion matrix of the ResNet-101 model identifying the anatomical parts of the stomach;

[0049] Figure 7 is a confusion matrix of the Swin-Transforemr-base model identifying the anatomical parts of the stomach;

[0050] Figure 8 is a confusion matrix of the VAN-Large model identifying the anatomical parts of the stomach;

[0051] Figure 9 is a confusion matrix of the present application identifying the anatomical parts of the stomach;

[0052] Figure 10 is an ROC curve of the present application identifying the upper part of the stomach;

[0053] Figure 11 is an ROC curve of the present application identifying the lower part of the stomach;

[0054] Figure 12 is an ROC curve of the present application identifying the small intestine;

[0055] Figure 13 is a PR curve of the present application detecting the gastric angle and the pylorus. DETAILED DESCRIPTION

[0056] The present application will be further described in detail below with reference to the embodiments, but the embodiments of the present application are not limited thereto.

[0057] Embodiment:

[0058] In conjunction with Figure 1 As shown in the figure, a magnetic control capsule endoscopy image stomach anatomical structure identification method based on deep learning includes:

[0059] Step 1, construction of capsule endoscopy image dataset: according to the annotation of experts, the effective image data is screened out, and the pictures are divided into fundus, body, antrum, pylorus, angle and small intestine according to anatomical parts. Because the texture color and other characteristics of the fundus and the body are similar, the two kinds of endoscopy images are merged together and named as the upper part of the stomach; the pylorus connects the antrum and the duodenum, and the angle is located at the connection between the antrum and the body, so the angle, the pylorus and the antrum in the endoscopy image basically exist at the same time, and therefore the images of the angle, the pylorus and the antrum are merged together and named as the lower part of the stomach.

[0060] Step 2, data set preprocessing: because the capsule endoscopy data set from the clinic is very limited, in order to avoid overfitting of the deep learning network and affect the detection accuracy, the affine transformation technology such as flipping and rotating is used to expand the capsule endoscopy data set to hundreds of thousands of images, and then the images of each category are divided into training set and test set according to the preset ratio (such as 8:2).

[0061] Step 3, construction of multi-scale fusion pyramid visual Transformer network: the second generation PVT is used as the benchmark model, and then the average pooling and 1x1 convolution are used to add the feature information in the upper pyramid to the lower pyramid, so that the feature information of the shallow layer is added to the deep layer, the global feature extraction ability of the model is strengthened, and a multi-scale fusion pyramid visual Transformer network is constructed, as shown in Figure 2 The multi-scale fusion pyramid visual Transformer network is composed of an input layer, an encoding layer, a decoding layer, a linear layer and a feedback network, and each component module is introduced as follows:

[0062] Input layer: it is the RGB three channels of the magnetically controlled capsule endoscopy image, which is expanded by data enhancement algorithm, facilitating model training.

[0063] Encoding layer: the input image of the input layer is encoded by repeatedly stacking the multi-head self-attention module of Transformer and the feedforward neural network eight times.

[0064] Decoding layer: the output of the encoding layer is decoded by repeatedly stacking the mask multi-head self-attention module of Transformer and the multi-head self-attention module eight times, to obtain the feature vector extracted from the image.

[0065] The encoding layer and the decoding layer are stacked according to the number of 3, 4, 6 and 3 respectively, to obtain four layers of feature layers of the pyramid. Convolution operation is used between each feature layer to reduce sampling, so as to compress the feature vector. Convolution and pooling operation are also used to add the input at the encoding layer of the next feature layer to the input of the next feature layer, so as to achieve multi-scale fusion.

[0066] Linear layer: After completing four feature pyramid operations, the feature layer is normalized by a linear layer with an input category of 3 to make probability predictions, obtaining the probability of each category corresponding to each image. The classification loss of the entire neural network is calculated by the cross-entropy loss function.

[0067] Feedback network: The loss error is calculated using the cross-entropy loss function, and then the gradient of each layer parameter in the feedback process is calculated by the loss error, and the parameters of each layer are adjusted.

[0068] The cross-entropy loss function is shown in equation (1):

[0069]

[0070] Where n is the number of categories of classification; y i is the true label, taking values of 0 and 1, i.e. if the category is y i , then 1, otherwise 0; represents the probability that the prediction result is the corresponding category.

[0071] All the training data in step 2 are input into the multi-scale fusion pyramid visual Transformer network, and the network is stopped when the performance on the test set is optimal after forward propagation and parameter update. The weight parameter data with the best classification effect is obtained to achieve efficient and accurate multi-classification.

[0072] Step 4, target detection network construction with attention module: add a channel attention module (Channel Attention, CA module) to the YOLOv5 deep learning network backbone to enhance its feature extraction capability, so as to achieve the purpose of efficient and accurate detection and positioning of the gastric angle and pylorus. The images in the lower part of the stomach image labeled with the gastric angle or pylorus are input into the target detection network for training and verification. Similarly, the parameter weight information is updated through the forward propagation and backward propagation process of the network to obtain the optimal positioning result.

[0073] As shown in Figure 3 , the target detection network is composed of an input layer, a convolutional layer, a bottleneck layer and a feedback network, and each component module is introduced as follows:

[0074] Input layer: When training and testing, the input data includes the image itself, the labeled anchor box center point coordinates and the length-width txt file. Through one-to-one correspondence, the image and the corresponding txt file are sequentially input into the network. When predicting, only image data needs to be input.

[0075] Convolutional layers: After two convolutions containing normalization and activation functions, a convolutional layer containing residual structures is used to prevent the gradient vanishing problem. Then, after two convolutional layers containing residual structures, a channel attention module containing information about the horizontal and vertical axes is used to enhance the previously extracted features. The above operation is repeated three times. Finally, a fast spatial pyramid pooling operation is used to enhance multi-scale feature fusion, followed by a channel attention module to further enhance feature extraction.

[0076] Bottleneck Layer: First, the final output of the convolutional layer is used as input for an upsampling operation. Then, the output of the penultimate channel attention module is concatenated to the above output, passed through a convolutional layer containing residual structures, and input into the next layer's upsampling process. Similarly, after the upsampling process, the output after the penultimate channel attention module is concatenated to the output after the upsampling process, and passed through a convolutional layer containing residual structures to obtain the first detector head for detecting large targets. Next, the above upsampling process is downsampled through convolution, concatenated with the input of the previous upsampling layer, and passed through a convolutional layer containing residual structures to obtain a detector head for medium-sized targets. Similarly, the output of the backbone network is concatenated to obtain a detector head for small targets.

[0077] Feedback network: Includes three loss functions, including predicted category, category confidence, and predicted bounding box location loss function based on Complete Intersection over Union (CIoU). The final loss function formulas are shown in equations (2)-(6):

[0078] Loss=λ1L cls +λ2L obj +λ3L IoU #(2)

[0079]

[0080]

[0081]

[0082]

[0083] Where, λ i For learnable weights, ρ 2 (b,b gt ) represents the Euclidean distance between the center points of the ground truth bounding box and the predicted bounding box, and b represents the ground truth bounding box. gt Let represent the predicted bounding box, and 'c' represent the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes. 'w' and 'h' represent the length and width of the predicted bounding box, respectively. T and hT The length and width of the real boundary box, v represents the difference between the aspect ratio of the real target box and the predicted box rectangle converted to angle, and the purpose of a is to limit the value of v in the interval [0-1].

[0084] Step 5, training of the deep learning network:

[0085] The classification model training process is as follows: the learning rate is set to 1x10 -4 , the training period is set to 100, and the above training set is input into the classification model in groups of 256 for training. The size of the input image is 480x480, the optimizer of the gradient descent algorithm is set to the ADAM function, and the loss function is the cross-entropy loss function. In order to ensure that the patient data of the training set and the validation set are different, and at the same time ensure that the proportion of the training set and the validation set is close to 8:2, the present application uses a total of 63168 images of the upper stomach, lower stomach and small intestine as the training set, and the validation set is a total of 15792 images of the upper stomach, lower stomach and small intestine.

[0086] The target detection network training process is as follows: the images with gastric angle or pylorus in the lower stomach and the corresponding txt file with annotation information are input into the target detection network. Similarly, in order to ensure that the patient data of the training set and the validation set are different, and at the same time ensure that the proportion of the training set and the validation set is close to 8:2, the present application contains 8409 images in the training set, and contains 1210 images in the test validation set. The learning rate is set to 1x10 -4 , the input training image adopts mosaic technology, affine transformation and other image enhancement algorithms to strengthen the detection ability of the target detection network. The training period is set to 100, the image is input into the network in groups of 64, the size of the input image is set to 480x480, the optimizer of the gradient descent algorithm is set to the ADAM function, and the loss function is three parts, including the predicted category, the category confidence and the predicted box position loss function based on the complete intersection over union (CIoU).

[0087] Step 6: performance evaluation of the method. The training and test performance is evaluated by five internationally accepted indicators, including precision, recall, F1 score, accuracy and mean average precision (mAP).

[0088] The experimental verification is as follows:

[0089] 1. Experimental data

[0090] The collected capsule endoscopy image data were augmented to 127014. Among them, 58103 were upper stomach images, 25335 were lower stomach images, and a total of 43340 were small intestine images. A total of 9619 images with gastric angle and pylorus were screened out in the lower stomach images.

[0091] Training set of classification network: In order to ensure the proportion of training set and test set, and the training set and the validation set should not have images of the same patient, the present application screened out 29244, 10024 and 23900 images of upper stomach, lower stomach and small intestine image data as training set respectively; test validation set: upper stomach, lower stomach and small intestine image data are 7311, 2506 and 5975 respectively. The training and test validation set of the target detection network are 8409 and 1210 images respectively.

[0092] 2. Verify the performance of the gastric anatomical site classification network and the gastric angle / pylorus target detection network model using the above magnetic control capsule gastroscope image.

[0093] 3. Experimental results

[0094] (1) Classification network performance

[0095] Table 1 shows the test results of the multi-scale fusion pyramid vision Transformer network model proposed by the present application and other related deep learning models in classifying and identifying gastric anatomical sites such as upper stomach, lower stomach and small intestine. It can be seen that compared with the other five models, the accuracy, precision, recall and F1 score of the classification network of the present application are the highest. The parameter amount is the same as that of the original PVT-v2-b2 model, which is much less than that of the other four models.

[0096] Table 1 Classification and identification performance of gastric anatomical sites of the present application and comparative models

[0097]

[0098] Figure 9 、 Figures 4-8 The confusion matrices of the present application and the comparative model in classifying and identifying gastric anatomical sites such as upper stomach, lower stomach and small intestine are shown in Tables 2, 3 and 4 respectively. Figures 4-9It can be seen that the ConvNeXt model has the highest accuracy in judging the small intestine, but the lowest accuracy in identifying the lower stomach. The PVTv2 model has the highest accuracy in identifying the lower stomach, but the lower accuracy in identifying the small intestine. The multi-scale fusion pyramid visual Transformer network model proposed in the present application has an identification accuracy of the small intestine only 1% lower than that of the ConvNeXt model, and an identification accuracy of the lower stomach only 3% lower than that of the PVTv2 model, which is acceptable considering the small amount of test data of the lower stomach. Although the SwinTransformer-base model has high accuracy in identifying the upper stomach, it has low accuracy in identifying the small intestine. Therefore, the multi-scale fusion pyramid visual Transformer network model proposed in the present application has the best overall performance in identifying the anatomical parts of the stomach, and the results are the most stable. At the same time, the multi-scale fusion pyramid visual Transformer network model proposed in the present application has the least number of parameters, indicating that it occupies less system resources and has high efficiency in completing the identification task. Figures 10-12 The ROC curve and the area under the curve AUC of the present application for identifying the upper stomach, the lower stomach and the small intestine prove the reliability of the method of the present application.

[0099] (2) Detection algorithm performance

[0100] The performance of the target detection network with the added attention module in detecting the gastric angle and pylorus is shown in Table 2. As can be seen from Table 2, among the present application and the other seven comparison methods, although the accuracy and recall rate of the method of the present application are not the highest, they are only in the middle, but the mAP, i.e. the area under the PR curve, of the method of the present application is the largest. At the same time, compared with the ordinary target detection network, the method of the present application has less increase in the number of parameters and runs faster than other improved algorithms. In the case of using GPU, the inference time required for each picture is only 0.1 ms, which can quickly process a large amount of picture data in practical applications. Figure 13 is the PR curve of the method of the present application for detecting the gastric angle and pylorus, and the area under the PR curve reaches 0.921, indicating that the gastric angle and pylorus detection effect of the method of the present application is very good.

[0101] Table 2 Comparison of gastric angle and pylorus detection algorithm performance of the present application and other methods

[0102]

[0103]

[0104] The experimental results show that the gastric anatomical part identification method of the magnetic control capsule endoscope image proposed in the present application has the advantages of fast speed and high accuracy, and has good application prospects in large-scale gastric lesion screening of the magnetic control capsule endoscope.

[0105] Although the present application has been described with reference to the explanatory embodiments thereof, the above embodiments are merely the preferred embodiments of the present application, and the embodiments of the present application are not limited to the above embodiments, and it should be understood that many other modifications and embodiments can be devised by those skilled in the art, and such modifications and embodiments will fall within the scope and spirit of the principles disclosed in the present application.

Claims

1. A method for identifying gastric anatomical structures from images obtained using a magnetically controlled capsule endoscopy based on deep learning, characterized in that... include: Step S100: Filter the images output by the magnetically controlled capsule endoscopy, classify the images according to anatomical location and use them as the capsule endoscopy image dataset, use affine transformation technology to amplify the capsule endoscopy image dataset, and divide the images of each category in the dataset into training set and test set according to a preset ratio. Step S200: Construct a multi-scale fusion pyramid visual Transformer network and an object detection network with an added attention module, respectively. Train the multi-scale fusion pyramid visual Transformer network and the object detection network with an added attention module using the training set, respectively. Test the trained multi-scale fusion pyramid visual Transformer network and the object detection network with an added attention module using the test set, respectively. Step S300: After the test is passed, the global features of the magnetically controlled capsule endoscopy images of the stomach are extracted using a multi-scale fusion pyramid visual Transformer network, and the magnetically controlled capsule endoscopy images are classified. The classified images are then located using a target detection network with an added attention module to identify the anatomical structures of the stomach. The construction method of the multi-scale fusion pyramid visual Transformer network is as follows: The second-generation PVT is used as the baseline model. Then, average pooling and 1×1 convolution are used to add the feature information in the upper pyramid to the lower pyramid, and the feature information in the shallow layer is added to the deep feature, thereby enhancing the global feature extraction capability. The multi-scale fusion pyramid visual Transformer network consists of a first input layer, an encoding layer, a decoding layer, a linear layer, and a first feedback network, wherein: First input layer: As the RGB three channels of the magnetically controlled capsule endoscopy image, the image is augmented through data augmentation algorithm to facilitate model training; Encoding layer: Used to encode the input image of the input layer by repeatedly stacking a Transformer multi-head self-attention module eight times and a feedforward neural network; Decoding layer: Used to decode the output of the encoding layer by repeatedly stacking the Transformer masked multi-head self-attention module eight times and the multi-head self-attention module to obtain the feature vector extracted from the image; The encoding and decoding layers are stacked in increments of 3, 4, 6, and 3 respectively to obtain four feature layers of the pyramid. Convolution operations are used to downsample between each feature layer, and convolution and pooling operations are used to directly add the input of the previous feature layer to the input of the encoding layer of the next feature layer. Linear layer: After completing four feature pyramid operations, the feature layer is normalized and probability predicted by a linear layer with input category 3 to obtain the probability of each category corresponding to each image, and the classification loss of the entire neural network is calculated by the cross-entropy loss function. First feedback network: The loss error is calculated using the cross-entropy loss function. Then, the gradient of the parameters of each layer during the feedback process is calculated using the loss error, and the parameters of each layer are adjusted accordingly. The target detection network with added attention modules is obtained by adding channel attention modules to the backbone of the YOLOv5 deep learning network to enhance its feature extraction capabilities and achieve the purpose of detecting and locating the gastric angle and pylorus. Images of the lower part of the stomach marked with gastric angle or pylorus are input into the target detection network for training and verification. The parameter weight information is updated through the forward and backward propagation process of the network to obtain the optimal localization result. The object detection network with added attention module consists of a second input layer, a convolutional layer, a bottleneck layer, and a second feedback network, wherein: The second input layer is used to send the image itself, the coordinates of the center point of the labeled anchor box, and the length and width of the txt file to the next layer in sequence; or it is used to input the image itself into the next layer. Convolutional layers: After performing two convolutions containing normalization and activation functions, convolutions containing residual structures are stacked to prevent the gradient vanishing problem. After the two convolutional layers containing residual structures, a channel attention module containing information of the horizontal and vertical axes is used to enhance the previously extracted features. The above operation is repeated three times. Finally, a fast spatial pyramid pooling operation is used to enhance multi-scale feature fusion. Then, a channel attention module is used again to enhance feature extraction. Bottleneck layer: It is used to perform an upsampling operation on the last output of the convolutional layer as input, and then concatenate the output of the penultimate channel attention to the above output. After passing through a convolutional layer containing residual structures, it is input into the upsampling process of the next layer. After the upsampling process is completed, the output after the third-to-last channel attention module is concatenated into the output after the upsampling process. After passing through a convolutional layer containing residual structures, the first detection head can be obtained to detect large targets. The above upsampling process is then passed through convolutional downsampling, and the input from the previous upsampling layer is concatenated with it. After passing through a convolutional layer containing residual structures, a detection head for medium-sized targets is obtained. Then, the output of the backbone network is spliced ​​together to obtain the detection head for small targets; The second feedback network contains three loss functions: predicted category, category confidence, and predicted bounding box location loss function based on complete intersection-union ratio (CIoU).

2. The method for identifying gastric anatomical structures from magnetically controlled capsule endoscopy images based on deep learning according to claim 1, characterized in that, The cross-entropy loss function is shown in equation (1): ； Where n is the number of categories; y i These are the actual labels, with values ​​of 0 and 1; that is, if the category is y... i If it is 1, then it is 1; otherwise it is 0. This represents the probability of the predicted result for the corresponding category.

3. The method for identifying gastric anatomical structures from magnetically controlled capsule endoscopy images based on deep learning according to claim 1, characterized in that, The final loss function formulas are shown in equations (2)-(6): ； ； ； ； ； Where, λ i For learnable weights, Let b be the Euclidean distance between the center points of the ground truth bounding box and the predicted bounding box, and b represent the ground truth bounding box. gt Let c represent the predicted bounding box, and 'c' represent the diagonal distance of the smallest closure region that can simultaneously contain both the predicted and ground truth bounding boxes. Let w and h be the length and width of the predicted bounding box, respectively. T and h T α represents the length and width of the ground truth bounding box, v represents the difference between the aspect ratio of the ground truth bounding box and the predicted bounding box rectangle converted into angle, and α is intended to limit the value of v to the range of [0-1].

Images