A method for automatic recognition of gastrointestinal organ images using capsule endoscopy

Abstract

A method for automatically recognizing gastrointestinal organ images by capsule endoscopy includes the following steps: 1) standardizing and marking a sequence of gastrointestinal images collected by capsule endoscopy; 2) extracting key frames by using hash similarity; 3) using a four-classification model combining multi-head attention with a separable convolutional architecture to determine the moment when the capsule endoscope enters the esophagus and cardia; 4) using a multi-head attention deep learning two-classification network that integrates edge information to determine the moment when the capsule endoscope enters the small intestine and large intestine. The present invention provides a method for automatically recognizing gastrointestinal organ images by capsule endoscopy, which improves the recognition accuracy of gastrointestinal segmentation in capsule endoscopy images, assists in reducing the reading time of clinicians, and assists in improving diagnostic efficiency.

Description

Technical Field

[0001] The invention belongs to the field of image processing, and in particular relates to a method for automatically recognizing gastrointestinal organ images using a capsule endoscope. Background Art

[0002] Wireless capsule endoscopy is a new gastrointestinal examination technology developed in the early 21st century. It overcomes the limitations of traditional endoscopy in the small intestine and is convenient, painless and safe. The examinee swallows a capsule endoscope. With the action of gravity and gastrointestinal peristalsis, the capsule endoscope captures a color image of the entire gastrointestinal tract. The video is taken in chronological order through the body, esophagus, stomach, small intestine and large intestine. When the doctor views the capsule endoscopy video, he first finds the connecting part between the organs, such as the cardia at the junction of the esophagus and stomach, and the pylorus at the junction of the stomach and small intestine. The moment when the capsule endoscope enters different organs is further determined. Finally, different viewing methods are adopted for each organ, and a diagnosis is made after the lesion is observed, combined with the organ location of the lesion. Since the capsule endoscopy takes about 8 to 13 hours to shoot, and the examinee's condition is complex and diverse, it takes a long time for the doctor to manually determine the moment when the capsule endoscope enters different organs.

[0003] In recent years, deep learning has made great breakthroughs in the field of image recognition. There are also gastrointestinal segmentation recognition methods based on deep learning. Usually, deep learning methods are used to directly classify various organs (esophagus, stomach, small intestine, and large intestine), but the accuracy of direct classification is not high. Therefore, in view of the clinical needs of capsule endoscopy film reading, designing accurate automatic recognition of gastrointestinal organs by capsule endoscopy is a difficult problem that needs to be solved urgently. Summary of the invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a method for automatically recognizing gastrointestinal organ images using capsule endoscopy, which improves the recognition accuracy of gastrointestinal segments in capsule endoscopy images, helps reduce the time for clinicians to read images, and helps improve diagnostic efficiency.

[0005] The technical solution adopted by the present invention to solve its technical problem is:

[0006] A method for automatically recognizing gastrointestinal organ images by capsule endoscopy comprises the following steps:

[0007] 1) Standardize the labeling of gastrointestinal image sequences acquired by capsule endoscopy;

[0008] 2) Extract key frames using hash similarity;

[0009] 3) A four-classification model combining multi-head attention and separable convolutional architecture is used to determine the moment when the capsule endoscope enters the esophagus and cardia;

[0010] 4) A multi-head attention deep learning binary classification network that integrates edge information is used to determine the moment when the capsule endoscope enters the small intestine and large intestine.

[0011] Furthermore, in the step 1), the time information of each frame of the image is calculated according to the frame rate value of the video captured by the capsule endoscope, and each frame of the image is named in the format of "time_frame value", and the position of the image in the video is quickly located by the sample standard labeling method.

[0012] Further, the process of step 2) is as follows:

[0013] 2.1) Key frame initialization: the first frame of the collected capsule endoscopy image sequence is set as the current key frame;

[0014] 2.2) Calculate the similarity between the key frame and the next frame using the difference hash algorithm, and set the threshold T1; if the similarity is less than the threshold T1, the next frame is set as the current key frame; otherwise, the next frame is judged to be a redundant image and deleted from the image sequence, and the current key frame remains unchanged;

[0015] 2.3) Determine whether the image reading is completed. If it is completed, terminate the comparison, otherwise repeat 2.2);

[0016] 2.4) The key frame set is the preprocessed image.

[0017] Furthermore, the process of step 3) is as follows:

[0018] 3.1) Train a four-classification network for in vitro, cardia, pylorus and other capsule endoscopy images, perform data augmentation on the training images, including flipping and rotating the images horizontally and vertically, and then enter the multi-head self-attention module to enhance the feature expression of key areas, generate weight masks for each position and weighted output, thereby enhancing the specific target area of ​​interest while weakening the irrelevant background area; then use depthwise separable convolution, including channel-by-channel convolution and point-by-point convolution, to reduce parameter operations and computational costs;

[0019] 3.2) In the capsule endoscopy image sequence, the capsule passes through the external body and the cardia in sequence, and it takes a short time to reach the cardia. The threshold T2 is set to classify the external body and cardia images before the T2 frame to improve the classification accuracy;

[0020] 3.3) In the classification results, the last frame of the external image is the starting position of the organ esophagus. Combined with the classification results of the cardia image, the moment when the capsule endoscope enters the esophagus and cardia can be determined;

[0021] 3.4) For the multiple pylorus images obtained through classification, the last one is taken as the junction point between the stomach and small intestine.

[0022] The process of step 4) is as follows:

[0023] 4.1) There are significant differences between the small and large intestines. In the network structure, the edge information of the image is integrated, and the network's ability to recognize the villi of the small intestine is enhanced through the extraction of multi-head attention.

[0024] 4.2) Classify the images after the pylorus, the junction of the stomach and small intestine in the capsule endoscopy image sequence, into large intestine and small intestine, set a threshold T3, and stop classification when the number of consecutively classified large intestine images is greater than T3; the first frame of the consecutively classified large intestine images is the dividing point between the large and small intestine organs. Combined with the determination of the pylorus in step 3), the moment when the capsule endoscope enters the small intestine and large intestine can be determined.

[0025] The beneficial effects of the present invention are mainly manifested in: improving the recognition accuracy of gastrointestinal tract segments in capsule endoscopy images, assisting in reducing the image reading time of clinical doctors, and assisting in improving diagnosis efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of the gastrointestinal image sequence taken by capsule endoscopy.

[0027] Figure 2 The present invention is a flow chart of a method for automatic recognition of gastrointestinal organs in capsule endoscopy images.

[0028] Figure 3 It is the flowchart of capsule endoscopy key frame image extraction.

[0029] Figure 4 It is a schematic diagram of the deep learning four-classification network structure. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings.

[0031] Reference Figure 1 to Figure 4 , a method for automatically recognizing gastrointestinal organs in capsule endoscopy images, comprising the following steps:

[0032] 1) Standardize the labeling of gastrointestinal image sequences collected by capsule endoscopy

[0033] like Figure 2 The figure shows a sequence of gastrointestinal organ images taken by capsule endoscopy. It is understandable that the examiner swallows a "capsule" with a camera, and through gravity and gastrointestinal peristalsis, the entire gastrointestinal tract of the human body is automatically photographed, passing through the body, esophagus, stomach, small intestine, and large intestine in sequence. The shooting rate of capsule endoscopy is generally 5fps, the running time in the body is about 8 to 13 hours, and the number of images taken is up to tens of thousands. When converting video to image processing, the shooting time of each frame of endoscopic image is calculated, and the image is named in the format of "time_frame value", which is convenient for doctors to quickly locate the position of the image in the video.

[0034] Shooting time = frame value / fps

[0035] 2) Extract key frames using hash similarity;

[0036] The purpose of key frame extraction is to remove a large number of similar capsule endoscopy images and reduce the burden of network classification, such as Figure 3 As shown, the specific steps are as follows:

[0037] 2.1) Initialize the key frame, set the first frame of the collected capsule endoscopy image sequence as the current key frame;

[0038] 2.2) Use the difference hash algorithm to calculate the similarity between the key frame and the next frame image, set the threshold T1, if the similarity value is less than the threshold T1, the next frame image is set as the current key frame, otherwise it is judged as a redundant image and deleted from the image sequence, and the current key frame remains unchanged. The difference hash algorithm first reduces the image to 9x8 size, a total of 72 pixels, discarding the image differences caused by different sizes and proportions; then convert the scaled image into a 256-level grayscale image; then subtract the two adjacent elements in each row of the grayscale matrix (left element minus right element) to obtain 8 different difference values, a total of 64 difference values, and then process the difference value. If it is a positive number or 0, remember it as 1; if it is a negative number, remember it as 0; finally, combine the 64 results together to get a hash value to represent an image, and calculate the distance of the hash value to determine the similarity of the two images. Generally, the Hamming distance is used, that is, the hash values ​​of the two images are calculated bit by bit to see if they are the same;

[0039] 2.3) Determine whether the image reading is completed. If it is completed, terminate the comparison, otherwise repeat 2.2);

[0040] 2.4) The key frame set is the preprocessed image.

[0041] 3) A four-classification model combining multi-head attention and separable convolutional architecture is used to determine the moment when the capsule endoscope enters the esophagus and cardia;

[0042] The in vitro, cardiac and pyloric capsule endoscopy images all have significant features: the in vitro images mainly contain the facial information of the examiner, which is significantly different from the gastrointestinal images; the cardiac images are characterized by the surrounding esophageal walls contracting toward the center, the periphery of the image is pink, and the center is white; when no object passes through, the pyloric image shows that the stomach wall contracts toward the center, and when an object passes through, the stomach wall opens and the center of the image appears as a round hole.

[0043] According to the differences of in vitro, cardia, pylorus and other capsule endoscopy images, the present invention trains a four-classification network structure diagram, see Figure 4. The training set contains 4 categories, one category each for in vitro, cardia, and pylorus capsule endoscopy images, and one category for the remaining capsule endoscopy images. First, the data enhancement method is used for the capsule endoscopy training set to improve the classification accuracy and generalization ability of the model, including flipping and rotating the image in the horizontal and vertical directions. Then, two layers of 3x3 convolution are performed on the training enhanced image, and the feature map after convolution is sent to the multi-head self-attention module (Multi-Head Self-Attention, MHSA). The feature map obtained by MHSA is subjected to the inverted residual bottleneck (Inverted Residual Bottleneck, IRB) operation. The 3x3 convolution in IRB is a depth-separable convolution, and the activation function is SiLU, ×L i It means that the MHSA+IRB module is repeated i times. Finally, the classification result is obtained through global average pooling and fully connected layer.

[0044] The classification effect indicators are accuracy, precision, and recall. Accuracy is the percentage of correctly classified results in the total samples; precision refers to the classification results, which means the probability of actually being a positive sample among all samples classified as positive; recall refers to the original samples, which means the probability of being classified as a positive sample among the samples that are actually positive;

[0045] During the classification process, considering that the capsule endoscope image sequence passes through the external body and the cardia in sequence, and it takes a short time to reach the cardia, we set the threshold T2 to classify the external body and cardia images before the T2th frame to improve the classification accuracy. In clinical practice, the operation of capsule endoscopes in different examiners varies greatly. After entering the stomach, the capsule endoscope may pass through the pylorus and enter the small intestine at one time; it may also be affected by gastric peristalsis, and the capsule endoscope may take long and repeated photos near the pylorus, and the number of pylorus images taken is large and discontinuous. For the latter, multiple pylorus images are obtained through classification. Considering the practical significance of the last pylorus image in clinical practice, we only focus on the last pylorus classification image.

[0046] In the classification results, the last in vitro image is judged as the starting position of the esophagus. Combined with the information of the cardia classification, the moment when the capsule endoscope enters the esophagus and cardia is determined.

[0047] 4) A multi-head attention deep learning binary classification network that integrates edge information is used to determine the moment when the capsule endoscope enters the small intestine and large intestine.

[0048] The capsule endoscopy process passes through the body, esophagus, stomach, small intestine, and large intestine in sequence. In step three, the location of the pylorus is determined. This step classifies the images after the pylorus in the capsule endoscopy image sequence (including the small intestine and large intestine). The binary classification network is trained based on the significant differences between the small intestine and the large intestine, such as: there are a lot of villi on the surface of the small intestine wall; the surface of the large intestine has no villi and appears smooth. The network contains two branches. The first branch is directly inserted into the training image data after enhancement. Figure 4 Network structure, get features Figure 1 Considering that the small intestine endoscopy image has rich edge information than the large intestine image, in branch 2, the edge detection is performed after the training image data is enhanced, and the obtained edge image is sent to Figure 4 In the network, we get the features Figure 2 ; Joint features Figure 1 , 2 , get the features Figure 3 , and finally the classification result is obtained through global average pooling and fully connected layers.

[0049] During the classification process, in order to reduce the classification pressure of the system and improve the classification accuracy, a threshold T3 is set. When the number of consecutively classified large intestine images is greater than T3, the classification is stopped. The first frame of the continuous image classified as large intestine is the dividing point between the large intestine and the small intestine, which can determine the moment when the capsule endoscope enters the small intestine and the large intestine.

[0050] In order to implement the method of this embodiment, a capsule endoscope gastrointestinal organ image automatic recognition device is designed, comprising:

[0051] Memory for storing computer programs;

[0052] The processor is used to implement the steps of the method for automatic recognition of gastrointestinal organs by capsule endoscopy when executing the computer program.

[0053] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable memory (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions for enabling an intelligent capsule endoscope image gastrointestinal organ image automatic recognition device (which can be a mobile phone, a computer, a capsule endoscope image gastrointestinal organ image automatic recognition device, etc.) to execute the methods described in each embodiment of the present invention.

[0054] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for automatic recognition of gastrointestinal organ images by capsule endoscopy, characterized in that: The method comprises the following steps: 1) Standardize the labeling of gastrointestinal image sequences acquired by capsule endoscopy; 2) Extract key frames using hash similarity; 3) A four-classification model combining multi-head attention and separable convolutional architecture is used to determine the moment when the capsule endoscope enters the esophagus and cardia; 4) A multi-head attention deep learning binary classification network that integrates edge information is used to determine the moment when the capsule endoscope enters the small intestine and large intestine; In the step 1), the time information of each frame of the image is calculated according to the frame rate value of the video shot by the capsule endoscope, each frame of the image is named in the format of "time_frame value", and the position of the image in the video is quickly located by a sample standard marking method; The process of step 2) is as follows: 2.1) Key frame initialization: the first frame of the collected capsule endoscopy image sequence is set as the current key frame; 2.2) Calculate the similarity between the key frame and the next frame using the difference hash algorithm, and set the threshold T1; if the similarity is less than the threshold T1, the next frame is set as the current key frame; otherwise, the next frame is judged to be a redundant image and deleted from the image sequence, and the current key frame remains unchanged; 2.3) Determine whether the image reading is completed. If it is completed, terminate the comparison, otherwise repeat 2.2); 2.4) The key frame set is the pre-processed image; The process of step 3) is as follows: 3.1) Train a four-classification network for in vitro, cardia, pylorus, and other capsule endoscopy images, perform data augmentation on the training images, including flipping and rotating the images horizontally and vertically, and then enter the multi-head self-attention module to enhance the feature expression of key areas, generate weight masks for each position, and weight the output, thereby enhancing the specific target area of ​​interest while weakening the irrelevant background area; Then, depth-wise separable convolution is used, which includes channel-by-channel convolution and point-by-point convolution, to reduce parameter calculation and computational cost. 3.2) In the capsule endoscopy image sequence, the capsule passes through the external body and the cardia in sequence, and it takes a short time to reach the cardia. The threshold T2 is set to classify the external body and cardia images before the T2 frame to improve the classification accuracy; 3.3) In the classification results, the last frame of the external image is the starting position of the organ esophagus. Combined with the classification results of the cardia image, the moment when the capsule endoscope enters the esophagus and cardia can be determined; 3.4) For the multiple pyloric images obtained by classification, the last one is taken as the junction point of the stomach and small intestine; The process of step 4) is as follows: 4.1) There are significant differences between the small and large intestines. In the network structure, the edge information of the image is integrated, and the network's ability to recognize the villi of the small intestine is enhanced through the extraction of multi-head attention. 4.2) Classify the images after the pylorus, the junction of the stomach and small intestine in the capsule endoscopy image sequence, into large intestine and small intestine, set a threshold T3, and stop classification when the number of consecutively classified large intestine images is greater than T3; the first frame of the consecutively classified large intestine images is the dividing point between the large and small intestine organs. Combined with the determination of the pylorus in step 3), the moment when the capsule endoscope enters the small intestine and large intestine can be determined.

Images