Colonoscopy image evaluation method and system based on EfficientNet structure
Through the colonoscopic image evaluation method based on EfficientNet structure, the MBConv convolution block and SE layer were used, combined with the BBPS evaluation standard, the accuracy and efficiency of intestinal preparation evaluation were solved, and automatic and accurate intestinal preparation evaluation was achieved to reduce the The burden of the doctor.
Patent Information
- Application Number
- CN202111339539.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-12
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-11-12
AI Technical Summary
The prior art is difficult to accurately and objectively evaluate the quality of intestinal preparation before colonoscopy, resulting in a high rate of intestinal preparation and increasing the burden on doctors.
Using colonoscopic image evaluation method based on EfficientNet structure, colorectal sequence images are acquired, preprocessed and classified, and MBConv convolution blocks and SE layers are used, combined with BBPS intestinal preparation evaluation criteria to automatically evaluate intestinal cleanliness and segment them. Evaluate whether the intestinal preparation is sufficient.
It realizes automatic and accurate evaluation of intestinal preparation, significantly reduces the burden on doctors, and improves the efficiency and accuracy of intestinal preparation evaluation.
Smart Images

Figure CN114119511B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image pattern recognition and classification, and in particular to a colonoscopy image evaluation method and system based on an EfficientNet structure. Background Art
[0002] Before the colonoscopy, the cleaning process of removing feces and digestive residual fluids from the colorectal canal and emptying foreign matter in the intestine is called bowel preparation. Bowel preparation is an important prerequisite for a successful colonoscopy. Currently, the intestine is usually cleaned by stopping eating in advance before colonoscopy and taking oral bowel cleansers. However, relevant domestic literature reports that the rate of inadequate bowel preparation in outpatients is 12%-25%, and the rate of inadequate bowel preparation in elderly patients is even higher, at 17%-32%. Foreign literature also reports that the incidence of inadequate bowel preparation in patients undergoing colonoscopy is 20%-25%. Therefore, it is very important to accurately and objectively evaluate the quality of bowel preparation. Summary of the invention
[0003] In view of this, the purpose of the present invention is to provide a colonoscopy image evaluation method and system based on the EfficientNet structure, which can effectively realize automatic and accurate evaluation of intestinal preparation and greatly reduce the burden on doctors.
[0004] To achieve the above object, the present invention adopts the following technical solution:
[0005] A colonoscopy image evaluation method based on the EfficientNet structure includes the following steps:
[0006] Step S1: acquiring colorectal sequence images and preliminarily classifying them into three sub-sequence images of the left colon, transverse colon, and right colon;
[0007] Step S2: preprocessing the subsequence images;
[0008] Step S3: obtaining the optimal classification based on the preprocessed subsequence images and the scalable EfficientNet network model;
[0009] Step S4: Based on the BBPS intestinal preparation evaluation standard, the three sub-sequence images of the left colon, transverse colon, and right colon are scored separately, and the scores of the images with the worst cleanliness of each segment of the intestine are accumulated to further evaluate whether the colonoscopy image is qualified.
[0010] Furthermore, the preprocessing includes: removing unqualified images and resetting the resolution to conform to the network.
[0011] Furthermore, the EfficientNet network model includes several MBConv convolution blocks, and its expansion method is used to solve the composite coefficients of three dimensions, as follows: Let the network depth d, network width w, and image resolution r coefficients be controlled by the parameters To measure, as shown in formula 1, its constraints are as shown in formula 2
[0012] (Formula 1)
[0013] (Formula 2).
[0014] Furthermore, the MBConv convolution block is as follows: the ReLU activation function caches the Swish activation function; the drop_connect method is used to replace the traditional drop method; BN is placed between the activation function and the convolution layer; a structure similar to the residual link is used, and the SE layer is used in the short connection part; the SE module takes into account the interdependence between the model channels, and includes two parts: Squeeze and Excitation.
[0015] A colonoscopy image evaluation system based on the EfficientNet structure, comprising a manual interaction unit, a data classification unit and an evaluation unit;
[0016] The manual interaction unit separates the colorectal sequence images into three sub-sequence images of the left colon, the transverse colon, and the right colon by manually identifying the hepatic flexure image and the splenic flexure image, and removes images that are obviously unqualified;
[0017] The data classification unit uses EfficientNet as the basic network, and can manually scale the network model according to the computing power configuration to obtain the best classification effect;
[0018] The evaluation unit is based on the BBPS intestinal evaluation standard, that is, scoring the three sub-sequence images of the left colon, transverse colon, and right colon respectively, and accumulating the image scores of the worst intestinal cleanliness of each segment, so as to evaluate the adequacy of the intestinal preparation.
[0019] Compared with the prior art, the present invention has the following beneficial effects:
[0020] The present invention realizes automatic and accurate evaluation of intestinal preparation, which can greatly reduce the burden on doctors. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a schematic flow chart of the method of the present invention;
[0022] Figure 2 is a schematic diagram of file name marking of liver flexure and spleen flexure images in one embodiment of the present invention;
[0023] Figure 3 This is a schematic diagram of the interface of the BBPS intestinal preparation automatic evaluation system in one embodiment of the present invention;
[0024] Figure 4 It is a schematic diagram of the content outputted to the colonoscopy report by the system in one embodiment of the present invention;
[0025] Figure 5 is an MBConv module in one embodiment of the present invention;
[0026] Figure 6 It is an SE module in one embodiment of the present invention. DETAILED DESCRIPTION
[0027] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0028] Please refer to Figure 1 The present invention provides a colonoscopy image evaluation system based on the EfficientNet structure, comprising
[0029] The manual interaction unit is used to separate the colorectal sequence images into three sub-sequence images of the left colon, the transverse colon, and the right colon by manually identifying the hepatic flexure images and the splenic flexure images, and to eliminate the images that are obviously unqualified;
[0030] The data classification unit uses EfficientNet as the basic network. The network model can be manually scaled according to the computing power configuration to obtain the best classification effect.
[0031] The evaluation unit is based on the BBPS intestinal evaluation, which scores the three sub-sequence images of the left colon, transverse colon, and right colon separately, and accumulates the image scores of the worst intestinal cleanliness in each segment to evaluate the adequacy of intestinal preparation.
[0032] In this embodiment, colon image acquisition generally adopts the method of withdrawing the scope and taking photos, that is, first inserting the scope from the liver hilum to the terminal ileum, checking and taking photos with the freeze button during the process of withdrawing the scope; there must be images for each structure and fixed range in the large intestine, from the terminal ileum, ileocecal valve, appendix crypt, ascending colon, hepatic flexure, transverse colon, splenic flexure, descending colon, sigmoid colon to rectum, there must be at least 1-2 high-quality images, and because the ascending colon, transverse colon, and descending colon are long, more images will be collected, and a total of 20-100 images can be collected.
[0033] In this embodiment, reference Figure 2, the colon is divided into three sections: left colon, transverse colon, and right colon, with the hepatic flexure and splenic flexure as the boundaries. For example, if a patient's colon sequence consists of 49 images, the initial file name is "0.bmp, 1.bmp, 2.bmp, n.bmp, ..., 49.bmp". The doctor can quickly browse and find the hepatic flexure and splenic flexure images, right-click the hepatic flexure image file name, and its file name will automatically change to "*G.bmp", with the letter "G" added after the sequence number as an identifier; similarly, the letter "P" is added after the sequence number of the splenic flexure image file name as an identifier, such as Figure 2 As shown. In this way, segmented evaluation can be performed based on the file name.
[0034] refer to Figure 3 , the manual interaction interface, the operation steps are as follows:
[0035] (1) Click "Open File" to import a series of colon images of a subject into the file list. Click on an image in the list to browse it one by one in the "Image Browsing Area".
[0036] (2) Right-click an image in the file list to mark the "liver flexure / spleen flexure" image; or you can remove unqualified images caused by improper photography.
[0037] (3) Click the "AI Automatic Evaluation" button and the "Score" column will give you the cleanliness score of each image, which is the score based on the four categories (0, 1, 2, 3) of the deep learning model.
[0038] (4) The “result evaluation area” gives the lowest evaluation results for the left colon, transverse colon, and right colon, and obtains the total BBPS score.
[0039] (5) If it is manually discovered that the automatic evaluation is wrong, the evaluation results of each segment can be corrected by operating the files in the file list to obtain the correct evaluation.
[0040] The contents of the inspection report are as follows: Figure 4 shown.
[0041] Preferably, in this embodiment, a colonoscopy generally acquires 20-100 color images with a large dynamic range, that is, a patient's examination report must be completed to complete this sequence of image detection, so in the actual clinical operation process, the system test speed is required to be high. Therefore, the deep learning network used must be able to take into account both accuracy and speed.
[0042] To improve the detection accuracy of images, improvements can be made from three dimensions: network depth, network width, and image resolution. For example, ResNet, VGG and other networks improve detection accuracy by increasing network depth, while GoogLeNet, MobileNet and other networks increase the number of channels to increase network width to achieve the purpose of improving accuracy. Although the detection results are optimized, it also brings about undesirable conditions such as network degradation or a sharp increase in computational complexity.
[0043] Preferably, in this embodiment, EfficientNet is selected as the deep learning network of the system artificial intelligence module. The network uses a composite model expansion method combined with neural structure search technology to scale the model from three dimensions: network depth, network width, and image resolution, and can obtain an optimal set of composite coefficients, that is, to obtain a scaling ratio that balances the three dimensions of network depth, network width, and image resolution. It is not only much faster than other networks, but also more accurate.
[0044] Preferably, in this embodiment, the composite model expansion method of the EfficientNet network structure is used to solve the composite coefficients of three dimensions, which is a constrained solution process. Suppose the network depth d, network width w, and image resolution r coefficients are respectively controlled by the parameters To measure, as shown in formula 1, its constraints are as shown in formula 2
[0045] (Formula 1)
[0046] (Formula 2).
[0047] in The reason for the square constraint is that if the width or resolution is doubled, the computational effort increases by 4 times, but if the depth is doubled, the computational effort only increases by 2 times.
[0048] When the zoom factor is fixed When the calculation amount is set to twice the original, the optimal parameters are obtained through grid search technology. , the basic model obtained is named: EfficientNet-B0.
[0049] When fixed Parameter values, using different To enlarge the network, we can get a sequence of models, EfficientNet-B1, ..., EfficientNet-B7.
[0050] In this embodiment, the colonoscopy image is a 4-class classification task. After experimental verification with a public data set, the first-generation model EfficientNet-B0 can meet the accuracy and speed requirements of the system on a machine with a GPU graphics card. On average, the evaluation of a subject can be completed within 1-3 seconds. The model parameter size of this model is only about 5M, and the average classification accuracy can reach more than 95%. In actual clinical applications, if the computing power configuration allows, the network scaling factor can be manually adjusted. To enlarge the network, we can obtain better classification accuracy. The structure of EfficientNet-B0 is shown in Table 2.
[0051] Table 2 EfficientNet-B0 network structure diagram
[0052]
[0053] It can be seen that the interior of the EfficientNet model is implemented through multiple MBConv convolution blocks. The specific structure of each MBConv convolution block is as follows Figure 5 As shown in the figure, the ReLU activation function caches the Swish activation function; the drop_connect method is used to replace the traditional drop method; BN is placed between the activation function and the convolution layer; a structure similar to the residual link is used, and the SE layer is used in the short connection part. The SE module takes into account the interdependence between model channels and includes two parts: Squeeze and Excitation. Figure 6 shown.
[0054] In order to verify the effectiveness of the network model, the Nerthus public dataset of colonoscopy images with cleanliness annotations collected by Bærum Hospital in Norway was selected. This dataset is extracted from 21 videos of subjects with 5525 frames of images, with a resolution of 720*576 and jpg format. Experienced endoscopists evaluated the intestinal cleanliness according to the Boston Scale, which is divided into 4 categories from 0-3. The number distribution of each category is shown in Table 3.
[0055] Table 3 Classification of Nerthus dataset
[0056]
[0057] The EfficientNet-B0 model was used for experiments and tested on this dataset (80% training set and 20% test set). The classification accuracy was high and the indicators are shown in Table 4.
[0058] method Acc Prec Rec Spec F1 score Mcc Comparison plan 0.949 0.901 0.901 0.960 0.899 0.863 The present invention 0.977 0.956 0.954 0.979 0.953 0.939
[0059] Among them, Acc (Accuracy), accuracy, represents the proportion of samples whose true values and predicted values are consistent among all samples; Prec (Precision), precision, represents the proportion of samples whose true values are positive among samples whose predicted values are positive; Rec (Recall), recall, represents the proportion of samples whose true values and predicted values are both positive among samples whose true values are positive; Spec (Specificity), specificity, represents the proportion of samples whose true values and predicted values are both negative among samples whose true values are negative; F1 score is the harmonic mean of precision and recall, an indicator of comprehensive performance; Mcc (Matthews correlation coefficient), Matthews correlation coefficient, represents the classification correlation coefficient between the sample predicted value and the true value, and its return value is between -1 and +1. The closer to +1, the better the classification performance.
[0060] In this embodiment, the Boston Bowel Preparation Scale (BBPS) can be used to classify the intestinal cleanliness represented by colonoscopy images into four categories, which are rated from 0 to 3 points as described in Table 1.
[0061] Table 1 Boston Bowel Preparation Scale (BBPS)
[0062]
[0063] The BBPS evaluation index divides the colon into three sections: the left colon, the transverse colon, and the right colon, based on the hepatic flexure and the splenic flexure. After the intestine is cleaned by extraction and perfusion, the evaluation is performed in sections. Each section of the colon is scored from 0 to 3 points, with a total score of 0 to 9 points. If the score of each section of the colon is ≥2, the intestinal preparation is considered to be adequate; if the total score of the three sections is less than 6 points or any section is less than 2 points, the intestinal preparation is considered to be inadequate.
[0064] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.
Claims
1. A colonoscopy image evaluation method based on EfficientNet structure, It is characterized in that The following steps are involved: Step S1: acquiring colorectal sequence images and preliminarily classifying them into three sub-sequence images of the left colon, transverse colon, and right colon; Step S2: preprocessing the subsequence images; Step S3: obtaining the optimal classification based on the preprocessed subsequence images and the scalable EfficientNet network model; Step S4: Based on the BBPS intestinal preparation evaluation standard, the three subsequence images of the left colon, transverse colon, and right colon are scored separately, and the scores of the images with the worst cleanliness of each segment of the intestine are accumulated to further evaluate whether the colonoscopy images are qualified; The preprocessing includes: removing unqualified images and resetting the resolution to meet the network requirements; The evaluation method is implemented by a colonoscopy image evaluation system based on the EfficientNet structure, and the system includes a manual interaction unit, a data classification unit and an evaluation unit; The manual interaction unit divides the colorectal sequence images into three sub-sequence images of the left colon, the transverse colon, and the right colon by manually identifying the hepatic flexure image and the splenic flexure image; removes the images that are obviously unqualified; and corrects the evaluation results of each segment when the automatic evaluation is found to be wrong manually; The data classification unit uses EfficientNet as the basic network, and can manually scale the network model according to the computing power configuration to obtain the best classification effect; The evaluation unit is based on the BBPS intestinal evaluation standard, that is, scoring the three sub-sequence images of the left colon, transverse colon, and right colon respectively, and accumulating the image scores of the worst intestinal cleanliness of each segment, so as to evaluate the adequacy of the intestinal preparation.
2. The colonoscopy image evaluation method based on the EfficientNet structure according to claim 1, It is characterized in that The EfficientNet network model includes several MBConv convolution blocks, and its expansion method is used to solve the composite coefficients of three dimensions, as follows: Let the network depth d, network width w, image resolution r, φ be a fixed scaling factor, and the coefficients are measured by parameters α, β, and γ, respectively, as shown in Formula 1, and its constraints are as shown in Formula 2 depth: d=a φ ,width: w=β φ ,resolution: r=γ φ (formula 1) α×β 2 ×γ 2 ≈2, α≥1, β≥1, γ≥1 (Formula 2).
3. The colonoscopy image evaluation method based on the EfficientNet structure according to claim 2, It is characterized in that The MBConv convolution block is specifically as follows: the ReLU activation function is cached with the Swish activation function; the drop_connect method is used to replace the traditional drop method; BN is placed between the activation function and the convolution layer; a structure similar to the residual link is used, and the SE layer is used in the short connection part; the SE module takes into account the interdependence between the model channels, and includes two parts: Squeeze and Excitation.
Citation Information
Patent Citations
Real-time intestinal tract cleanliness scoring system and method based on artificial intelligence
CN110916606A
