Method, device and storage medium for identifying colon polyp images

Abstract

This application relates to a method, device and storage medium for identifying colon polyp images. The method for identifying colon polyp images includes the steps of: performing color averaging and edge feature map fusion on a colon region image to obtain a gradient feature fusion map of the colon region image; and calling a classification model for colon polyp typing based on a pre-constructed attention division module ResNeSt Block to identify the gradient feature fusion map of the colon region image. This application constructs identification samples by performing color averaging and edge feature map fusion on the colon region image, effectively unifying and enhancing image features, and performing feature extraction through a feature extraction network constructed based on ResNeSt Block to more accurately predict colon polyp typing results and assist doctors in completing the estimation of pathological characteristics of colon polyps.

Description

[Technical field]

[0001] The present application relates to the field of medical technology, and in particular to a method, an apparatus, and a storage medium for recognizing colonic polyp images.

[0002] This application claims priority to a Chinese patent application filed with the China Patent Office on July 23, 2021, bearing application number 202110833639.8 and titled "Colorectal Polyp Image Identification Method, Apparatus and Storage Medium," the entire contents of which are incorporated herein by reference. [Background technology]

[0003] Colon polyps refer to protruding lesions that protrude into the intestinal cavity. They are one of the common intestinal diseases. There are adenomatous polyps and nonadenomatous polyps (hamartomatous polyps, metaplastic polyps, and inflammatory polyps), among which adenomatous polyps belong to precancerous lesions. The incidence and mortality rate of colorectal cancer in China has been increasing year by year, and many patients are already in the middle to late stages at the time of detection, with a 5-year survival rate of less than 50%. Colonoscopy is the most intuitive and effective method to detect lesions, which can improve the detection of early intestinal cancer and reduce mortality. Electronic staining endoscopy has been an important development in the field of digestive endoscopy in recent years, and narrow band imaging (NBI), endoscopic intelligent spectrocolorimetric technology (FICE), high-definition intelligent electronic staining endoscopy (i-scan), etc. have been widely used to improve the detectability of colorectal adenomas. Research has shown that electron staining technology has similar high sensitivity, specificity and accuracy in distinguishing between colorectal polypoid lesions, tumors and non-tumorous lesions, and electron staining typing has good application value in the diagnosis and treatment of colorectal lesions. However, typing currently relies on the doctor's experience and visual observation, and there are subjective differences. Summary of the Invention [Problem to be solved by the invention]

[0004] In order to solve the above technical problem, or at least partially solve the above technical problem, the present application provides a method, an apparatus, and a storage medium for identifying colon polyp images. [Means for solving the problem]

[0005] In a first aspect, the present application provides a method for identifying a colon polyp image, the method for identifying a colon polyp image comprising: performing color averaging and edge feature map fusion on the colon region image to obtain a gradient feature fusion map of the colon region image; The method includes a step of invoking a pre-constructed colon polyp typing classification model based on the attention division module ResNeSt Block, and identifying a gradient feature fusion map of the colon region image.

[0006] In addition, in order to predict or determine the typing of colon polyps, the above identification method may further include a step of evaluating the colon polyp typing of the colon site based on the identification result.

[0007] Optionally, the method for identifying a colon polyp image includes: performing color averaging and edge feature map fusion on the pre-acquired colon polyp sample image to obtain a fusion map of gradient features of the colon polyp sample image; training the classification model using the gradient feature fusion map of the colon polyp sample image and a feature extraction network based on a pre-constructed ResNeSt Block; and optimizing the classification model based on a loss function during the training of the classification model; After the training of the classification model is completed, a step of obtaining a classification model for colon polyp typing based on the ResNeSt Block is included.

[0008] Optionally, training a classification model based on the gradient feature fusion map of the colon polyp sample image and a feature extraction network based on the pre-constructed ResNeSt Block is performed. Training the constructed feature extraction network based on the ResNeSt Block on the gradient feature fusion map of the colon polyp sample image; Activating the output of the ResNeSt Block with the Mish function.

[0009] Optionally, the Mish function is f(x)=x*tanh(ln(1+e x )) and In the formula, x is each pixel value of the output feature map of the ResNeSt Block.

[0010] Optionally, training a gradient feature fusion map of the colon polyp sample images in the ResNeSt Block feature extraction network includes: Segmenting the gradient feature fusion map of the colon polyp sample image to obtain k packet convolutions; splitting each packet convolution to obtain r first feature maps; For the k*r first feature maps, first perform two convolution operations of 1*1 and 3*3, and then obtain r second feature maps again through packet convolution of each group, and then fuse the r second feature maps and input them into the Split Attention layer to enhance the features through the attention mechanism, and obtain k results; The method includes a step of continuously fusing k results, and then inputting the results to a 1*1 convolutional layer and performing convolutional calculations to obtain the output of the ResNeSt Block, where k and r are both hyperparameters.

[0011] Optionally, in the process of training the classification model, optimizing the classification model with a loss function includes: The method further includes a step of calculating a loss value by weighting the positive samples based on the ratio of the number of positive samples to the number of negative samples in the colon polyp sample image.

[0012] Optionally, the color averaging process comprises: determining RGB channel weight values ​​for the image based on the color average values ​​of the image; and weighting the RGB channels of the image based on the RGB channel weight values ​​of the image, the image being an image of a colon region or an image of a colon polyp sample.

[0013] Optionally, the edge feature map fusion comprises: scaling the RGB channel weighted image to obtain a scaled image; edge extracting the scaled image to obtain an edge feature map; fusing the scaled image and the edge feature map into a gradient feature fusion map.

[0014] In a second aspect, the present application provides a method for implementing a method comprising: a memory; a processor; and a computer program stored in the memory and executable by the processor, When the computer program is executed by the processor, it executes steps including any of the above-described methods for identifying colon polyp images.

[0015] In a third aspect, the present application provides an apparatus for predicting colon polyp typing results, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the computer program, when executed by the processor, performing steps including any of the above-described methods for identifying colon polyp images.

[0016] In a fourth aspect, the present application provides a method for performing a colonoscopy on a subject, the method comprising the step of predicting or determining the colonoscopic colonic polyp typing of the subject in real time using any of the above-described methods for identifying colonic polyp images or the above-described device for predicting colonic polyp typing results.

[0017] As used herein, the colon polyp typing refers to morphological typing such as non-neoplastic polyp, neoplastic polyp, or adenocarcinoma.

[0018] The typing standard may be a typing standard for conventional endoscopy or a typing standard for magnifying glass-coupled narrow band imaging (ME-NBI). Examples of the typing standard for ME-NBI endoscopy include the NICE typing standard, the Sano typing standard, the Pit typing standard, the Hiroshima typing standard, the Showa typing standard, the Jikei typing standard, and the JNET typing standard. The NICE typing standard is preferred.

[0019] The colon polyp image may be an image acquired in real time by an endoscope, or may be a stored image. Effect of the Invention

[0020] The above technical solutions provided by the embodiments of the present application have the following advantages over the prior art: In each embodiment of the present application, a discrimination sample is constructed by performing color averaging processing and edge feature map fusion processing on the colon region images, which effectively unifies and enhances the image features, and the feature extraction is performed by the ResNeSt Block, which can more accurately predict the colon polyp typing results and assist doctors in completing the estimation of the pathological characteristics of colon polyps. [Brief description of the drawings]

[0021] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the invention.

[0022] In order to more clearly describe the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art are briefly described below. Obviously, those skilled in the art can obtain other drawings based on these drawings without paying creative labor. [Figure 1] 1 is a flow chart of a method for predicting NICE typing results of colon polyps provided by various embodiments of the present application. [Diagram 2] FIG. 2 is a schematic diagram of a feature extraction network based on ResNeSt Block provided by various embodiments of the present application. [Diagram 3] 1 is a graph of an improved Mish function in accordance with various embodiments of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] It should be understood that the specific examples described herein are only used to illustrate the present invention and are not intended to limit the present invention.

[0024] In the following description, the use of suffixes expressing elements such as "module", "component" or "unit" is merely for facilitating the description of the present invention and does not have any specific meaning in itself. Therefore, "module", "component" or "unit" can be used interchangeably.

[0025] As used herein, the term "colorectal polyp typing" refers to morphological typing of non-neoplastic polyps, neoplastic polyps, or adenocarcinoma, and typing standards include, for example, the NICE typing criteria, the Sano typing criteria, the Pit typing criteria, the Hiroshima typing criteria, the Showa typing criteria, the Jikei typing criteria, and the JNET typing criteria.

[0026] Example 1 In an embodiment of the present invention, a method for predicting the NICE typing result of colon polyps is provided. As shown in FIG. 1, the method for predicting the NICE typing result of colon polyps includes: S101: performing color averaging and edge feature map fusion on a colon region image to obtain a gradient feature fusion map of the colon region image; S102: A step of calling a classification model of NICE typing pre-constructed based on the attention typing module ResNeSt Block to identify a gradient feature fusion map of the colon region image; S103: Evaluating the NICE typing type of the colon polyps in the colon region based on the identification result.

[0027] In the embodiment of the present application, a discrimination sample is constructed by performing color averaging processing and edge feature map fusion processing on the colon region image, and image features are effectively unified and enhanced, and feature extraction is performed by ResNeSt Block. Therefore, a computer-aided NICE typing system based on deep learning can be constructed, which can more accurately predict the NICE typing results of colon polyps and assist doctors in completing the estimation of the pathological characteristics of colon polyps.

[0028] In a specific implementation, colon polyp images are obtained in the modes of white light, NBI, FICE, BLI, LCI, I-SCAN, etc. NICE typing results include type 1, type 2, type 3 and normal bowel.

[0029] In some embodiments, before invoking a pre-built NICE typing classification model based on the divided attention module ResNeSt Block, performing color averaging and edge feature map fusion on the pre-acquired colon polyp sample image to obtain a gradient feature fusion map of the colon polyp sample image; A step of training a separation model by the gradient feature fusion map of the colon polyp sample image and a feature extraction network based on the pre-constructed ResNeSt Block; a step of optimizing the classification model by a loss function during the training process of the classification model; and a step of optimizing the classification model by the loss function. The formula for the loss function is as follows:

number

[0030] In the embodiment of the present invention, a training sample is constructed by performing color averaging processing and edge feature map fusion processing on the colon region images, which can effectively unify and enhance the image features, and further improve the prediction accuracy of the NICE typing results of colon polyps. The classification model is optimized by the loss function, which effectively increases the training speed of the model.

[0031] In some embodiments, training a classification model based on the gradient feature fusion map of the colon polyp sample image and a feature extraction network based on the pre-constructed ResNeSt Block includes: training the ResNeSt Block feature extraction network on a gradient feature fusion map of the colon polyp sample images; The method includes adding the output of the ResNeSt Block to the gradient feature fusion map of the colon polyp sample image and activating it with the improved Mish function.

[0032] Optionally, the improved Mish function is f(x)=x*tanh(ln(1+e x )) and In the formula, x is the pixel value of the output feature map of the ResNeSt Block.

[0033] Based on the above improved Mish function, the embodiment of the present invention effectively improves the feature fitting ability of the network.

[0034] Optionally, in the ResNeSt Block feature extraction network, training the gradient feature fusion map of the colon polyp sample image includes: Segmenting the gradient feature fusion map of the colon polyp sample image to obtain k packet convolutions; splitting each packet convolution to obtain r first feature maps; For the k*r first feature maps, first perform two convolution operations of 1*1 and 3*3, and then obtain r second feature maps again through packet convolution of each group, and fuse the r second feature maps and input them into the Split Attention layer to enhance the features through the attention mechanism, and obtain k results; The method includes a step of continuously fusing k results, and then inputting the results to a 1*1 convolutional layer and performing convolutional calculations to obtain the output of the ResNeSt Block, where k and r are both hyperparameters.

[0035] In detail, as shown in Figure 2, the process flow of the ResNeSt Block is as follows: a) Split the input (input of gradient feature fusion map) to obtain Cardinal 1-Cardinal k, and sum up k packet convolutions; b) continuing the image splitting at each packet convolution to obtain Split 1 - Split r and summing the r first feature maps; c) Then, for the k*r first feature maps, first perform 1*1 convolution calculation, then perform 3*3 convolution calculation again, and after two convolution operations, obtain r feature maps again by packet convolution of each group, which are then fused and input to the Split Attention layer to enhance the features through the attention mechanism, and finally obtain k results; d) Keep fusing the k results, input the results to a 1*1 convolutional layer to perform convolution calculations, and obtain an output; e) Adding the output of the ResNeSt Block to the input, activating the activation function, and finally obtaining a feature map.

[0036] Here, the activation function is selected to be a smoother Mish function, which improves the feature fitting ability of the network. The curve is shown in Figure 3.

[0037] The feature extraction network of the ResNeSt Block has a total of 16 groups of ResNeSt Blocks, and each group of blocks is composed of three layers of convolutions, so the network connection structure is as follows: Input → Conv*1 → ResNeSt Block(Conv*2+Attention+Conv*1)*16 → Fully connected layer.

[0038] The most effective part of ResNet is the residual module, and in the embodiment of the present invention, the original residual structure is modified to ResNeSt Block (attention division module). The feature extraction network can be composed of any number of ResNeSt Blocks, the number of which is inversely proportional to the training / prediction speed and proportional to the prediction accuracy. However, if the number of blocks is too large, the network fitting ability may degenerate and the accuracy may decrease. In the embodiment of the present invention, by adopting 16 groups of ResNeSt Blocks, not only can the prediction accuracy be effectively improved, but also the training speed is guaranteed.

[0039] Optionally, in the process of training the classification model, optimizing the classification model with the loss function includes: The method includes a step of weighting the positive samples based on the ratio of the number of positive samples to the number of negative samples in the colon polyp sample images, and calculating a loss value.

[0040] In addition, in the process of training the classification model, the model is pre-trained using a dataset on ImageNet to ensure that the starting parameters have good feature extraction and generalization capabilities. The initial learning rates for training the model are all 0.0001, and a fixed step decay method is used to decrease the learning rate. Based on this, the loss function simultaneously calculates the loss value by weighting the positive samples based on the ratio of the number of positive samples to the number of negative samples. The specific formula is:

number

[0041] The optimizer uses the AdamW optimizer algorithm, which includes weight decay and L2 regularization functions, which can effectively improve the training speed and model accuracy. The model reaches the optimal solution after about 300 iterations.

[0042] In some embodiments, the color averaging process includes: determining RGB channel weight values ​​for the image based on the color average values ​​of the image; and weighting the RGB channels of the image based on the RGB channel weight values ​​of the image, the image being an image of the colon region or an image of a colon polyp sample.

[0043] Here, the edge feature map fusion includes: scaling the RGB channel weighted image to obtain a scaled image; edge extracting the scaled image to obtain an edge feature map; and fusing the scaled image into the edge feature map and gradient feature fusion map.

[0044] In the embodiment of the present invention, based on the data of the original image samples, a training sample is constructed by superimposing the color averaging processing and gradient feature map fusion methods, and through the above two methods, the image features can be effectively unified and enhanced.

[0045] Here, color averaging involves: 1. First, add the R, G, and B channel values ​​of all pixels in the image and divide them by the number of pixels to obtain the average values ​​of each RGB channel, i.e., v(r), v(g), and v(b); 2. A step of calculating the total average value K, K=(v(r)+v(g)+v(b)) / 3; 3. Calculating each channel weight value, i.e., Kb=K / v(b), Kg=K / v(g), Kr=K / v(r); 4. The R, G, and B channel values ​​of the image are weighted and multiplied by Kr, Kg, and Kb, respectively, using the addWeighted function of python-opencv or similar.

[0046] By using color averaging processing, the color differences of the intestines caused by light and the patient themselves can be reduced, and the color tones of the dataset can be more uniform, thereby improving the training of the feature extraction network and the inspection efficiency.

[0047] Gradient feature map fusion involves 1. uniformly scaling the dataset to m pixels*n pixels to obtain a scaled image H, such as 430 pixels*368 pixels; 2. Converting the scaled image into a grayscale image, and then performing edge extraction using an edge extraction operator (e.g., Scharr, Canny, Sobel, Laplace, etc.) to obtain an edge feature map J; and fusing the scaled image H and the edge feature map J into a four-channel data, i.e., gradient feature fusion map.

[0048] In various embodiments of the present invention, a deep learning-based computer-assisted NICE typing system can be constructed, which can more accurately predict the NICE typing results of colon polyps and assist doctors in completing the pathological characteristic inference of colon polyps.

[0049] Example 2 An embodiment of the present invention provides an apparatus for predicting a result of NICE typing of colon polyps, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor.

[0050] The computer program, when executed by the processor, performs the steps of a method for predicting a NICE typing outcome of colon polyps as described in any of Example 1.

[0051] Here, the prediction device for the NICE typing results of colon polyps may be an endoscopic examination device, and the memory may be a cloud memory.

[0052] Example 3 In an embodiment of the present invention, a computer-readable storage medium is provided having stored thereon a program for predicting the results of NICE typing of colon polyps, which, when executed by a processor, realizes steps in a method for predicting the results of NICE typing of colon polyps as described in any of the embodiments of Example 1.

[0053] Example 4 In an embodiment of the present invention, a method of performing colonoscopy on a subject is provided, which includes a step of predicting or determining in real time the type of NICE typing of the subject's colonic polyps under endoscopic screening, using the method for predicting the results of NICE typing of colonic polyps described in Example 1, or the device for predicting the results of NICE typing of colonic polyps described in Example 2.

[0054] The specific implementations of Examples 2, 3 and 4 can refer to Example 1 to obtain corresponding technical effects.

[0055] It should be noted that, as used herein, the terms "comprise," "comprise," or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a set of elements includes not only those elements, but also other elements not expressly listed or that are inherent to such process, method, article, or apparatus. Without more limitations, an element qualified by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes this element.

[0056] The numbers of the above-mentioned embodiments of the present invention are given for the purpose of explanation only and do not indicate superiority or inferiority of the embodiments.

[0057] Through the description of the above embodiments, those skilled in the art can understand that the methods of the above embodiments can be realized by adding a necessary general-purpose hardware platform to the software, and of course, the hardware implementation is more preferable in many cases. Based on this understanding, the essential part of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk, etc.) that contains some instructions for executing the methods described in the various embodiments of the present invention on one terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.).

[0058] Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above specific embodiments, and the above specific embodiments are merely illustrative rather than limiting. A person skilled in the art can make many forms within the scope of the present invention under the teachings of the present invention without departing from the spirit of the present invention and the scope protected by the claims.

Claims

1. A method for identifying a colon polyp image, which is executed in a colon polyp typing result prediction device, the method comprising: a memory; a processor; and a computer program stored in the memory and executable by the processor, the method comprising: performing color averaging and edge feature map fusion on the pre-stored colon region image to obtain a gradient feature fusion map of the colon region image; Invoking a pre-constructed colon polyp typing classification model based on a divided attention module ResNeSt Block to identify a gradient feature fusion map of the colon segment image; The step of obtaining a gradient feature fusion map of the colon region image includes: Scaling the color-averaged image to obtain a scaled image; edge extracting the scaled image to obtain an edge feature map; fusing the scaled image and the edge feature map to obtain a gradient feature fusion map as four-channel data.

2. The method for identifying a colon polyp image includes the steps of: performing color averaging and edge feature map fusion on the pre-acquired colon polyp sample image to obtain a gradient feature fusion map of the colon polyp sample image; A step of training the classification model using the gradient feature fusion map of the colon polyp sample image and a feature extraction network based on ResNeSt Block that has been previously constructed; and a step of optimizing the classification model by a loss function during the training of the classification model, the loss function being expressed as follows: [0010] and after completing the training of the classification model, obtaining a classification model of colon polyp typing based on the ResNeSt Block, in which L represents a loss value, n represents the total number of categories, i represents the true category of the current data, t represents a 1D tensor of category probability distribution obtained through the feature extraction network, t[i] represents the predicted probability value of the classification model in the i category, and w(i) represents a weighting coefficient for the predicted loss value of the i category; 【number】 2. The method for identifying colon polyp images according to claim 1, wherein: represents exponential re-addition for all values ​​of t.

3. Training the classification model using the gradient feature fusion map of the colon polyp sample image and a pre-constructed feature extraction network based on ResNeSt Block includes: Training the pre-constructed ResNeSt Block-based feature extraction network on a gradient feature fusion map of the colon polyp sample image; 3. The method for identifying colon polyp images according to claim 2, further comprising a step of activating an output of a ResNeSt Block with a Mish function.

4. The Mish function is f(x)=x*tanh(ln(1+e ｘ )) and The method for identifying colon polyp images according to claim 3, wherein in the formula, x is a pixel value of the output feature map of the ResNeSt Block.

5. Training a gradient feature fusion map of the colon polyp sample image in the pre-constructed ResNeSt Block-based feature extraction network includes: Segmenting the gradient feature fusion map of the colon polyp sample image to obtain k packet convolutions; splitting each packet convolution to obtain r first feature maps; For the k*r first feature maps, first perform two convolution operations of 1*1 and 3*3, and then obtain r second feature maps again through packet convolution of each group, and then fuse the r second feature maps and input them into a Split Attention layer to enhance the features through the attention mechanism, and obtain k results; The method for identifying colon polyp images according to claim 3, further comprising: a step of continuously fusing the k results, inputting the results to a 1*1 convolutional layer for convolutional calculation to obtain an output of ResNeSt Block, wherein k and r are both hyperparameters.

6. In the step of training the classification model, optimizing the classification model using a loss function includes: The method for identifying colon polyp images according to claim 2, further comprising a step of calculating a loss value by weighting positive samples based on a ratio of the number of positive samples to negative samples in the colon polyp sample image.

7. The color averaging process includes: determining RGB channel weight values ​​for the image based on the color average values ​​of the image; and weighting the RGB channels of the image based on the RGB channel weighting values ​​of the image, wherein the image is an image of a colon region or an image of a colon polyp sample.

8. 2. The method for identifying colon polyp images according to claim 1, wherein the colon polyp typing standard is the NICE typing standard, the Sano typing standard, the Pit typing standard, the Hiroshima typing standard, the Showa typing standard, the Jikei typing standard, or the JNET typing standard.

9. 9. The method for identifying colon polyp images according to claim 8, wherein the colon polyp typing standard is the NICE typing standard.

10. A device for predicting colon polyp typing results, comprising: A computer program comprising: a memory; a processor; and a computer program stored in the memory and executable by the processor; A device for predicting colon polyp typing results, characterized in that, when the computer program is executed by the processor, the device executes steps including the method for identifying colon polyp images described in any one of claims 1 to 9.

11. 1. A computer-readable storage medium, comprising: The computer-readable storage medium stores a program for predicting colon polyp typing results, and when the program for predicting colon polyp typing results is executed by a processor, the computer-readable storage medium executes steps including the method for identifying colon polyp images described in any one of claims 1 to 9.

Images