Intelligent recognition system for intestinal adenomas
By using a deep learning system to segment and preprocess digital colon biopsy slides, a qualified intestinal adenoma recognition model was trained, solving the problems of accuracy and efficiency in colonoscopy intestinal adenoma recognition and achieving high-precision automated diagnosis.
Patent Information
- Application Number
- CN202310470740.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2025-10-31
- Estimated Expiration
- 2043-04-27
AI Technical Summary
In existing technologies, the identification and diagnosis of intestinal adenomas under colonoscopy relies on manual identification, which has problems such as strong subjectivity, unstable accuracy and low efficiency.
A deep learning system based on multiple staining configurations for digital colon biopsy slides was used to obtain a qualified intestinal adenoma recognition model through slide segmentation, preprocessing, training, validation, and testing. This model was then used to identify intestinal adenomas in digital colon biopsy slides.
It improves the accuracy and efficiency of intestinal adenoma identification, overcomes the subjectivity of manual diagnosis, has good generalization ability, and provides doctors with a high-precision auxiliary diagnostic tool.
Smart Images
Figure CN116486402B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical device technology, and in particular to an intelligent recognition system for intestinal adenomas. Background Technology
[0002] Colorectal cancer (CRC) is the third most common cancer worldwide, with over 940,000 new cases diagnosed annually. Most CRC cases develop from adenomatous polyps. It is estimated that over 50% of Westerners will develop colorectal adenomas in their lifetime, with 5%–16% of these developing into CRC. Detecting and removing these adenomas via colonoscopy can reduce the early incidence of CRC, and identifying intestinal adenomas based on biopsy specimens is crucial for developing precise treatment plans for patients.
[0003] However, the identification and diagnosis of intestinal biopsy sections stained with hematoxylin-eosin (HE) under a microscope requires a lot of repetitive work, and the criteria for manual identification and diagnosis of intestinal adenomas are relatively subjective, resulting in unstable accuracy and low identification efficiency in the identification of intestinal adenomas and early screening of CRC.
[0004] Therefore, this invention proposes an intelligent identification system for intestinal adenomas. Summary of the Invention
[0005] This invention provides an intelligent intestinal adenoma recognition system, which uses deep learning based on a set of digital colon biopsy slides with various staining configurations. This enables the trained, qualified intestinal adenoma recognition model to identify intestinal adenomas in digital colon biopsy slides with different staining configurations, overcoming the subjectivity of manual diagnosis, improving the accuracy of intestinal adenoma recognition, and possessing the advantages of high precision, high efficiency, and good generalization ability. It provides assistance to doctors in the identification and diagnosis of intestinal adenomas in patients and in the early screening of CRC.
[0006] This invention provides an intelligent identification system for intestinal adenomas, comprising:
[0007] The slice segmentation module is used to segment a collection of digital colon biopsy slices with various staining configurations to obtain a training set, a validation set, and a test set.
[0008] The deep learning module is used to train, validate, and test the initial customized model sequentially based on the training set, validation set, and test set to obtain a qualified intestinal adenoma recognition model.
[0009] The final identification module is used to input the digital slices of the colon biopsy to be identified into the qualified intestinal adenoma identification model to obtain the intestinal adenoma identification results.
[0010] Preferably, the slicing module includes:
[0011] The slice acquisition unit is used to acquire a set of digital colon biopsy slices with various staining configurations and determine the slice category of all digital colon biopsy slices in the set of digital colon biopsy slices;
[0012] The slice partitioning unit is used to divide the set of digitized colon biopsy slices with the maximum total number of slices into a training set, a validation set, and a test subset based on the slice category and a preset ratio.
[0013] The aggregation unit is used to aggregate the remaining digitized colon biopsy slices (excluding the set with the largest total number of slices) and the test subset to obtain the test set.
[0014] Preferred deep learning modules include:
[0015] The preprocessing unit is used to preprocess the training set and validation set to obtain a standard validation set and multiple standard training sets;
[0016] The training unit is used to train the initial customized model based on all standard training sets to obtain multiple training models;
[0017] The validation unit is used to select the intestinal adenoma recognition model from all training models based on the standard validation set and the preset index evaluation method.
[0018] The testing unit is used to input the test set into the intestinal adenoma recognition model based on the block overlap method, obtain the recognition accuracy of the test recognition results, and determine whether the intestinal adenoma recognition model meets the requirements based on the recognition accuracy. If it does, the intestinal adenoma recognition model is regarded as a qualified intestinal adenoma recognition model. Otherwise, the initial model is retrained, verified, and tested using a new set of digital colon biopsy slices with multiple staining configurations until the latest recognition accuracy meets the requirements. Then, the latest intestinal adenoma recognition model is regarded as a qualified intestinal adenoma recognition model.
[0019] Preferably, the preprocessing unit includes:
[0020] The filtering subunit is used to filter the background region of all digitally sliced colon biopsies in the training set based on the OTSU algorithm to obtain the main colon biopsy region.
[0021] The annotation subunit is used to perform pixel-level annotation on the main regions of colon biopsies containing intestinal adenomas in the training set to obtain the first training set.
[0022] The scaling subunit is used to process the labeled biopsy main regions in the first training set based on multiple preset field-of-view magnifications to obtain the second training set corresponding to each preset field-of-view magnification.
[0023] The segmentation subunit is used to segment the labeled biopsy main regions in the second training set with all preset field of view magnifications based on two first preset step sizes, to obtain a set of intestinal adenoma training image blocks with different pixel sizes and a set of non-intestinal adenoma training image blocks with different field of view magnifications.
[0024] The summarization subunit is used to summarize the set of intestinal adenoma training image blocks with the same field of view magnification and the same pixel size, and the set of non-intestinal adenoma training image blocks, to obtain the standard training set with the corresponding field of view magnification and corresponding pixel size.
[0025] The preprocessing subunit is used to preprocess the validation set to obtain the standard validation set.
[0026] Preferably, the preprocessing subunit includes:
[0027] The filtering end is used to filter the background area of all digital colon biopsy slices in the validation set based on the OTSU algorithm to obtain the main colon biopsy area.
[0028] The standard end is used to perform pixel-level annotation on the main regions of colon biopsies containing intestinal adenomas in the validation set to obtain the standard validation set.
[0029] Preferably, the training unit includes:
[0030] The image processing subunit is used to perform data augmentation, random scaling, and random perturbation on training image patches in all standard training sets to obtain a diverse set of training image patches with corresponding pixel sizes and field of view ratios.
[0031] The model training subunit is used to train the initial customized model separately based on all the diverse training image patch sets to obtain multiple training models.
[0032] The training image blocks include intestinal adenoma training image blocks from the intestinal adenoma training image block set and non-intestinal adenoma training image blocks from the non-intestinal adenoma training image block set.
[0033] Preferably, the image processing subunit includes:
[0034] The data augmentation end is used to randomly rotate and mirror training image patches in all standard training sets to obtain augmented training image patches with corresponding pixel sizes and field of view magnification.
[0035] The random scaling end is used to randomly scale augmented training image patches of the same pixel size and field of view based on a preset scaling ratio, so as to obtain image patches of diverse sizes corresponding to the pixel size of the corresponding field of view.
[0036] The random perturbation end is used to randomly perturb the basic attribute parameters of image patches with the same field of view and the same pixel size, and then summarize them to obtain a set of diverse training image patches with the corresponding field of view and pixel size.
[0037] Preferably, the verification unit includes:
[0038] The verification and recognition subunit is used to input all the digitized colon biopsy slices contained in the standard verification set into each training model to obtain the verification and recognition results of each training model;
[0039] The indicator evaluation subunit is used to calculate the evaluation indicator for each training model based on the true positive, false positive, true negative and false negative results in the verification and identification results and the preset indicator evaluation method.
[0040] The model selection subunit is used to select the intestinal adenoma recognition model from all training models based on evaluation metrics.
[0041] Preferably, the indicator evaluation subunit includes:
[0042] The calculation end is used to calculate the accuracy, sensitivity, and specificity of the training model based on the true positive, false positive, true negative, and false negative results in the verification and identification results, as well as the preset accuracy calculation formula, preset sensitivity calculation formula, and preset specificity calculation formula.
[0043] The aggregation end is used to use the accuracy, sensitivity, and specificity of the trained model as evaluation metrics for the corresponding trained model.
[0044] Preferably, the method for obtaining the recognition accuracy of the test recognition results by inputting the test set into the intestinal adenoma recognition model based on the block overlap method includes:
[0045] Based on the second preset step size, all digitized colon biopsy slices in the test set are segmented into test image blocks of the first pixel size;
[0046] All test image blocks are input into the intestinal adenoma recognition model to obtain the model recognition result for each test image block;
[0047] The model recognition results of all test image blocks are centered based on the second pixel size to obtain the test recognition result of each test image block, and the accuracy of the test recognition result is calculated.
[0048] The second pixel is smaller than the first pixel.
[0049] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.
[0050] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description
[0051] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:
[0052] Figure 1 This is a schematic diagram of the intelligent recognition system for intestinal adenomas in an embodiment of the present invention;
[0053] Figure 2 This is a schematic diagram illustrating the initialization of the custom model (DeepLab v2 model) in an embodiment of the present invention;
[0054] Figure 3 This is a schematic diagram of training image blocks with 10x, 20x and 40x field of view in an embodiment of the present invention. Detailed Implementation
[0055] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0056] Example 1:
[0057] This invention provides an intelligent identification system for intestinal adenomas, with reference to... Figure 1 ,include:
[0058] The slice segmentation module is used to segment a collection of digital colon biopsy slices with various staining configurations to obtain a training set, a validation set, and a test set.
[0059] The deep learning module is used to train, validate, and test the initial customized model sequentially based on the training set, validation set, and test set to obtain a qualified intestinal adenoma recognition model.
[0060] The final identification module is used to input the digital slices of the colon biopsy to be identified into the qualified intestinal adenoma identification model to obtain the intestinal adenoma identification results.
[0061] In this embodiment, the staining configuration includes the concentration of the staining solution or other adjustable parameters during the staining process when staining colon biopsy sections. Different hospitals have different staining configurations when staining colon biopsy sections. Therefore, when this embodiment performs deep learning based on digital colon biopsy sections with multiple staining configurations, the final qualified intestinal adenoma recognition model can identify intestinal adenomas in digital colon biopsy sections with different (hospital) staining configurations.
[0062] In this embodiment, the set of digitized colon biopsy slides is a collection of digitized images obtained by digitizing colon biopsy slides with the same staining configuration using a scanner with a 40x objective lens (eyepiece magnification fixed at 10x).
[0063] In this embodiment, the training set is a set of colon biopsy digital slices containing the same staining configuration, obtained by dividing the set of colon biopsy digital slices with multiple staining configurations, and used to train the initial customized model.
[0064] In this embodiment, the validation set is a set of colon biopsy digital slices containing the same staining configuration, obtained by dividing the set of colon biopsy digital slices with multiple staining configurations, and used to validate and screen the trained model (i.e., the model obtained after training the initial customized model).
[0065] In this embodiment, the test set is a set of colon biopsy digital slices with multiple staining configurations obtained by dividing the set of colon biopsy digital slices with multiple staining configurations. It is used to test the validated and screened model (i.e., the model obtained after training and validating the initial customized model).
[0066] In this embodiment, the qualified intestinal adenoma recognition model is a model that meets the requirements and is used to identify intestinal adenomas in digital colon biopsy slices with different staining configurations after training, validating and testing the initial customized model in sequence based on the training set, validation set and test set.
[0067] In this embodiment, the colon biopsy digital slice to be identified is the colon biopsy digital slice that needs to be identified and judged by the intestinal adenoma intelligent identification system (qualified intestinal adenoma identification model) in this embodiment to determine whether there is an intestinal adenoma.
[0068] In this embodiment, the intestinal adenoma identification result is the result obtained by the qualified intestinal adenoma identification model after identifying intestinal adenoma in the digital section of the colon biopsy to be identified (including the judgment result of whether there is intestinal adenoma in the digital section of the colon biopsy to be identified, and the location of intestinal adenoma in the digital section of the colon biopsy to be identified when it is determined that there is intestinal adenoma).
[0069] The beneficial effects of the above technologies are as follows: Deep learning based on a collection of digital colon biopsy slides with various staining configurations enables the trained qualified intestinal adenoma recognition model to identify intestinal adenomas in digital colon biopsy slides with different staining configurations, overcoming the subjectivity of manual diagnosis, improving the accuracy of intestinal adenoma recognition, and possessing the advantages of high precision, high efficiency and good generalization ability, thus providing assistance to doctors in the identification and diagnosis of intestinal adenomas in patients and early screening for CRC.
[0070] Example 2:
[0071] Based on Example 1, the slicing module includes:
[0072] The slice acquisition unit is used to acquire a set of digital colon biopsy slices with various staining configurations and determine the slice category of all digital colon biopsy slices in the set of digital colon biopsy slices;
[0073] The slice partitioning unit is used to divide the set of digitized colon biopsy slices with the maximum total number of slices into a training set, a validation set, and a test subset based on the slice category and a preset ratio.
[0074] The aggregation unit is used to aggregate the remaining digitized colon biopsy slices (excluding the set with the largest total number of slices) and the test subset to obtain the test set.
[0075] In this embodiment, a set of digital colon biopsy slides with various staining configurations was obtained. For example, 411 digital colon biopsy slides were collected at Hospital A (including 232 digital colon biopsy slides diagnosed as intestinal adenomas and 179 digital colon biopsy slides of normal mucosa or chronic inflammation (non-intestinal adenomas)). To further test the generalization ability of the model, 63 digital colon biopsy slides were collected at Hospital B and 105 digital colon biopsy slides were collected at Hospital C. The staining configurations of the digital colon biopsy slides collected from different hospitals were different.
[0076] In this embodiment, the slide categories include intestinal adenoma slides (i.e., colon biopsy slides containing intestinal adenomas) and non-intestinal adenoma slides (i.e., colon biopsy slides not containing intestinal adenomas).
[0077] In this embodiment, the preset ratio is the preset ratio for dividing a set of digital colon biopsy slices with various staining configurations into a training set, a validation set, and a test set.
[0078] In this embodiment, based on the slice type and a preset ratio, the set of digitized colon biopsy slices with the maximum total number of slices is divided into a training set, a validation set, and a test subset, for example:
[0079] Of the 411 digitized colon biopsy slides collected at Hospital A, 177 were designated as the training set (containing 156 slides with intestinal adenomas and the remainder with non-intestinal adenomas), 40 were designated as the validation set (containing 20 slides with intestinal adenomas and the remainder with non-intestinal adenomas), and 194 were designated as the test subset (containing 56 slides with intestinal adenomas and the remainder with non-intestinal adenomas).
[0080] In this embodiment, the remaining set of digitized colon biopsy slices (excluding the set with the maximum total number of slices) and the test subset are combined to obtain the test set, for example:
[0081] The test set was obtained by combining 194 digitized colon biopsy slides (i.e., the test subset, which includes 56 slides of intestinal adenoma and the rest of non-intestinal adenoma) from 411 digitized colon biopsy slides collected at Hospital A, 63 digitized colon biopsy slides (including 50 slides of intestinal adenoma and the rest of non-intestinal adenoma) collected at Hospital B, and 105 digitized colon biopsy slides (including 61 slides of intestinal adenoma and the rest of non-intestinal adenoma) collected at Hospital C.
[0082] The beneficial effects of the above technology are as follows: the collection of digital colon biopsy slides with various staining configurations is divided and re-summarized based on the slide category and preset ratio, resulting in a training set containing the same staining configuration, a validation set containing the same staining configuration, and a test set containing different staining configurations. This provides data for subsequent training, validation, and testing of the initial customized model, enabling the subsequently obtained qualified intestinal adenoma recognition model to have high accuracy, good generalization ability, and compatibility.
[0083] Example 3:
[0084] Based on Example 1, the deep learning module, refer to Figure 2 ,include:
[0085] The preprocessing unit is used to preprocess the training set and validation set to obtain a standard validation set and multiple standard training sets;
[0086] The training unit is used to train the initial customized model based on all standard training sets to obtain multiple training models;
[0087] The validation unit is used to select the intestinal adenoma recognition model from all training models based on the standard validation set and the preset index evaluation method.
[0088] The testing unit is used to input the test set into the intestinal adenoma recognition model based on the block overlap method, obtain the recognition accuracy of the test recognition results, and determine whether the intestinal adenoma recognition model meets the requirements based on the recognition accuracy. If it does, the intestinal adenoma recognition model is regarded as a qualified intestinal adenoma recognition model. Otherwise, the initial model is retrained, verified, and tested using a new set of digital colon biopsy slices with multiple staining configurations until the latest recognition accuracy meets the requirements. Then, the latest intestinal adenoma recognition model is regarded as a qualified intestinal adenoma recognition model.
[0089] In this embodiment, the standard validation set is the set of preprocessed colon biopsy digital slices obtained after preprocessing the validation set.
[0090] In this embodiment, the standard training set is the set of preprocessed colon biopsy digital slices obtained after preprocessing the training set.
[0091] In this embodiment, the training model is the model obtained by training the initial customized model using a single standard training set. The training models obtained by training the initial customized model using different standard training sets are different.
[0092] In this embodiment, reference Figure 2 The initial customized model is the preset DeepLab v2 (ResNet-34) model (that is, a semantic segmentation model based on ResNet-34. DeepLab v2 is a semantic segmentation model, and ResNet-34 is a residual neural network with 3×3 convolutional layers in the middle except for the first layer which is a 7×7 convolutional layer and the last layer which is a fully connected layer).
[0093] In this embodiment, the preset index evaluation method is a method that calculates the evaluation index based on the preset evaluation index calculation method and the verification recognition result obtained after inputting the standard validation set into the training model. Based on the evaluation index, the intestinal adenoma recognition model can be screened out from all training models.
[0094] In this embodiment, the intestinal adenoma identification model is a model selected from all training models based on a standard validation set and a preset index evaluation method. This model has only undergone training and validation, but has not undergone testing.
[0095] In this embodiment, the block overlap method is as follows: all digitized colon biopsy slices in the test set are divided into test image blocks with a pixel size of 2200×2200, and then input into the intestinal adenoma recognition model to obtain the recognition result of each test image block. Only the recognition result of the portion of the test image block with a pixel size of 2000×2000 at the center of each test image block is taken as the test recognition result. This method can further preserve the environmental information around the test image block (i.e., the image information of the image area around the test image block).
[0096] In this embodiment, the test recognition result is the result of identifying intestinal adenomas in the digital colon biopsy slices in the test set after inputting the test set into the intestinal adenoma recognition model based on the block overlap method.
[0097] In this embodiment, the recognition accuracy is the ratio of the total number of correctly recognized results (i.e., the total number of correctly recognized test image blocks) to the total number of all recognition results (i.e., the total number of all test image blocks).
[0098] In this embodiment, the intestinal adenoma recognition model is judged to meet the requirements based on the recognition accuracy. For example, a threshold judgment method can be used (that is, when the recognition accuracy is not less than the accuracy threshold (e.g., 90%), the intestinal adenoma recognition model is judged to meet the requirements; otherwise, the intestinal adenoma recognition model is judged not to meet the requirements).
[0099] The beneficial effects of the above technology are as follows: the initial customized model is trained on the standard validation set and multiple standard training sets obtained after preprocessing to obtain multiple training models. The intestinal adenoma recognition model is selected from the multiple training models by combining the preset index evaluation method. Then, the intestinal adenoma recognition model is cyclically tested based on the block overlap method, so that the qualified intestinal adenoma recognition model obtained by deep learning has high recognition accuracy and can also identify intestinal adenoma in digital sections of colon biopsy with different staining configurations.
[0100] Example 4:
[0101] Based on Example 3, the preprocessing unit, reference Figure 3 ,include:
[0102] The filtering subunit is used to filter the background region of all digital colon biopsy slices in the training set based on the OTSU algorithm (Otsu's method - maximum inter-class variance method) to obtain the main colon biopsy region;
[0103] The annotation subunit is used to perform pixel-level annotation on the main regions of colon biopsies containing intestinal adenomas in the training set to obtain the first training set.
[0104] The scaling subunit is used to process the labeled biopsy main regions in the first training set based on multiple preset field-of-view magnifications to obtain the second training set corresponding to each preset field-of-view magnification.
[0105] The segmentation subunit is used to segment the labeled biopsy main regions in the second training set with all preset field of view magnifications based on two first preset step sizes, to obtain a set of intestinal adenoma training image blocks with different pixel sizes and a set of non-intestinal adenoma training image blocks with different field of view magnifications.
[0106] The summarization subunit is used to summarize the set of intestinal adenoma training image blocks with the same field of view magnification and the same pixel size, and the set of non-intestinal adenoma training image blocks, to obtain the standard training set with the corresponding field of view magnification and corresponding pixel size.
[0107] The preprocessing subunit is used to preprocess the validation set to obtain the standard validation set.
[0108] In this embodiment, the background regions of all digitized colon biopsy slices in the training set are filtered using the OTSU algorithm (Otsu's method—maximum inter-class variance method) to obtain the main colon biopsy region, for example:
[0109] When the background area is dark, the region consisting of pixels exceeding the preset segmentation threshold in the grayscale images of all colon biopsy digital slices in the training set is taken as the background region, and the region consisting of pixels not exceeding the preset segmentation threshold in the grayscale images of all colon biopsy digital slices in the training set is taken as the main colon biopsy region (i.e., the foreground region in the colon biopsy digital slice, which is also the region that only contains the colon).
[0110] In this embodiment, pixel-level annotation refers to the operation of marking out the area of intestinal adenoma in the main colon biopsy area containing intestinal adenoma.
[0111] In this embodiment, pixel-level annotation is performed on the main colon biopsy regions containing intestinal adenomas in the training set to obtain the first training set, including:
[0112] Based on the manual annotation instructions, the manually annotated regions in each main colon biopsy region containing intestinal adenomas in the training set are determined;
[0113] Based on multiple first feature recognition methods (such as Harris corner detection algorithm and LoG-based multi-scale feature detector), the feature point set corresponding to each first feature recognition method is identified in the main region of the colon biopsy to be labeled, and based on the feature point set, the first total number of feature points corresponding to each first feature recognition method in the main region of the colon biopsy to be labeled and the second total number of feature points corresponding to each first feature recognition method in the manually labeled region are calculated.
[0114] The range of dimensionality reduction factors is calculated based on the first and second totals, including:
[0115] Based on the first total number of feature points corresponding to each first feature recognition method in the main region of the colon biopsy to be annotated and the second total number of feature points corresponding to each first feature recognition method in the manually annotated region, the optimal multiple range corresponding to each first feature recognition method is calculated:
[0116]
[0117] In the formula, [A, B] represents the optimal multiple range corresponding to the currently calculated first feature recognition method, A is the lower limit of the optimal multiple range corresponding to the currently calculated first feature recognition method, and B is the upper limit of the optimal multiple range corresponding to the currently calculated first feature recognition method. To obtain and The smaller value in the equation, m2 is the second total number of feature points corresponding to the first feature recognition method currently calculated in the manually labeled area, and m1 is the first total number of feature points corresponding to the first feature recognition method currently calculated in the manually labeled area. min This is the preset minimum total number of feature points allowed in the resulting low-dimensional image after dimensionality reduction (to ensure that the reduced image still contains a certain number of feature points, and also to ensure the accuracy of subsequent identification of adjacent connected regions in manually labeled areas of the low-dimensional image). To The value after rounding down. To The value after rounding down;
[0118] The average of the upper limits of the optimal fold range corresponding to all first feature recognition methods is taken as the final upper limit of the dimensionality reduction fold range, thereby determining a more reasonable dimensionality reduction fold range (that is, ensuring that the number of feature points in the dimensionality-reduced image is not too small, and also ensuring that the difference between the feature points in the main region of the colon biopsy to be labeled and the manually labeled region after dimensionality reduction based on the dimensionality reduction fold range is not too small).
[0119] A list of dimensionality reduction gradients is generated based on the dimensionality reduction range (i.e., the dimensionality reduction gradient is determined based on a preset method based on the dimensionality reduction range (the larger the dimensionality reduction range, the larger the dimensionality reduction gradient, and vice versa), and a list of dimensionality reduction gradients is generated based on the dimensionality reduction gradients. For example, if the dimensionality reduction range is [1, 10,] and the dimensionality reduction gradient is 2, then the list of dimensionality reduction gradients is 1, 3, 5, 7, 9).
[0120] Based on the dimensionality reduction factor in the dimensionality reduction gradient list (i.e., the ratio of the dimension before dimensionality reduction to the dimension after dimensionality reduction; if the dimension after dimensionality reduction determined based on the dimensionality reduction factor is not an integer, then the dimension after dimensionality reduction determined based on the dimensionality reduction factor is rounded down), PCA dimensionality reduction is performed on the main region of the colon biopsy to be labeled, and a list of low-dimensional images corresponding to the dimensionality reduction gradient list is obtained (i.e., a list containing low-dimensional images corresponding to different dimensionality reduction factors).
[0121] Based on the second feature recognition algorithm, the adjacent connected regions of the manually labeled regions in each low-dimensional image in the low-dimensional image list are identified (for example, the range of image attribute parameters (such as brightness, contrast, hue and saturation) of the manually labeled regions in the low-dimensional images is determined, and the pixels whose image attribute parameters meet the range of image attribute parameters are selected as pixels in the adjacent connected regions in turn. Then, the selection is continued among the adjacent pixels of the adjacent connected regions to obtain new adjacent connected regions until the low-dimensional images are traversed and the final adjacent connected regions are obtained). Each low-dimensional image and its corresponding adjacent connected regions are summarized to obtain the first labeled region of each dimensionality reduction factor (that is, the region composed of the low-dimensional image and its corresponding adjacent connected regions).
[0122] Common features are extracted from the first labeled regions in all low-dimensional images with different dimensionality reduction factors (e.g., after aligning all low-dimensional images with different dimensionality reduction factors, the overlapping regions of the first labeled regions in all low-dimensional images are taken) to obtain the standard labeled regions (that is, the regions obtained after common feature extraction from the first labeled regions in all low-dimensional images with different dimensionality reduction factors).
[0123] After labeling and summarizing the standard labeled regions in all unlabeled colon biopsy master regions containing intestinal adenomas in the training set, the first training set is obtained.
[0124] The above steps determine a reasonable range of dimensionality reduction factors based on the total number of feature points and establish a reasonable list of dimensionality reduction gradients. Then, dimensionality reduction processing is performed on the main region of the colon biopsy to be annotated. Based on the dimensionality-reduced low-dimensional image and other feature recognition algorithms, adjacent connected regions of the manually annotated region are identified, thereby achieving pixel-level intelligent extension of the manually annotated region. After commonality extraction, the intelligently extended region is filtered, thus achieving accurate extension of the manually annotated region based on the manual annotation, which greatly improves the accuracy of the annotation results.
[0125] In this embodiment, the first training set is the new training set obtained by marking out the intestinal adenoma regions in all the main regions of colon biopsy containing intestinal adenomas in the training set.
[0126] In this embodiment, the labeled biopsy master region is the new colon biopsy master region obtained after labeling the intestinal adenoma region in the colon biopsy master region containing intestinal adenoma in the first training set.
[0127] In this embodiment, the second training set is a new training set obtained by processing the marked biopsy main region in the first training set based on various preset field-of-view magnifications, such as: a 10x field-of-view training set, a 20x field-of-view training set, and a 40x field-of-view training set.
[0128] In this embodiment, the annotated biopsy main regions in the first training set are processed based on multiple preset field-of-view magnifications to obtain a second training set corresponding to each preset field-of-view magnification, for example:
[0129] After converting the labeled biopsy main regions in the first training set into 10x, 20x, and 40x fields of view, training sets with 10x, 20x, and 40x fields of view are obtained.
[0130] In this embodiment, the first preset step size is a pixel size of 640 and a pixel size of 320.
[0131] In this embodiment, the labeled biopsy main regions in the second training set with all preset field-of-view magnifications are segmented based on two first preset step sizes to obtain a set of intestinal adenoma training image blocks with different pixel sizes at different field-of-view magnifications and a set of non-intestinal adenoma training image blocks, namely:
[0132] Based on pixel sizes of 640 and 320, the labeled biopsy main regions in the second training set with all preset field-of-view multiples are segmented to obtain sets of intestinal adenoma training image blocks with sizes of 10x field of view (640×640 pixels) and non-intestinal adenoma training image blocks, sets of 20x field of view (640×640 pixels) and non-intestinal adenoma training image blocks, sets of 40x field of view (640×640 pixels) and non-intestinal adenoma training image blocks, sets of 10x field of view (320×320 pixels) and non-intestinal adenoma training image blocks, sets of 20x field of view (320×320 pixels) and non-intestinal adenoma training image blocks, sets of 40x field of view (320×320 pixels) and non-intestinal adenoma training image blocks.
[0133] In this embodiment, Figure 3 The images in the image are, in order, training image patches of size 10x640×640 pixels with a field of view, training image patches of size 20x640×640 pixels with a field of view, training image patches of size 40x640×640 pixels with a field of view, and training image patches of size 40x320×320 pixels with a field of view.
[0134] In this embodiment, the training image block set for intestinal adenoma is the set of all training image blocks containing intestinal adenoma regions.
[0135] In this embodiment, the set of non-intestinal adenoma training image blocks is the set of all training image blocks that do not contain intestinal adenoma regions.
[0136] The beneficial effects of the above techniques are as follows: by performing background filtering, pixel-level annotation, field-of-view scaling conversion, and image segmentation on all digital colon biopsy slices in the training set, a training set with multiple field-of-view scaling and pixel sizes is obtained, enabling the trained model to adapt to digital colon biopsy slices with multiple pixel sizes and multiple field-of-view scaling, thereby improving the compatibility of the trained model.
[0137] Example 5:
[0138] Based on Example 4, the preprocessing subunit includes:
[0139] The filtering end is used to filter the background area of all digital colon biopsy slices in the validation set based on the OTSU algorithm to obtain the main colon biopsy area.
[0140] The standard end is used to perform pixel-level annotation on the main regions of colon biopsies containing intestinal adenomas in the validation set to obtain the standard validation set.
[0141] The beneficial effects of the above techniques are as follows: by performing background filtering and pixel-level annotation on all digitized colon biopsy slices in the validation set, the data for subsequent validation and screening of the training model is improved.
[0142] Example 6:
[0143] Based on Example 4, the training unit includes:
[0144] The image processing subunit is used to perform data augmentation, random scaling, and random perturbation on training image patches in all standard training sets to obtain a diverse set of training image patches with corresponding pixel sizes and field of view ratios.
[0145] The model training subunit is used to train the initial customized model separately based on all the diverse training image patch sets to obtain multiple training models.
[0146] The training image blocks include intestinal adenoma training image blocks from the intestinal adenoma training image block set and non-intestinal adenoma training image blocks from the non-intestinal adenoma training image block set.
[0147] In this embodiment, the diverse training image patch set is the training image patch set obtained by performing data augmentation, random scaling, and random perturbation on the training image patches in all standard training sets.
[0148] In this embodiment, the initial customized model is trained separately based on all the diverse training image patch sets to obtain multiple training models, namely:
[0149] Each set of diverse training image patches for each field of view magnification and each pixel size is input into the initial customized model for separate training to obtain the training model corresponding to each set of diverse training image patches for each field of view magnification and each pixel size.
[0150] During training, by using the skip-layer fusion method, the model can simultaneously learn the low-level, mid-level, and high-level features of diverse training image patches. This allows the model to learn the image features of diverse training image patches more extensively, and further improves the recognition accuracy of the trained model.
[0151] The beneficial effects of the above technologies are as follows: Based on the diverse training image patch set with different field-of-view multiples and pixel sizes obtained after data augmentation, random scaling, and random perturbation, the initial customized model is trained separately, which makes it easier to select the best-performing training model from the multiple training models obtained after training as the intestinal adenoma recognition model, and also enhances the compatibility of the training model with different staining configurations.
[0152] Example 7:
[0153] Based on Embodiment 6, the image processing subunit includes:
[0154] The data augmentation end is used to randomly rotate and mirror training image patches in all standard training sets to obtain augmented training image patches with corresponding pixel sizes and field of view magnification.
[0155] The random scaling end is used to randomly scale augmented training image patches of the same pixel size and field of view based on a preset scaling ratio, so as to obtain image patches of diverse sizes corresponding to the pixel size of the corresponding field of view.
[0156] The random perturbation end is used to randomly perturb the basic attribute parameters of image patches of different sizes with the same field of view multiple and the same pixel size (which can enhance the compatibility of the trained model with different coloring configurations) and then summarize them to obtain a set of diverse training image patches with the corresponding pixel size for the corresponding field of view multiple.
[0157] In this embodiment, the enhanced training image patch is a new training image patch obtained by randomly rotating and mirroring the training image patches in all standard training sets (so that the trained model can overcome the recognition error caused by the uncertainty of the observation direction).
[0158] In this embodiment, the preset scaling factor is the preset ratio for randomly scaling the enhanced training image patch, for example, 1.0x to 1.5x.
[0159] In this embodiment, the image blocks with diverse sizes are training image blocks obtained by randomly scaling enhanced training image blocks of the same pixel size and field of view based on a preset scaling factor (so that the trained model can overcome the recognition error caused by the difference in the scanning parameters of the scanner).
[0160] In this embodiment, the basic attribute parameters of image patches of different sizes with the same field of view multiple and the same pixel size are randomly perturbed, that is, the brightness, contrast, hue and saturation of the image patches of different sizes are randomly perturbed (one or more of the parameters of brightness, contrast, hue and saturation of a single image patch of different sizes are added or subtracted by a certain value, for example, the brightness of the same image patch of different sizes is reduced by half. Here, the value of the brightness reduction of the pixels in a single image patch of different sizes is uniform, but the specific value is randomly generated), which can enhance the compatibility of the trained model with different coloring configurations.
[0161] In this embodiment, the basic attribute parameters include the brightness, contrast, hue, and saturation of image blocks (pixels) of varying sizes.
[0162] The beneficial effects of the above techniques are as follows: by randomly rotating and mirroring the training image patches in all standard training sets, randomly scaling them based on a preset scaling factor, and randomly perturbing their basic attribute parameters, the model trained on the diverse training image patch set obtained after the above operations can overcome the recognition error caused by the uncertainty of the observation direction, overcome the recognition error caused by the difference in the scanning parameters of the scanner, and enhance the compatibility of the trained model with different staining configurations.
[0163] Example 8:
[0164] Based on Example 3, the verification unit includes:
[0165] The verification and recognition subunit is used to input all the digitized colon biopsy slices contained in the standard verification set into each training model to obtain the verification and recognition results of each training model;
[0166] The indicator evaluation subunit is used to calculate the evaluation indicator for each training model based on the true positive, false positive, true negative and false negative results in the verification and identification results and the preset indicator evaluation method.
[0167] The model selection subunit is used to select the intestinal adenoma recognition model from all training models based on evaluation metrics.
[0168] In this embodiment, the verification and identification result is the identification result obtained after inputting all the digital colon biopsy slices contained in the standard verification set into each training model, which identifies whether there is an intestinal adenoma in all the digital colon biopsy slices in the standard verification set.
[0169] In this embodiment, a true positive is the total number of verification results that correctly identify the presence of intestinal adenomas.
[0170] In this embodiment, a false positive is the total number of verification results that identify the presence of intestinal adenoma but do not actually contain intestinal adenoma.
[0171] In this embodiment, a true negative is the total number of verification results that correctly identify the absence of intestinal adenomas.
[0172] In this embodiment, a false negative is the total number of verification results that identify an intestinal adenoma when it is not present but actually exists.
[0173] In this embodiment, the evaluation index is an index used to evaluate the performance of the trained model, calculated based on the true positive, false positive, true negative and false negative results in the verification and identification results and a preset index evaluation method.
[0174] In this embodiment, an intestinal adenoma identification model is selected from all training models based on evaluation metrics. For example, the training model with the highest average value of all evaluation metrics is selected as the intestinal adenoma identification model. If the evaluation metrics include slice analysis time, the training model with the shortest slice analysis time is selected as the intestinal adenoma identification model.
[0175] The beneficial effects of the above technology are as follows: Based on the preset index evaluation method and the verification and recognition results obtained after inputting all the digital colon biopsy slices contained in the standard validation set into each training model, the evaluation index that can evaluate the model performance of the training model is calculated, thereby realizing the verification and screening of the training model, so that the screened intestinal adenoma recognition model has good model performance.
[0176] Example 9:
[0177] Based on Example 8, the index evaluation subunit includes:
[0178] The calculation end is used to calculate the accuracy, sensitivity, and specificity of the training model based on the true positive, false positive, true negative, and false negative results in the verification and identification results, as well as the preset accuracy calculation formula, preset sensitivity calculation formula, and preset specificity calculation formula.
[0179] The aggregation end is used to use the accuracy, sensitivity, and specificity of the trained model as evaluation metrics for the corresponding trained model.
[0180] In this embodiment, the preset accuracy calculation formula is as follows:
[0181]
[0182] The default sensitivity calculation formula is:
[0183]
[0184] The default specificity calculation formula is:
[0185]
[0186] Among them, TP is a true positive, FP is a false positive, TN is a true negative, and FN is a false negative.
[0187] Accuracy refers to the accuracy of the verification and identification results, sensitivity refers to the true positive rate of the verification and identification results, and specificity refers to the true negative rate of the verification and identification results.
[0188] The beneficial effects of the above technology are as follows: it enables the use of the accuracy, sensitivity, and specificity calculated based on the true positive, false positive, true negative, and false negative results in the verification and identification results, as well as the preset accuracy calculation formula, preset sensitivity calculation formula, and preset specificity calculation formula, as evaluation indicators for training models. This results in the final selected intestinal adenoma identification model having good accuracy, sensitivity, and specificity, that is, having a high identification accuracy, true positive rate, and true negative rate.
[0189] Example 10:
[0190] Based on Example 3, the method for the test unit to input the test set into the intestinal adenoma recognition model using the block overlap method and obtain the recognition accuracy of the test recognition results includes:
[0191] Based on the second preset step size, all digitized colon biopsy slices in the test set are segmented into test image blocks of the first pixel size;
[0192] All test image blocks are input into the intestinal adenoma recognition model to obtain the model recognition result for each test image block;
[0193] The model recognition results of all test image blocks are centered based on the second pixel size to obtain the test recognition result of each test image block, and the accuracy of the test recognition result is calculated.
[0194] The second pixel is smaller than the first pixel.
[0195] In this embodiment, the second preset step size is a pixel size of 2200.
[0196] In this embodiment, the test image block with the first pixel size is a test image block with a size of 2200×2200 pixels.
[0197] In this embodiment, the test image block is the image block obtained by segmenting all digitized colon biopsy slices in the test set based on a second preset step size.
[0198] In this embodiment, the model recognition result is the result obtained after inputting all test image blocks into the intestinal adenoma recognition model, which identifies whether an intestinal adenoma exists in each test image block.
[0199] In this embodiment, the second pixel size is a pixel size of 2000×2000.
[0200] In this embodiment, the model recognition results of all test image blocks are centered based on the second pixel size to obtain the test recognition result of each test image block, which is:
[0201] The model recognition result of the 2000×2000 pixel portion of the test image block located at the center of the 2200×2200 pixel test image block is taken as the test recognition result.
[0202] In this embodiment, the accuracy of the test recognition results is calculated as the ratio of the total number of correctly recognized results (i.e., the total number of correctly recognized test image blocks) to the total number of all recognition results (i.e., the total number of incorrectly recognized test image blocks).
[0203] The beneficial effects of the above technologies are as follows: the model recognition results are centered based on the block overlap method, which allows the intestinal adenoma recognition model to refer to the environmental information around the test image block during the test recognition process. It also enables the use of test image blocks of different pixel sizes to test the intestinal adenoma recognition model, further increasing the compatibility of the qualified intestinal adenoma recognition model with the pixel size of colon biopsy sections.
[0204] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. An intelligent identification system for intestinal adenomas, characterized in that, include: The slice segmentation module is used to segment a collection of digital colon biopsy slices with various staining configurations to obtain a training set, a validation set, and a test set. The deep learning module is used to train, validate, and test the initial customized model sequentially based on the training set, validation set, and test set to obtain a qualified intestinal adenoma recognition model, including: The preprocessing unit is used to preprocess the training and validation sets to obtain a standard validation set and multiple standard training sets, including: The filtering subunit is used to filter the background region of all digitally sliced colon biopsies in the training set based on the OTSU algorithm to obtain the main colon biopsy region. The annotation subunit is used to perform pixel-level annotation on the main regions of colon biopsies containing intestinal adenomas in the training set, obtaining the first training set, which includes: Based on the manual annotation instructions, the manually annotated regions in each main colon biopsy region containing intestinal adenomas in the training set are determined; Based on multiple first feature recognition methods, the feature point set corresponding to each first feature recognition method is identified in the main area of the colon biopsy to be labeled, and the first total number of feature points corresponding to each first feature recognition method in the main area of the colon biopsy to be labeled and the second total number of feature points corresponding to each first feature recognition method in the manually labeled area are calculated based on the feature point set. Based on the first total number of feature points corresponding to each first feature recognition method in the main region of the colon biopsy to be annotated and the second total number of feature points corresponding to each first feature recognition method in the manually annotated region, the optimal multiple range corresponding to each first feature recognition method is calculated: In the formula, [A, B] represents the optimal multiple range corresponding to the currently calculated first feature recognition method, A is the lower limit of the optimal multiple range corresponding to the currently calculated first feature recognition method, and B is the upper limit of the optimal multiple range corresponding to the currently calculated first feature recognition method. To obtain and The smaller value in the equation, m2 is the second total number of feature points corresponding to the first feature recognition method currently calculated in the manually labeled area, and m1 is the first total number of feature points corresponding to the first feature recognition method currently calculated in the manually labeled area. min This is the preset minimum total number of feature points allowed to be contained in the low-dimensional image obtained after dimensionality reduction. To The value after rounding down. To The value after rounding down; The average of the upper limits of the optimal multiple range corresponding to all first feature recognition methods is taken as the final upper limit of the dimensionality reduction multiple range, thereby determining a more reasonable dimensionality reduction multiple range; Generate a list of dimensionality reduction gradients based on the dimensionality reduction factor range; Based on the dimensionality reduction factor in the dimensionality reduction gradient list, PCA dimensionality reduction is performed on the main region of the colon biopsy to be labeled, and a list of low-dimensional images corresponding to the dimensionality reduction gradient list is obtained. Based on the second feature recognition algorithm, the adjacent connected regions of the manually annotated regions in each low-dimensional image in the low-dimensional image list are identified, and the first annotated region for each dimensionality reduction factor is obtained by summing up each low-dimensional image and its corresponding adjacent connected regions. Common features are extracted from the first labeled regions in all low-dimensional images with dimensionality reduction factors to obtain standard labeled regions; After labeling and summarizing the standard labeled regions in all unlabeled colon biopsy master regions containing intestinal adenomas in the training set, the first training set is obtained. The scaling subunit is used to process the labeled biopsy main regions in the first training set based on multiple preset field-of-view magnifications to obtain the second training set corresponding to each preset field-of-view magnification. The segmentation subunit is used to segment the labeled biopsy main regions in the second training set with all preset field of view magnifications based on two first preset step sizes, to obtain a set of intestinal adenoma training image blocks with different pixel sizes and a set of non-intestinal adenoma training image blocks with different field of view magnifications. The summarization subunit is used to summarize the set of intestinal adenoma training image blocks with the same field of view magnification and the same pixel size, and the set of non-intestinal adenoma training image blocks, to obtain the standard training set with the corresponding field of view magnification and corresponding pixel size. The preprocessing subunit is used to preprocess the validation set to obtain the standard validation set; The training unit is used to train the initial customized model based on all standard training sets to obtain multiple training models; The validation unit is used to select the intestinal adenoma recognition model from all training models based on the standard validation set and the preset index evaluation method. The testing unit is used to input the test set into the intestinal adenoma recognition model based on the block overlap method, obtain the recognition accuracy of the test recognition results, and determine whether the intestinal adenoma recognition model meets the requirements based on the recognition accuracy. If it does, the intestinal adenoma recognition model is regarded as a qualified intestinal adenoma recognition model. Otherwise, the initial model is retrained, verified, and tested using a new set of digital colon biopsy slices with multiple staining configurations until the latest recognition accuracy meets the requirements. Then, the latest intestinal adenoma recognition model is regarded as a qualified intestinal adenoma recognition model. The final identification module is used to input the digital slices of the colon biopsy to be identified into the qualified intestinal adenoma identification model to obtain the intestinal adenoma identification results.
2. The intelligent identification system for intestinal adenomas according to claim 1, characterized in that, The slicing module includes: The slice acquisition unit is used to acquire a set of digital colon biopsy slices with various staining configurations and determine the slice category of all digital colon biopsy slices in the set of digital colon biopsy slices; The slice partitioning unit is used to divide the set of digitized colon biopsy slices with the maximum total number of slices into a training set, a validation set, and a test subset based on the slice category and a preset ratio. The aggregation unit is used to aggregate the remaining digitized colon biopsy slices (excluding the set with the largest total number of slices) and the test subset to obtain the test set.
3. The intelligent identification system for intestinal adenomas according to claim 1, characterized in that, The preprocessing subunit includes: The filtering end is used to filter the background area of all digital colon biopsy slices in the validation set based on the OTSU algorithm to obtain the main colon biopsy area. The standard end is used to perform pixel-level annotation on the main regions of colon biopsies containing intestinal adenomas in the validation set to obtain the standard validation set.
4. The intelligent recognition system for intestinal adenomas according to claim 3, characterized in that, Training units include: The image processing subunit is used to perform data augmentation, random scaling, and random perturbation on training image patches in all standard training sets to obtain a diverse set of training image patches with corresponding pixel sizes and field of view ratios. The model training subunit is used to train the initial customized model separately based on all the diverse training image patch sets to obtain multiple training models. The training image blocks include intestinal adenoma training image blocks from the intestinal adenoma training image block set and non-intestinal adenoma training image blocks from the non-intestinal adenoma training image block set.
5. The intelligent identification system for intestinal adenomas according to claim 4, characterized in that, The image processing subunit includes: The data augmentation end is used to randomly rotate and mirror training image patches in all standard training sets to obtain augmented training image patches with corresponding pixel sizes and field of view magnifications. The random scaling end is used to randomly scale augmented training image patches of the same pixel size and field of view based on a preset scaling ratio, so as to obtain image patches of diverse sizes corresponding to the pixel size of the corresponding field of view. The random perturbation end is used to randomly perturb the basic attribute parameters of image patches with the same field of view and the same pixel size, and then summarize them to obtain a set of diverse training image patches with the corresponding field of view and pixel size.
6. The intelligent identification system for intestinal adenomas according to claim 1, characterized in that, The verification unit includes: The verification and recognition subunit is used to input all the digitized colon biopsy slices contained in the standard verification set into each training model to obtain the verification and recognition results of each training model; The indicator evaluation subunit is used to calculate the evaluation indicator for each training model based on the true positive, false positive, true negative and false negative results in the verification and identification results and the preset indicator evaluation method. The model selection subunit is used to select the intestinal adenoma recognition model from all training models based on evaluation metrics.
7. The intelligent identification system for intestinal adenomas according to claim 6, characterized in that, The indicator evaluation sub-unit includes: The calculation end is used to calculate the accuracy, sensitivity, and specificity of the training model based on the true positive, false positive, true negative, and false negative results in the verification and identification results, as well as the preset accuracy calculation formula, preset sensitivity calculation formula, and preset specificity calculation formula. The aggregation end is used to use the accuracy, sensitivity, and specificity of the trained model as evaluation metrics for the corresponding trained model.
8. The intelligent identification system for intestinal adenomas according to claim 1, characterized in that, The method for obtaining the recognition accuracy of the test recognition results by inputting the test set into the intestinal adenoma recognition model based on the block overlap method includes: Based on the second preset step size, all colon biopsy digitized slices in the test set are segmented into test image blocks of the first pixel size; All test image blocks are input into the intestinal adenoma recognition model to obtain the model recognition result for each test image block; The model recognition results of all test image blocks are centered based on the second pixel size to obtain the test recognition result of each test image block, and the accuracy of the test recognition result is calculated. The second pixel is smaller than the first pixel.
Citation Information
Patent Citations
Computer diagnosis method and system of pancreatic neuroendocrine tumor malignant degrees
CN108766555A