Enteroscope image evaluation model training method and device, equipment and storage medium
By pre-processing and annotating the historical videos collected by colonoscopy, training and fusing contaminants and examination object recognition models, the problem of inaccurate assessment of intestinal cleanliness and colonoscopy quality in the prior art is solved, and a more accurate colonoscopy image evaluation is achieved.
Patent Information
- Application Number
- CN202510382399.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-28
- Publication Date
- 2025-06-27
AI Technical Summary
In the prior art, there are inaccurate problems in the examination of intestinal cleanliness and colonoscopy quality assessment, mainly due to strong human subjectivity and high image quality requirements.
By obtaining the historical examination video collected by the colonoscopy, pre-processing is performed to obtain historical colonoscopy images, and annotation is performed based on the pollutant category and the examination object. Then, the pollutant identification model is trained and the object identification model is checked, and the model is fused to finally obtain the colonoscopy image evaluation model.
Accurate evaluation of colonoscopic images is achieved, and the accuracy and reliability of intestinal cleanliness and colonoscopic quality evaluation is improved.
Smart Images
Figure CN120219885A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of medical assistance technologies, and in particular, to a method, apparatus, electronic device, and storage medium for training a colonoscopy image evaluation model. Background Art
[0002] Colonoscopy is an important means for diagnosing and treating colorectal diseases. Poor cleanliness caused by insufficient bowel preparation may lead to a decrease in mucosal visibility, an increase in the missed diagnosis rate of lesions, an extended examination time, and increased costs. Therefore, real-time monitoring of bowel cleanliness can better improve the efficiency and accuracy of the examination.
[0003] Currently, the conventional method is to make a judgment based on a manual subjective method. Specifically, by obtaining images of the intestine and then having a doctor manually view and judge to determine the cleanliness of the intestine. However, this method requires the collected images to have relatively high quality, and at the same time, the judgment of cleanliness has strong human subjectivity, resulting in inaccurate examination and judgment of bowel cleanliness. Summary of the Invention
[0004] The purpose of the embodiments of the present application is to provide a method, apparatus, electronic device, and storage medium for training a colonoscopy image evaluation model to solve the technical problem of inaccurate examination of bowel cleanliness and evaluation of colonoscopy quality in the related art.
[0005] In a first aspect, the embodiments of the present application provide a method for training a colonoscopy image evaluation model, including:
[0006] Obtain historical examination videos collected by colonoscopy and preprocess the historical examination videos to obtain historical colonoscopy images;
[0007] Perform annotation processing on the historical colonoscopy images based on pollutant categories to obtain a first set of colonoscopy images, and perform annotation processing on the historical colonoscopy images based on examination objects to obtain a second set of colonoscopy images, where the examination objects include adenomas and polyps;
[0008] Train a pollutant recognition model according to the first set of colonoscopy images to obtain a trained pollutant recognition model;
[0009] Train an examination object recognition model according to the second set of colonoscopy images to obtain a trained examination object recognition model;
[0010] Fuse the pollutant recognition model and the examination object recognition model, and train the fused model to obtain a colonoscopy image evaluation model.
[0011] In a second aspect, the embodiments of the present application provide a device for training a colonoscopy image evaluation model, including:
[0012] An image acquisition module, configured to acquire a historical examination video collected by a colonoscope, and preprocess the historical examination video to obtain a historical colonoscopy image;
[0013] An image processing module, configured to perform annotation processing on the historical colonoscopy image based on the pollutant category to obtain a first set of colonoscopy images, and perform annotation processing on the historical colonoscopy image based on the examination object to obtain a second set of colonoscopy images, where the examination objects include adenomas and polyps;
[0014] A first training module, configured to train a pollutant recognition model according to the first set of colonoscopy images to obtain a trained pollutant recognition model;
[0015] A second training module, configured to train an examination object recognition model according to the second set of colonoscopy images to obtain a trained examination object recognition model;
[0016] A fusion training module, configured to fuse the pollutant recognition model and the examination object recognition model, and train the fused model to obtain a colonoscopy image evaluation model.
[0017] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the training method of the colonoscopy image evaluation model described in any one of the above are implemented.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the training method of the colonoscopy image evaluation model described in any one of the above are implemented.
[0019] The embodiments of the present application provide a training method, device, electronic device and storage medium for a colonoscopy image evaluation model. By obtaining relevant historical data for training processing, a model for evaluating the quality of colonoscopy is obtained. Specifically, historical examination videos collected by colonoscopy are obtained, and historical colonoscopy images are obtained through corresponding preprocessing for model training. First, the historical colonoscopy images are respectively labeled based on pollutants and examination objects to obtain a first set of colonoscopy images and a second set of colonoscopy images. Then, a pollutant recognition model is trained based on the first set of colonoscopy images and an examination object recognition model is trained based on the second set of colonoscopy images. Next, the trained pollutant recognition model and examination object recognition model are fused, and finally, the fused model is trained and fine-tuned to obtain a colonoscopy image evaluation model. It realizes the training and optimization of the model by combining the cleanliness and clarity of colonoscopy images during the training process of the colonoscopy image evaluation model, so that when conducting quality evaluation, the colonoscopy images can be analyzed more accurately, the quality level of the colonoscopy images can be obtained, and the accuracy and reliability of the quality evaluation of colonoscopy images are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 FIG. is a flowchart of the steps of the training method for the colonoscopy image evaluation model provided by the embodiments of the present application;
[0021] Figure 2 FIG. is a flowchart of the steps of training the pollutant recognition model provided by the embodiments of the present application;
[0022] Figure 3 FIG. is a flowchart of the steps of training the examination object recognition model provided by the embodiments of the present application;
[0023] Figure 4 FIG. is another flowchart of the training method for the colonoscopy image evaluation model provided by the embodiments of the present application;
[0024] Figure 5 FIG. is a structural diagram of the training device for the colonoscopy image evaluation model provided by the embodiments of the present application;
[0025] Figure 6 FIG. is a structural diagram of the electronic device provided by the embodiments of the present application;
[0026] Figure 7 FIG. is another structural diagram of the electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0027] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope of protection of the present application.
[0028] It should be understood that the various steps recorded in the method embodiments disclosed in the present application can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope disclosed in the present application is not limited in this regard.
[0029] The term "including" and its variants used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0030] In the related art, the conventional method is to make a judgment based on an artificial subjective method. Specifically, by obtaining an image of the intestine, and then a doctor makes a manual view and judgment to determine the cleanliness of the intestine. However, this method requires the collected image to have relatively high quality, and at the same time, the judgment of cleanliness has strong human subjectivity, resulting in inaccurate judgment of the cleanliness of the intestine.
[0031] To solve the technical problems existing in the related art, the embodiments of the present application provide a training method for a colonoscopy image evaluation model. Please refer to Figure 1 , Figure 1 is a schematic flowchart of the steps of the training method for the colonoscopy image evaluation model provided by the embodiments of the present application. The method includes steps 101 to 105.
[0032] Step 101, obtain the historical examination video collected by the colonoscope, and preprocess the historical examination video to obtain historical colonoscopy images.
[0033] In one embodiment, when performing model training, obtain the data for model training. Specifically, obtain the historical examination video collected by the colonoscope, and preprocess the collected historical examination video to obtain historical colonoscopy images that can be used for training.
[0034] Exemplarily, when performing model training, the relevant historical data collected is used for evaluating the quality and cleanliness of colonoscopy images. Therefore, when obtaining the data for training, historical examination videos collected by each colonoscope can be obtained in the medical system as the basis for model training. At the same time, to improve the efficiency and accuracy of training, after obtaining the historical examination videos, corresponding preprocessing is performed on the historical examination videos to obtain historical colonoscopy images that can be used for model training.
[0035] In practical applications, the historical examination videos are colonoscopy videos collected during the examination process. At this time, by obtaining corresponding image frames in the historical examination videos and performing processing such as deduplication, cropping, and screening on the image frames, the data that can be used for training, that is, historical colonoscopy images, is obtained.
[0036] Furthermore, after obtaining the historical examination videos, when preprocessing the historical examination videos, it includes: obtaining image frames from the historical examination videos according to the set frame rate to obtain a number of image frames; performing image screening on the image frames to obtain a number of screened image frames; and performing image cropping processing on each of the screened image frames to obtain historical colonoscopy images.
[0037] Specifically, when performing colonoscopy quality assessment processing, it is an analysis and assessment of the collected colonoscopy images. Therefore, after obtaining the historical examination videos, images are obtained in the historical examination videos. By setting the corresponding frame rate, image frames are obtained in the historical examination videos to extract a number of image frames from the historical examination videos, and then the extracted image frames are processed such as deduplication, cropping, and screening to obtain historical colonoscopy images that can be used for model training.
[0038] In practical applications, when obtaining image frames from the historical examination videos, they are obtained through the set fixed frame rate, that is, setting the frame rate interval for obtaining image frames, and extracting image frames from the historical examination videos in an interval manner. For the extracted image frames, to ensure the efficiency and accuracy of training, etc., screening processing can be performed on the extracted image frames. Specifically, images with high similarity are screened and filtered to avoid multiple image frames with high similarity being used as training images. Among them, when performing screening processing on the extracted image frames, methods such as combining the average hash algorithm, perceptual hash algorithm, difference hash algorithm, and SIFT\GIST algorithm can be used to calculate the similarity of the images to obtain several image sets with high similarity, and then through image selection, the images used for model training in each similarity image set are obtained.
[0039] Exemplarily, when performing processing such as screening on the extracted image frames, it includes: calculating the similarity values between every two image frames in the image frames, and obtaining a similarity threshold; performing image screening processing on the image frames according to the similarity values and the similarity threshold to obtain several screened image frames; wherein, the several image frames include a first image and a second image, the first image is an image frame with a similarity value less than the similarity threshold, and the second image is an image frame obtained by screening from the image set composed of image frames with a similarity value greater than or equal to the similarity threshold.
[0040] When performing screening of image frames, by calculating the similarity values between every two image frames in all the image frames, and at the same time, a corresponding similarity threshold can also be set. Then, by comparing the calculated similarity values with the similarity threshold, the processing of the image frames is completed. Among them, for the image frames with a similarity value greater than or equal to the similarity threshold, they can form an image set, and then select an image frame from the formed image set as the representative image frame of the image set for model training. For the image frames without similar image frames, that is, the similarity values are less than the similarity threshold, they will be directly used as the image frames for training.
[0041] At the same time, for a formed image set, when selecting an image frame from it, the center point of the set can be found in the form of clustering, and then select the image frame closest to the center point to represent the image set as the image frame for model training.
[0042] Furthermore, after completing the screening of the extracted image frames, all the screened image frames will be subjected to image cropping processing to obtain historical colonoscopy images for training. Specifically, when performing cropping processing, mainly the useless information in the image frames is removed, such as colonoscopy device information, patient information, etc. By identifying the information in the image frames, the areas where the useless information is located are cropped.
[0043] Step 102, performing labeling processing on the historical colonoscopy images based on the pollutant category to obtain a first set of colonoscopy images, and performing labeling processing on the historical colonoscopy images based on the examination object to obtain a second set of colonoscopy images, where the examination objects include adenomas and polyps.
[0044] In one embodiment, after completing the processing of the historical examination video to obtain historical colonoscopy images, automatic labeling processing of the information of the historical colonoscopy images can be performed before training. Specifically, performing labeling processing on the historical colonoscopy images according to the pollutant category to obtain a first set of colonoscopy images, and performing labeling processing on the historical colonoscopy images based on the examination object to obtain a second set of colonoscopy images, where the examination object is the pathology or lesion that can be seen in colonoscopy examinations, such as adenomas and polyps.
[0045] Exemplarily, the evaluation of colonoscope quality includes the evaluation of the cleanliness and clarity of the colonoscope. Whether there are contaminants in the image affects the level of cleanliness, and whether adenomas and / or polyps can be completely shown represents the level of clarity. Therefore, before training, historical colonoscope images can be labeled respectively based on contaminants and examination objects to obtain corresponding first colonoscope image sets and second image sets, which are used to train the model to identify contaminants and examination objects respectively.
[0046] Step 103: Train the contaminant recognition model according to the first colonoscope image set to obtain a trained contaminant recognition model.
[0047] In one embodiment, after processing historical colonoscope images to obtain the first colonoscope image set, the constructed contaminant recognition model is trained using the obtained first colonoscope image set, and a trained contaminant recognition model is obtained when the training is completed. The contaminant recognition model is used to identify contaminants in colonoscope images, and the contaminants at least include: feces (defined as the area covered by formed solid contaminants), fecal water (defined as the area covered by semi-solid contaminants and the liquid area with color that makes the mucosa not clearly visible), foam (defined as the area with foam that makes the covered mucosa not clearly visible), and blood (defined as the area with blood that makes the covered mucosa not clearly visible).
[0048] Exemplarily, when training the contaminant recognition model using the processed first colonoscope image set, the feature information of the contaminants contained in each image in the first colonoscope image set is extracted to determine the feature information corresponding to each category of contaminants, and then the constructed contaminant recognition model is trained based on the feature information. When the training is completed, the contaminant recognition model identifies and extracts the contaminant features in the image. When there are contaminant features, it indicates the presence of contaminants, otherwise it is determined that there are no contaminants.
[0049] Further, referring to Figure 2 , Figure 2 is a schematic flowchart of a step for training the contaminant recognition model provided by an embodiment of the present application, where this step includes steps 201 to 204.
[0050] Step 201: Extract the features of the contaminants in each image of the first colonoscope image set to obtain the feature information of the contaminants contained in each image;
[0051] Step 202: Train the constructed contaminant recognition model according to the feature information to obtain an intermediate model;
[0052] Step 203: Obtain the initial quantization parameters for each pollutant in the pollutants, and adjust the intermediate model according to the initial quantization parameters;
[0053] Step 204: Optimize the quantization parameters of the adjusted intermediate model according to the first colonoscopy image set, and obtain a trained pollutant recognition model when the optimization is completed. The quantization parameters at least include: total pollution area, average pollution area, maximum pollution area, pollution frame frequency, pollution area duration, pollution area standard deviation, and pollution severity index.
[0054] Specifically, when training the pollutant recognition model using the first colonoscopy image set, identify the pollutants contained in each image in the first colonoscopy image set, extract the features of the pollutants in the image, obtain the feature information of the pollutants contained in each image, and then train the constructed pollutant recognition model according to the extracted feature information to obtain an intermediate model. The obtained intermediate model is used to segment and identify pollutants. Then, for the obtained intermediate model, adjust the corresponding quantization parameters so that the finally obtained pollutant recognition model can obtain a quantization result for pollutant recognition, such as a quantization score.
[0055] In order to enable the finally obtained colonoscopy image evaluation model to have a scoring ability, when performing segmentation and recognition processing on pollutants, the quantization parameters of the pollutant recognition model can be adjusted. Specifically, after obtaining the intermediate model, set the initial quantization parameters for the intermediate model, and then use the first colonoscopy image set to optimize the quantization parameters of the adjusted intermediate model again. That is, while ensuring that the pollutant recognition model has the ability to recognize pollutants, optimize the set initial quantization parameters. The quantization parameters to be adjusted at least include: total pollution area, average pollution area, maximum pollution area, pollution frame frequency, pollution area duration, pollution area standard deviation, and pollution severity index.
[0056] Furthermore, when adjusting and optimizing the quantization parameters, verify the correlation of the quantization parameters. Specifically, by constructing a quantization parameter correlation test module, and then after obtaining the case parameters using the pollutant segmentation - related parameter quantization calculation model, input them into the correlation test module. The module takes whether the case detects adenomas / polyps, the overall polyp / adenoma detection rate in the cohort, and the overall polyp / adenoma detection number rate in the cohort as the outcomes, and respectively conducts a normality test, a chi - square test, logistic regression, and ROC curve plotting for each parameter by taking step thresholds. Among them, the polyp / adenoma detection rate=(the number of patients who detect polyps / adenomas in the cohort) / the total number of patients in the cohort, and the overall polyp / adenoma detection number rate=(the total number of polyps / adenomas detected in the cohort) / the total number of patients in the cohort.
[0057] Step 104: Train the examination object recognition model according to the second colonoscopy image set to obtain a trained examination object recognition model.
[0058] In one embodiment, after obtaining the second colonoscopy image set, use the second image set to train the examination object recognition model to obtain a trained examination object recognition model for recognizing the examination objects included in the colonoscopy images, where the examination objects at least include adenomas and polyps.
[0059] Exemplarily, the colonoscopy image evaluation model for performing colonoscopy quality assessment processing needs to accurately recognize pollutants and examination objects. Therefore, when training the colonoscopy image evaluation model for pollutant recognition and quantification processing, it is also necessary to train the ability to recognize examination objects. Specifically, referring to Figure 3 , Figure 3 FIG. is a schematic flowchart of a step for training the examination object recognition model provided in an embodiment of the present application, where this step includes steps 301 to 303.
[0060] Step 301: Input each image in the second colonoscopy image set into the trained pollutant recognition model to obtain the object recognition result of each image.
[0061] Step 302: Perform annotation processing on the corresponding images in the second colonoscopy image set according to the object recognition results to obtain a third image set.
[0062] Step 303: Train the constructed examination object recognition model according to the third image set, and obtain a trained examination object recognition model when the training is completed.
[0063] Specifically, when training the constructed examination object recognition model, input each image in the second colonoscopy image set into the trained pollutant recognition model to obtain the object recognition result of each image in the second colonoscopy image set, and then perform annotation processing on the corresponding images in the second colonoscopy image set according to the obtained object recognition results to obtain a third image set. Furthermore, train the constructed examination object recognition model according to the obtained third image set, and obtain a trained examination object recognition model when the training is completed.
[0064] Exemplarily, when training the second colonoscopy image set, first, based on the trained pollutant recognition model, determine the recognition results of the pollutants in each image of the second colonoscopy image set, and perform annotation processing on each image in the second colonoscopy image set according to the obtained recognition results to obtain a second image combination annotated with pollutant information, that is, the third image set. For each image in the second colonoscopy image set, the examination object may not be recognized due to the presence of pollutants. Therefore, through the processing of the trained pollutant recognition model, re-annotate the second colonoscopy image set, and then train the examination object recognition model so that the examination object recognition model can accurately recognize the examination object in the presence of pollutants.
[0065] Step 105: Fuse the pollutant recognition model and the examination object recognition model, and train the fused model to obtain a colonoscopy image evaluation model.
[0066] In one embodiment, after obtaining the trained pollutant recognition model and examination object recognition model, fuse and train the two models to obtain a colonoscopy image evaluation model.
[0067] Exemplarily, for a pollutant recognition model that can identify and quantify pollutants and an examination object recognition model that can identify an examination object, it is necessary to fuse them into a single model for quality assessment processing of colonoscopy images. At this time, through model fusion processing and fine-tuning and other training of the fused model, a colonoscopy image evaluation model for quality assessment can be obtained.
[0068] Among them, when performing model fusion processing, the fusion method used is not limited. At the same time, when performing fine-tuning processing, a certain number of verification images can be obtained from historical colonoscopy images as data for fine-tuning and verification. Use some images to fine-tune the fused model and the remaining images to verify the fine-tuned model. Then, when the verification passes, a trained colonoscopy image evaluation model can be obtained, which can effectively improve the accuracy and training efficiency of the model.
[0069] Further, after the colonoscopy image evaluation model is trained and used, by analyzing and processing the videos / images collected by the colonoscope, the current colonoscopy image evaluation result can be obtained, specifically an evaluation score, to determine whether a reminder and re-cleaning process are needed based on the obtained evaluation score. Specifically, when the colonoscopy image evaluation model is used, it includes: obtaining the colonoscopy images collected by the colonoscope; inputting the colonoscopy images into the pre-trained colonoscopy image evaluation model, and outputting the image evaluation result corresponding to the colonoscopy images; obtaining the preset image evaluation criteria, and generating the evaluation display information corresponding to the colonoscopy images based on the image evaluation result, the image evaluation criteria, and the colonoscopy images.
[0070] Exemplarily, for the collected colonoscopy images, input them into the aforementioned trained colonoscopy image evaluation model. By identifying and processing the pollutants and examination objects in the colonoscopy images, the quantitative result corresponding to the colonoscopy images, that is, the image evaluation result, is obtained. Then, the corresponding evaluation display information is generated based on the obtained image evaluation result, the image evaluation criteria, and the colonoscopy images. Specifically, when judging according to the image evaluation result and the image evaluation criteria, for example, if the obtained evaluation score is lower than the score threshold, it is determined that a cleaning process is needed. Otherwise, it means that the cleaning quality is good at this time. At the same time, for the judgment result, it can be displayed together with the other collected colonoscopy images.
[0071] At the same time, for the convenience of timely processing, reminder processing needs to be carried out in a timely manner at an appropriate time. Specifically, when comparing the image evaluation result with the image evaluation criteria, if the evaluation value in the image evaluation result is less than the standard value in the image evaluation criteria, corresponding prompt information can be generated.
[0072] Further, referring to Figure 4 , Figure 4 is another flow chart of the training method of the colonoscopy image evaluation model provided by the embodiment of the present application, where this step includes step S1 to step S4.
[0073] Step S1, data collection and preprocessing.
[0074] Automatically collect colonoscopy examination videos through the data collection port, and perform necessary preprocessing on the videos to obtain the required pictures.
[0075] Specifically, when collecting data, the collected video data is diverse. In addition to colonoscopy video data, it can also include video data collected by other examination devices. Therefore, when collecting data, the collected video data can be screened according to information such as the instrument model and examination category used to obtain the corresponding scenario video data, and corresponding preprocessing is performed on this data.
[0076] Among them, during preprocessing, it includes:
[0077] (1) Extract picture frames from the colonoscopy images according to a fixed frame rate;
[0078] (2) Use a picture similarity evaluation module that combines the average hashing algorithm, perceptual hashing algorithm, difference hashing algorithm, and SIFT\GIST algorithms to evaluate the picture similarity, and remove the similar pictures obtained in the previous step according to an adjustable specific threshold;
[0079] (3) Identify the pictures with a clean digestive tract cavity in the pictures obtained in the previous step according to a previously trained picture cleanliness model;
[0080] (4) Crop the edges of the pictures obtained in the previous step and irrelevant endoscopic picture information.
[0081] Step S2: Construct a quantitative evaluation database for colonoscopy cleanliness by classifying and annotating the collected data.
[0082] Among them, during classification and annotation processing, it includes:
[0083] (1) Continuously collect colonoscopy videos from the previous data collection process, and obtain a corresponding picture library of colonoscopy with contaminants after preprocessing;
[0084] (2) Respectively segment and annotate the classification of contaminants in the pictures in the picture library of colonoscopy with contaminants according to the examples of contaminants in the digestive endoscopy diagnosis and treatment specifications. Its classification mainly includes: feces (defined as the area covered by formed solid contaminants), fecal water (defined as the area covered by semi-solid substances with a certain degree of pollution and the liquid area with color that makes the mucosa not clearly visible), foam (defined as the area with foam that makes the covered mucosa not clearly visible), blood (defined as the area with blood that makes the covered mucosa not clearly visible);
[0085] (3) Review the examination videos and report the detection situation of adenomas / polyps for each case. The report content includes but is not limited to: the number of adenomas / polyps, the estimated size of adenomas / polyps, and the corresponding location. And adenomas and polyps are the examination objects during colonoscopy;
[0086] (4) Integrate the picture library of colonoscopy with contaminants obtained from the above colonoscopy examination cases and the detection situation of adenomas / polyps for each case to form a quantitative evaluation database for colonoscopy cleanliness.
[0087] Step S3: Use the quantitative evaluation database for colonoscopy cleanliness to train a quantitative evaluation model for colonoscopy cleanliness.
[0088] (1) Use the colonoscopy examination cleanliness quantification evaluation database to train the corresponding pollutant segmentation models for four types of pollutants respectively with the colonoscopy pictures with segmentation annotations in the pollutant inspection picture library.
[0089] (2) Construct a quantification calculation model for the relevant parameters of pollutant segmentation, which can automatically calculate the corresponding quantification parameters for each pollutant, including but not limited to the total pollution area, average pollution area, maximum pollution area, pollution frame frequency, pollution area duration, pollution area standard deviation, pollution severity index, etc. of the case.
[0090] (3) Construct a correlation test module for quantification parameters. After obtaining the case parameters through the quantification calculation model for the relevant parameters of pollutant segmentation, input them into the correlation test module. The module takes whether adenomas / polyps are detected in the case, the overall polyp / adenoma detection rate in the cohort, and the overall polyp / adenoma detection number rate in the cohort as the outcomes respectively, and conducts normality tests, chi-square tests, logistic regression, and ROC curve plotting for each parameter by taking step thresholds.
[0091] Among them, the polyp / adenoma detection rate = (the number of patients with polyps / adenomas detected in the cohort) / the total number of patients in the cohort, and the overall polyp / adenoma detection number rate in the cohort = (the total number of polyps / adenomas detected in the cohort) / the total number of patients in the cohort.
[0092] (4) Select practical parameters according to the statistics corresponding to each parameter threshold, and the conditions are:
[0093] ① Under the normality test, the parameter distribution conforms to the normal distribution.
[0094] ② Under the chi-square test, there are significant differences in the detection outcome statistics between the group below the set threshold and the group above the threshold.
[0095] ③ The logistic regression model added with the corresponding parameter has a good fit.
[0096] ④ The area under the ROC curve of the corresponding parameter > 0.5.
[0097] Among them, condition ① must be met, and at least one of conditions ②, ③, and ④ must be met. Then select this parameter, and take the value of the step threshold with the best significance of its test statistic as the threshold of this parameter.
[0098] Step S4, construct and integrate modules such as data collection, preprocessing, colonoscopy cleanliness quantification model, scoring, and reminder to form a colonoscopy examination cleanliness quantification evaluation method and system.
[0099] (1) Construct a scoring and reminder module, which reminds the doctor to perform flushing or terminate the examination in time by displaying the cumulative situation of the current cleanliness quantification parameters and the boundary from the threshold in real time during the colonoscopy examination, as Figure 4 shown.
[0100] (2) Integrate the data collection, preprocessing, colonoscopy cleanliness quantification model, scoring and reminder modules, so that the entire system can regularly collect data automatically for the update of the quantification model evaluation;
[0101] (3) Construct a cut-off and adjustment module to visualize the evaluation parameters and thresholds of the quantification model and allow manual adjustment.
[0102] Furthermore, after completing the training and optimization of the colonoscopy image evaluation model, during use, accurate and reliable analysis and processing can be performed based on the collected colonoscopy images to obtain the current image evaluation result of the colonoscopy.
[0103] In summary, the present application discloses a training method for a colonoscopy image evaluation model. By obtaining relevant historical data for training and processing, a model for evaluating the quality of colonoscopies is obtained. Specifically, historical examination videos collected by colonoscopies are obtained, and historical colonoscopy images are obtained through corresponding preprocessing for model training. First, the historical colonoscopy images are respectively labeled based on pollutants and examination objects to obtain a first set of colonoscopy images and a second set of colonoscopy images. Then, a pollutant recognition model is trained based on the first set of colonoscopy images and an examination object recognition model is trained based on the second set of colonoscopy images. Next, the trained pollutant recognition model and examination object recognition model are fused, and finally, the fused model is trained and fine-tuned to obtain a colonoscopy image evaluation model. It realizes the training and optimization of the model by combining the cleanliness and clarity of the colonoscopy during the training process of the colonoscopy image evaluation model, so that during quality evaluation, the colonoscopy images can be analyzed more accurately to obtain the quality level of the colonoscopy images, improving the accuracy and reliability of colonoscopy quality evaluation.
[0104] According to the method described in the above embodiments, this embodiment will be further described from the perspective of the training device of the colonoscopy image evaluation model. The training device of the colonoscopy image evaluation model can be specifically implemented as an independent entity or integrated in an electronic device, such as a terminal. The terminal can include a mobile phone, a tablet computer, etc.
[0105] Please refer to Figure 5 , Figure 5 which is a schematic structural diagram of the training device of the colonoscopy image evaluation model provided by the embodiments of the present application. As Figure 5 shown, the training device 500 of the colonoscopy image evaluation model provided by the embodiments of the present application includes:
[0106] An image acquisition module 501, configured to acquire historical examination videos collected by colonoscopies and preprocess the historical examination videos to obtain historical colonoscopy images;
[0107] The image processing module 502 is configured to perform annotation processing on historical colonoscopy images based on pollutant categories to obtain a first set of colonoscopy images, and perform annotation processing on historical colonoscopy images based on examination objects, where the examination objects include adenomas and polyps, to obtain a second set of colonoscopy images.
[0108] The first training module 503 is configured to train a pollutant recognition model according to the first set of colonoscopy images to obtain a trained pollutant recognition model.
[0109] The second training module 504 is configured to train an examination object recognition model according to the second set of colonoscopy images to obtain a trained examination object recognition model.
[0110] The fusion training module 505 is configured to fuse the pollutant recognition model and the examination object recognition model, and train the fused model to obtain a colonoscopy image evaluation model.
[0111] In one embodiment, the image acquisition module 501 is further configured to:
[0112] Obtain a plurality of image frames by acquiring image frames of a historical examination video at the set frame rate.
[0113] Perform image screening on the image frames to obtain a plurality of screened image frames.
[0114] Perform image cropping processing on each of the plurality of screened image frames to obtain historical colonoscopy images.
[0115] In one embodiment, the image acquisition module 501 is further configured to:
[0116] Calculate the similarity value between every two image frames in the image frames, and obtain a similarity threshold.
[0117] Perform image screening processing on the image frames according to the similarity value and the similarity threshold to obtain a plurality of screened image frames.
[0118] Among them, the plurality of image frames include a first image and a second image. The first image is an image frame with a similarity value less than the similarity threshold, and the second image is an image frame selected from the image set composed of image frames with a similarity value greater than or equal to the similarity threshold.
[0119] In one embodiment, the first training module 503 is further configured to:
[0120] Extract features of pollutants in each image of the first set of colonoscopy images to obtain the feature information of the pollutants contained in each image.
[0121] Train the constructed pollutant recognition model according to the feature information to obtain an intermediate model.
[0122] Obtain the initial quantization parameters for each pollutant in the pollutants, and adjust the intermediate model according to the initial quantization parameters;
[0123] Optimize the quantization parameters of the adjusted intermediate model according to the first set of colonoscopy images, and obtain a trained pollutant recognition model when the optimization is completed, where the quantization parameters at least include: total pollution area, average pollution area, maximum pollution area, pollution frame frequency, pollution area duration, pollution area standard deviation, and pollution severity index.
[0124] In one embodiment, the first training module 504 is further configured to:
[0125] Input each image in the second set of colonoscopy images into the trained pollutant recognition model to obtain the object recognition result of each image;
[0126] Perform annotation processing on the corresponding images in the second set of colonoscopy images according to the object recognition results to obtain a third set of images;
[0127] Train the constructed examination object recognition model according to the third set of images, and obtain a trained examination object recognition model when the training is completed.
[0128] In one embodiment, the training device 500 of the colonoscopy image evaluation model further includes a quality evaluation module for:
[0129] Obtain the colonoscopy images collected by the colonoscope;
[0130] Input the colonoscopy images into a pre-trained colonoscopy image evaluation model, and output the image evaluation result corresponding to the colonoscopy images;
[0131] Obtain the image evaluation criteria, and generate evaluation display information corresponding to the colonoscopy images according to the image evaluation results, the image evaluation criteria, and the colonoscopy images.
[0132] In one embodiment, the training device 500 of the colonoscopy image evaluation model further includes a prompt feedback module for:
[0133] Compare the evaluation score in the image evaluation result with the standard score in the image evaluation criteria, and generate corresponding prompt information when the evaluation score is less than the standard score.
[0134] In addition, please refer to Figure 6 , Figure 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may be a mobile terminal such as a smart phone, a tablet computer, etc. As Figure 6As shown, the electronic device 600 includes a processor 601 and a memory 602. Among them, the processor 601 is electrically connected to the memory 602.
[0135] The processor 601 is the control center of the electronic device 600, connecting various parts of the entire electronic device through various interfaces and circuits. By running or loading application programs stored in the memory 602, and invoking data stored in the memory 602, it executes various functions of the electronic device 600 and processes data, thereby monitoring the entire electronic device 600.
[0136] In this embodiment, the processor 601 in the electronic device 600 will load the instructions corresponding to the processes of one or more application programs into the memory 602 according to the following steps, and the processor 601 will run the application programs stored in the memory 602, thereby implementing any step in the training method of the colonoscopy image evaluation model provided in the above embodiment.
[0137] The electronic device 600 can implement the steps in any embodiment of the training method of the colonoscopy image evaluation model provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that any training method of the colonoscopy image evaluation model provided in the embodiments of the present application can achieve. For details, please refer to the previous embodiments and will not be repeated here.
[0138] Please refer to Figure 7 , Figure 7 which is another structural schematic diagram of the electronic device provided in the embodiments of the present application. As Figure 7 shown, Figure 7 shows the specific structural block diagram of the electronic device provided in the embodiments of the present application. The electronic device 700 can be used to implement the training method of the colonoscopy image evaluation model provided in the above embodiments. The electronic device 700 can be a mobile terminal such as a smart phone or a laptop computer, etc.
[0139] The RF circuit 710 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices. The RF circuit 710 may include various existing circuit elements for performing these functions. For example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a subscriber identity module (SIM) card, a memory, and so on. The RF circuit 710 can communicate with various networks such as the Internet, an enterprise intranet, a wireless network or communicate with other devices through a wireless network. The above-mentioned wireless network may include a cellular phone network, a wireless local area network or a metropolitan area network. The above-mentioned wireless network can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, and may even include those protocols that have not been developed yet.
[0140] The memory 720 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method of the colonoscopy image evaluation model in the above embodiments. The processor 780 executes various functional applications and the training method of the colonoscopy image evaluation model by running the software programs and modules stored in the memory 720.
[0141] The memory 720 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some examples, the memory 720 may further include a memory remotely located with respect to the processor 780, and these remote memories may be connected to the electronic device 700 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0142] The input unit 730 can be used to receive uploaded digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls. Specifically, the input unit 730 may include a touch-sensitive surface 731 and other input devices 732. The touch-sensitive surface 731, also known as a touch display screen or a touchpad, can collect touch operations of a user on or near it (such as operations of the user using a finger, a stylus, or any suitable object or accessory on or near the touch-sensitive surface 731), and drive corresponding connection devices according to a preset program. Optionally, the touch-sensitive surface 731 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch position of the user, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 780, and can also receive commands sent by the processor 780 and execute them. In addition, various types such as resistive, capacitive, infrared, and surface acoustic wave can be used to implement the touch-sensitive surface 731. In addition to the touch-sensitive surface 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, power switch keys, etc.), a trackball, a mouse, a joystick, etc.
[0143] The display unit 740 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device 700. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 740 may include a display panel 741. Optionally, the display panel 741 can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 731 can cover the display panel 741. When the touch-sensitive surface 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides a corresponding visual output on the display panel 741 according to the type of touch event. Although in the figure, the touch-sensitive surface 731 and the display panel 741 are implemented as two independent components to achieve input and output functions, in some embodiments, the touch-sensitive surface 731 and the display panel 741 can be integrated to achieve input and output functions.
[0144] The electronic device 700 may further include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. Among them, the ambient light sensor can adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor can generate an interruption when the flip cover is closed or opened. As a kind of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. As for other sensors that the electronic device 700 can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be elaborated here.
[0145] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the electronic device 700. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, and the speaker 761 converts it into a sound signal for output. On the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data. Then, after the audio data is output to the processor 780 for processing, it is sent to another terminal, for example, via the RF circuit 710, or the audio data is output to the memory 720 for further processing. The audio circuit 760 may also include an earphone jack to provide communication between the external earphone and the electronic device 700.
[0146] The electronic device 700 can help users receive requests, send information, etc. through a transmission module 770 (such as a Wi-Fi module), which provides users with wireless broadband Internet access. Although the transmission module 770 is shown in the figure, it can be understood that it does not belong to the essential components of the electronic device 700 and can be omitted completely within the scope of not changing the essence of the invention according to needs.
[0147] The processor 780 is the control center of the electronic device 700, connecting various parts of the entire mobile phone through various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 720, and calling the data stored in the memory 720, it executes various functions of the electronic device 700 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 780 may include one or more processing cores; in some embodiments, the processor 780 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 780 either.
[0148] The electronic device 700 further includes a power supply 790 (such as a battery) for powering each component. In some embodiments, the power supply can be logically connected to the processor 780 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 790 may also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.
[0149] Although not shown, the electronic device 700 further includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be elaborated here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal further includes a memory, and one or more programs, where one or more programs are stored in the memory and are configured to be executed by one or more processors to implement any step in the training method of the colonoscopy image evaluation model provided in the above embodiment.
[0150] In specific implementation, the above-mentioned each module can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above-mentioned each module, reference can be made to the method embodiment above, which will not be elaborated here.
[0151] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions or by controlling relevant hardware through instructions. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. For this purpose, an embodiment of the present application provides a storage medium in which multiple instructions are stored. When these instructions are executed by a processor, any step in the training method of the colonoscopy image evaluation model provided in the above embodiments can be implemented.
[0152] Among them, the storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.
[0153] Since the instructions stored in the storage medium can execute the steps in any embodiment of the training method of the colonoscopy image evaluation model provided in the embodiments of the present application, the beneficial effects achievable by any training method of the colonoscopy image evaluation model provided in the embodiments of the present application can be realized. For details, please refer to the previous embodiments and will not be elaborated here.
[0154] The above has introduced in detail a training method, device, electronic device, and storage medium of a colonoscopy image evaluation model provided in the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application. Moreover, for those of ordinary skill in the technical field, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements are also regarded as the protection scope of the present application.
Claims
1. A method for training a colonoscopy image evaluation model, characterized in that: include: Acquire historical examination videos collected by colonoscopy, and preprocess the historical examination videos to obtain historical colonoscopy images; The historical colonoscopy images are annotated based on the pollutant category to obtain a first colonoscopy image set, and the historical colonoscopy images are annotated based on the inspection object to obtain a second colonoscopy image set, wherein the inspection object includes adenoma and polyp; Training a pollutant recognition model according to the first colonoscopy image set to obtain a trained pollutant recognition model; training the inspection object recognition model according to the second colonoscopy image set to obtain a trained inspection object recognition model; The pollutant identification model and the inspection object identification model are fused, and the fused model is trained to obtain a colonoscopy image evaluation model.
2. The method according to claim 1, characterized in that The preprocessing of the historical examination video to obtain the historical colonoscopy image includes: Acquiring image frames of the historical inspection video according to the set frame rate to obtain a plurality of image frames; Performing image screening on the image frames to obtain a plurality of screened image frames; Image cropping is performed on each of the screened image frames to obtain a historical colonoscopy image.
3. The method according to claim 2, characterized in that The step of performing image screening on the image frame to obtain a plurality of screened image frames comprises: Calculating the similarity value between every two image frames in the image frames, and obtaining a similarity threshold; Performing image screening processing on the image frame according to the similarity value and the similarity threshold to obtain a plurality of screened image frames; The plurality of image frames include a first image and a second image, the first image being an image frame whose similarity value is less than the similarity threshold, and the second image being an image frame obtained by screening an image set consisting of images whose similarity value is greater than or equal to the similarity threshold.
4. The method according to claim 1, characterized in that The step of training the pollutant recognition model according to the first colonoscopy image set to obtain a trained pollutant recognition model includes: Extracting features of pollutants in each image in the first colonoscopy image set to obtain feature information of pollutants contained in each image; Training the constructed pollutant identification model according to the characteristic information to obtain an intermediate model; Obtaining an initial quantitative parameter of each of the pollutants, and adjusting the intermediate model according to the initial quantitative parameter; The quantization parameters of the adjusted intermediate model are optimized according to the first colonoscopy image set, and a trained pollutant recognition model is obtained when the optimization is completed, wherein the quantization parameters include at least: the total contaminated area, the mean contaminated area, the maximum contaminated area, the contaminated frame frequency, the contaminated area duration, the contaminated area standard deviation and the pollution severity index.
5. The method according to claim 4, characterized in that The step of training the inspection object recognition model according to the second colonoscopy image set to obtain a trained inspection object recognition model includes: Inputting each image in the second colonoscopy image set into the trained contaminant recognition model to obtain an object recognition result for each image; Annotate the corresponding images in the second colonoscopy image set according to the object recognition result to obtain a third image set; The constructed inspection object recognition model is trained according to the third image set, and a trained inspection object recognition model is obtained when the training is completed.
6. The method according to claim 1, characterized in that The method further comprises: Obtaining colonoscopic images collected by colonoscopy; Inputting the colonoscopic image into a pre-trained colonoscopic image evaluation model, and outputting an image evaluation result corresponding to the colonoscopic image; A preset image evaluation standard is obtained, and evaluation display information corresponding to the colonoscopic image is generated according to the image evaluation result, the image evaluation standard and the colonoscopic image.
7. The method according to claim 6, characterized in that After obtaining the preset image evaluation standard, the method further includes: The evaluation score in the image evaluation result is compared with the standard score in the image evaluation standard, and when the evaluation score is smaller than the standard score, corresponding prompt information is generated.
8. A training device for a colonoscopy image evaluation model, characterized in that: include: An image acquisition module is used to acquire historical examination videos collected by colonoscopy, and pre-process the historical examination videos to obtain historical colonoscopy images; An image processing module, configured to label the historical colonoscopic images based on pollutant categories to obtain a first colonoscopic image set, and label the historical colonoscopic images based on inspection objects to obtain a second colonoscopic image set, wherein the inspection objects include adenomas and polyps; A first training module, used to train a pollutant recognition model according to the first colonoscopy image set to obtain a trained pollutant recognition model; a second training module, configured to train the inspection object recognition model according to the second colonoscopy image set to obtain a trained inspection object recognition model; The fusion training module is used to fuse the pollutant recognition model and the inspection object recognition model, and train the fused model to obtain a colonoscopy image evaluation model.
9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.