Enteroscope image evaluation model training method and device, equipment and storage medium

Abstract

The invention discloses a training method and device of an enteroscope image evaluation model, electronic equipment and a storage medium, and the method comprises the steps: obtaining a historical examination video collected by an enteroscope, and carrying out the preprocessing of the historical examination video, and obtaining a historical enteroscope image; training the constructed image filtering model based on the historical enteroscopy image, and performing image filtering processing on the historical enteroscopy image by using the trained image filtering model to obtain a qualified enteroscopy image set; performing boundary labeling on each image in the qualified enteroscopy image set based on intestinal mucosa, and training the constructed image segmentation model by using the labeled qualified enteroscopy image set to obtain a trained image segmentation model; and carrying out model fine tuning processing on the trained image filtering model and the trained image segmentation model, and obtaining an enteroscope image evaluation model when fine tuning processing is completed. And the quality and the accuracy of colonoscopy are improved.

Description

Technical Field

[0001] The present application relates to the field of medical auxiliary technology, and in particular to a training method, device, electronic device and storage medium for a colonoscopy image evaluation model. Background Art

[0002] Colonoscopy is widely used in the screening of colorectal cancer and precancerous lesions, and has become an effective standard for screening and detecting colorectal neoplastic lesions. Endoscopic resection of early lesions can increase the detection rate of early colorectal cancer and thereby reduce the mortality rate of colorectal cancer.

[0003] At present, colonoscopy is usually based on the doctor's operating experience. For example, when an inexperienced doctor performs a colonoscopy, the intestinal mucosa is not fully exposed, resulting in low quality of the colonoscopy. Low-quality colonoscopy can lead to missed diagnosis of lesions, making it impossible to guarantee the accuracy of the examination results. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a training method, device, electronic device and storage medium for a colonoscopy image evaluation model to solve the technical problem of inaccurate colonoscopy examination quality and evaluation in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a method for training a colonoscopy image evaluation model, comprising:

[0006] Obtain historical examination videos collected by colonoscopy, and pre-process the historical examination videos to obtain historical colonoscopy images;

[0007] Based on the historical colonoscopy images, the constructed image filtering model is trained, and the historical colonoscopy images are subjected to image filtering processing using the trained image filtering model to obtain a set of qualified colonoscopy images;

[0008] Annotating the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and using the annotated qualified colonoscopy image set to train the constructed image segmentation model to obtain a trained image segmentation model;

[0009] The trained image filtering model and the trained image segmentation model are fine-tuned, and a colonoscopy image evaluation model is obtained after the fine-tuning is completed.

[0010] In a second aspect, an embodiment of the present application provides a training device for a colonoscopy image evaluation model, comprising:

[0011] A data acquisition module is used to acquire historical examination videos collected by colonoscopy and pre-process the historical examination videos to obtain historical colonoscopy images;

[0012] A first training module is used to train the constructed image filtering model based on the historical colonoscopy images, and use the trained image filtering model to perform image filtering processing on the historical colonoscopy images to obtain a set of qualified colonoscopy images;

[0013] A second training module is used to annotate the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and train the constructed image segmentation model using the annotated qualified colonoscopy image set to obtain a trained image segmentation model;

[0014] The fine-tuning processing module is used to perform model fine-tuning processing on the trained image filtering model and the trained image segmentation model, and obtain the colonoscopy image evaluation model after the fine-tuning processing is completed.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the training method for the colonoscopy image evaluation model described in any one of the above when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in the training method of the colonoscopy image evaluation model described in any one of the above.

[0017] The embodiment of the present application provides a method, device, electronic device and storage medium for training a colonoscopy image evaluation model. The method obtains relevant historical data for training and processing, and obtains a model for quality evaluation of colonoscopy images. Specifically, historical inspection videos collected by colonoscopy are obtained, and historical colonoscopy images are obtained through corresponding preprocessing for model training data. First, the image filtering model is trained using historical colonoscopy images, and the trained image filtering model is used to filter the historical colonoscopy images to obtain a qualified colonoscopy image set that can be used for intestinal mucosal visualization evaluation. Then, the qualified colonoscopy image set is used to train the image segmentation model, and finally the trained image filtering model and image segmentation model are fine-tuned to obtain a colonoscopy image evaluation model. The colonoscopy image evaluation model is trained based on the image's eligibility features and boundary features, and then the colonoscopy video and image of the colonoscopy are automatically filtered and screened, and the intestinal mucosal visualization degree of the qualified colonoscopy images is evaluated to obtain an accurate proportion of the intestinal mucosal visualization degree, etc., thereby improving the quality and accuracy of colonoscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1It is a flowchart of the steps of the method for training a colonoscopy image evaluation model provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of the steps of training an image filtering model and obtaining a qualified colonoscopy image set provided in an embodiment of the present application;

[0020] Figure 3 It is a flowchart of the steps of obtaining an image segmentation model provided in an embodiment of the present application;

[0021] Figure 4 This is a flow chart of the steps of obtaining a colonoscopy image evaluation model provided in an embodiment of the present application;

[0022] Figure 5 It is a structural schematic diagram of a training device for a colonoscopy image evaluation model provided in an embodiment of the present application;

[0023] Figure 6 is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0024] Figure 7 This is another structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be understood that the various steps described in the method embodiments disclosed in the present application can be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] In related technologies, colonoscopy is usually performed based on the doctor's operating experience. For example, when an inexperienced doctor performs a colonoscopy, the intestinal mucosa is not fully exposed, resulting in low quality of the colonoscopy. Low-quality colonoscopy can lead to missed diagnosis of lesions, making it impossible to guarantee the accuracy of the examination results.

[0029] In order to solve the technical problems existing in the related art, the present application embodiment provides a training method for a colonoscopy image evaluation model, see Figure 1 , Figure 1 It is a flowchart diagram of the steps of the method for training a colonoscopy image evaluation model provided in an embodiment of the present application, and the method includes steps 101 to 104.

[0030] Step 101, obtain historical examination videos collected by colonoscopy, and pre-process the historical examination videos to obtain historical colonoscopy images.

[0031] In one embodiment, when performing model training, data for model training is obtained, specifically, historical examination videos collected by colonoscopy are obtained, and the collected historical examination videos are preprocessed to obtain historical colonoscopy images that can be used for training.

[0032] Exemplarily, when conducting model training, for the acquired training data, relevant historical information can be obtained in the medical system, such as the inspection video collected during the colonoscopy, as the basis for model training. At the same time, in order to improve the efficiency and accuracy of training, the historical inspection video is preprocessed accordingly after obtaining it, so as to obtain the historical colonoscopy images used for model training.

[0033] In practical applications, historical inspection videos are colonoscopy videos collected during the colonoscopy process. At this time, corresponding image frames are obtained from the historical inspection videos, and the image frames are deduplicated, cropped, and screened to obtain data that can be used for training, namely historical colonoscopy images.

[0034] Furthermore, after acquiring the historical inspection video, when preprocessing the historical inspection video, it includes: extracting image frames of the historical inspection video according to the set frame rate to obtain a number of image frames; filtering the image frames based on the time series relationship to obtain a number of filtered image frames; and cropping each of the filtered image frames to obtain a historical colonoscopy image.

[0035] Specifically, when performing colonoscopy quality assessment processing, analysis and evaluation processing is performed on the collected colonoscopy images. Therefore, after obtaining the historical inspection video, images are acquired in the historical inspection video. The corresponding sampling frame rate can be set to extract image frames in the historical inspection video according to the sampling frame rate to obtain corresponding image frames. Then, the extracted image frames can be pre-processed such as deduplication, cropping and screening to obtain historical colonoscopy images that can be used for model training.

[0036] In practical applications, when obtaining image frames in historical inspection videos, they are obtained at a set fixed frame rate, such as extracting 10 frames per second, and extracting image frames at a set fixed frame rate. For the extracted image frames, in order to avoid invalid repeated training, the extracted image frames can be filtered, such as screening images with high similarity to avoid multiple image frames with high similarity being used as training images. Among them, when screening the extracted image frames, the image similarity can be calculated by combining the average hash algorithm, the perceptual hash algorithm, the difference hash algorithm, and the SIFT\GIST algorithm to obtain several image sets with high similarity, and then through the selection of images, the images used for model training in each similarity image set are obtained.

[0037] Exemplarily, when the extracted image frames are screened and processed, it includes: calculating the image similarity value between two adjacent image frames in the image frames based on the time series relationship, and obtaining a pre-set similarity threshold; performing image screening processing on the image frames according to the image similarity value and the similarity threshold, and obtaining a number of screened image frames; wherein the number of image frames include a first image and a second image, the first image is an image frame whose similarity value is less than the similarity threshold, and the second image is an image frame obtained by screening an image set consisting of images whose similarity value is greater than or equal to the similarity threshold.

[0038] When screening image frames, based on the temporal relationship of each video frame in the historical inspection video, the similarity value between two adjacent image frames in the image frame is calculated, and the corresponding similarity threshold can also be set, and then the image frame screening process is completed by comparing the calculated similarity value with the similarity threshold. Among them, for image frames with similarity values ​​greater than or equal to the similarity threshold, an image set can be formed, and then an image frame is selected from the formed image set as a representative image frame of the image set for model training, and for image frames without similar image frames, that is, the similarity value is less than the similarity threshold, it will be directly used as an image frame for training.

[0039] At the same time, for an image set, when selecting one of the image frames, the center point of the set can be found by clustering, and then the image frame closest to the center point is selected to represent the image set as the image frame for model training.

[0040] Furthermore, after the extracted image frames are screened, all the screened image frames will be cropped to obtain historical colonoscopy images for training. Specifically, during the cropping process, useless information in the image frames is mainly removed, such as colonoscopy equipment information, patient information, etc., and the area where the useless information is located is cropped by identifying the information in the image frames.

[0041] Step 102: training the constructed image filtering model based on the historical colonoscopy images, and performing image filtering processing on the historical colonoscopy images using the trained image filtering model to obtain a set of qualified colonoscopy images.

[0042] In one embodiment, for the historical colonoscopy images obtained through corresponding processing, there may be images that are not suitable for evaluating the visual proportion of intestinal mucosa, such as images with low cleanliness or clarity, and in vitro images or biopsy images, etc., and the unsuitable images need to be accurately filtered. Therefore, when training, it is necessary to ensure that the model to be evaluated subsequently has effective image filtering capabilities, and by constructing a corresponding image filtering model and training it with the obtained historical colonoscopy images, an image filtering model that can be used to filter and classify colonoscopy images is obtained. At the same time, for subsequent training, the trained image filtering model can be used to perform image filtering on historical colonoscopy images, and unqualified images in historical colonoscopy images can be filtered to obtain a qualified colonoscopy image set.

[0043] For example, when obtaining the degree of visualization of the intestinal mucosa in a colonoscopy, it is first necessary to obtain qualified colonoscopy images that can be used for visualization analysis, and then obtain visualization results through analysis. For unqualified images, they can be labeled as biopsy, water absorption, blurry, and poor cleanliness images. When it is necessary to determine the specific unqualified type, the constructed image filtering model can be trained to have classification capabilities. When it is only necessary to determine whether the colonoscopy image is qualified, all unqualified labels can be directly classified into the same category (i.e., unqualified), which is set according to the specific situation.

[0044] Furthermore, when the constructed image filtering model is trained using historical colonoscopy images, and a qualified colonoscopy image set is obtained using the trained image filtering model, reference can be made to Figure 2 , Figure 2It is a flowchart of the steps of training an image filtering model and obtaining a set of qualified colonoscopy images provided in an embodiment of the present application, wherein the steps include steps 201 to 204.

[0045] Step 201, selecting a first training set from historical colonoscopy images, and annotating each image in the first training set based on a filter label to obtain an annotated first training set, wherein the filter label includes a qualified label and an unqualified label;

[0046] Step 202, training the constructed image filtering model using the annotated first training set to obtain a trained image filtering model;

[0047] Step 203, using the trained image filtering model to perform labeling processing on each image in the historical colonoscopy images, to obtain an output label of each image output by the trained image filtering model;

[0048] Step 204 , performing image filtering processing on the historical colonoscopy images according to the output labels to obtain a set of qualified colonoscopy images.

[0049] Specifically, a first training set is selected from historical colonoscopy images, and each image in the first training set is labeled based on a filter label to obtain a labeled first training set, wherein the filter label includes a qualified label and an unqualified label. The constructed image filtering model is then trained using the labeled first training set to obtain a trained image filtering model. When data that can be used for subsequent training is obtained, each image in the historical colonoscopy images is labeled using the trained image filtering model to obtain an output label corresponding to each image. The historical colonoscopy images are then filtered according to the output labels to obtain a combination of qualified colonoscopy images.

[0050] Exemplarily, when training an image filtering model based on historical colonoscopy images, it is necessary to pre-label the historical colonoscopy images, that is, to classify the historical colonoscopy images into qualified images and unqualified images. At this time, some images can be selected from the historical colonoscopy images for training, that is, the first training set. During training, by extracting features from the images corresponding to the unqualified labels, unqualified feature information is obtained, and then the constructed image filtering model is trained based on the unqualified features, so that the image filtering model can accurately identify and extract unqualified features, so as to have the ability to recognize unqualified colonoscopy images.

[0051] Furthermore, in order to be able to perform a subsequent analysis of the degree of visualization of the intestinal mucosa, after obtaining the trained image filtering model, the trained image filtering model can be used to re-label each image in the historical colonoscopy images to obtain a new output label for each image, and then the historical colonoscopy images are filtered according to the output labels to obtain a set of qualified colonoscopy images, that is, images that can be used to accurately analyze the degree of visualization of the intestinal mucosa.

[0052] Step 103, annotating the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and using the annotated qualified colonoscopy image set to train the constructed image segmentation model to obtain a trained image segmentation model.

[0053] In one embodiment, after obtaining a set of qualified colonoscopy images, each image in the qualified colonoscopy images is subjected to boundary annotation processing based on the intestinal mucosa, and then the constructed image segmentation model is trained using the set of qualified colonoscopy images after the annotation processing to obtain a trained image segmentation model, wherein the image segmentation model is used to segment and identify the boundaries of the intestinal mucosa in the colonoscopy images, determine the specific boundary information, and then obtain the visualization ratio of the intestinal mucosa by calculation.

[0054] For example, when training the image segmentation model, the boundary features of the marked boundaries are obtained by extracting the boundary features, and then the image segmentation model is trained based on the boundary features. Figure 3 , Figure 3 It is a flowchart of the steps of obtaining an image segmentation model provided in an embodiment of the present application, wherein the steps include steps 301 to 305.

[0055] Step 301, annotating the boundary of each image in the qualified colonoscopy image set based on the intestinal mucosa to obtain the annotated qualified colonoscopy image set, and extracting features from the boundary of each image in the annotated qualified colonoscopy image set to obtain boundary features of each image;

[0056] Step 302, obtaining a training set from the annotated qualified colonoscopy image set, and training the constructed image segmentation model based on the training set to obtain an image segmentation model to be verified;

[0057] Step 303, obtaining a verification set from the labeled qualified colonoscopy image set, and verifying the image segmentation model to be verified based on the verification set to determine whether the verification is passed;

[0058] Step 304: if the verification is successful, the image segmentation model to be verified is used as the trained image segmentation model;

[0059] Step 305, if the verification fails, the following steps are executed periodically: the boundaries of each image in the qualified colonoscopy image set are annotated based on the intestinal mucosa to obtain a set of annotated qualified colonoscopy images, and the boundaries of each image in the annotated set of qualified colonoscopy images are extracted to obtain boundary features of each image.

[0060] Specifically, when training the constructed image segmentation model, each image in the qualified colonoscopy image set is preliminarily annotated with boundaries based on the intestinal mucosa, and then for each annotated image, features can be extracted from the boundaries to obtain boundary features of each image, and then a training set is obtained from the annotated qualified colonoscopy image set to train the constructed image segmentation model, and a verification set is obtained from the annotated qualified colonoscopy image set to verify the trained image segmentation model to be verified to determine whether the training is completed. When the verification passes, the image segmentation model to be verified is used as the trained image segmentation model, otherwise it will be necessary to continue training, specifically by periodically executing step 301, that is, returning to step 301 to restart training when the verification fails until the subsequent verification passes.

[0061] In the actual training process, within a training cycle, the training data can be divided into a training set and a validation set. Within a training cycle, the training set is used for training, and the validation set is used to verify whether the trained model is applicable. If the verification passes, the training is considered complete, otherwise training is required again.

[0062] Furthermore, when retraining, the following steps will be performed: re-extracting the features of the boundaries of each image in the labeled qualified colonoscopy image set, re-obtaining the boundary features of each image, and re-performing subsequent training, that is, if the verification fails, returning to step 301 and restarting. At the same time, when returning to step 301 and restarting the training, the training set and the verification set used subsequently can be randomly re-divided.

[0063] In addition, when the image segmentation model is periodically trained, the training cycle is not unlimited, that is, the number of active optimization and adjustment of the model parameters in the image segmentation model is limited. After a certain number of active parameter optimization and adjustment, if the image segmentation model still fails to pass the verification, it will be necessary to determine the reason why the training is not completed. Since the training of the image segmentation model is based on the trained image filtering model to filter the historical colonoscopy images to obtain a set of qualified colonoscopy images, therefore, when the training of the image segmentation model is still not completed after a certain training cycle, the image filtering model can be re-trained and adjusted, specifically including: if the training cycle of the constructed image segmentation model using the labeled qualified colonoscopy image set is greater than the cycle threshold, then execute the steps: train the constructed image filtering model based on the historical colonoscopy images, and use the trained image filtering model to perform image filtering processing on the historical colonoscopy images to obtain a set of qualified colonoscopy images.

[0064] That is, when there are too many training cycles for training the image segmentation model using training data, it means that there are certain problems with the training data used for training, that is, the obtained set of qualified colonoscopy images is abnormal, such as containing unqualified colonoscopy images. Since filtering may not be 100% successful, there may be a situation where the proportion of unqualified colonoscopy images is too high. Therefore, it is necessary to further optimize the image filtering model, such as parameter fine-tuning, so that the obtained image filtering model can accurately complete the filtering process of historical colonoscopy images, and then the image segmentation model can be trained based on the set of qualified colonoscopy images obtained by filtering.

[0065] Step 104, fine-tuning the trained image filtering model and the trained image segmentation model, and obtaining a colonoscopy image evaluation model after the fine-tuning is completed.

[0066] In one embodiment, after completing the training to obtain the trained image segmentation model, the trained image filtering model and the trained image segmentation model are fine-tuned, and then the corresponding colonoscopy image evaluation model is obtained when the fine-tuning process is completed.

[0067] For example, when fine-tuning the trained image filtering model and image segmentation model, the two models are fused and then the fused model is fine-tuned. Figure 4 , Figure 4 It is a flow chart of the steps of obtaining a colonoscopy image evaluation model provided in an embodiment of the present application, wherein the steps include steps 401 to 403.

[0068] Step 401, performing model fusion processing on the trained image filtering model and the trained image segmentation model to obtain a corresponding fusion model;

[0069] Step 402, selecting a fine-tuning set from the historical colonoscopy images, and performing a filtering label and boundary annotation process on each image in the fine-tuning set;

[0070] Step 403, fine-tune the parameters of the fusion model using the fine-tuning set after the annotation processing to obtain the corresponding colonoscopy image evaluation model.

[0071] Specifically, when fine-tuning, the trained image filtering model and image segmentation model are first fused to obtain a new model, and then a fine-tuning set is selected from the historical colonoscopy images, that is, a certain number of colonoscopy images are obtained, and each image in the fine-tuning set is annotated with filter labels and boundaries. The annotated fine-tuning set is then used to fine-tune the parameters of the fused model, and a corresponding colonoscopy image evaluation model is obtained when the fine-tuning is completed.

[0072] When fine-tuning the fused model, the colonoscopy images for model fine-tuning can be randomly selected from historical colonoscopy images or new colonoscopy images. Then, the filter labels and boundaries of the colonoscopy images can be manually and actively annotated, and the annotated colonoscopy images can be used to fine-tune the fusion model to obtain a colonoscopy quality assessment model.

[0073] Furthermore, after the training is completed to obtain the colonoscopy image evaluation model, when in use, the video / image collected by the colonoscope is analyzed and processed to obtain the current colonoscopy image evaluation result, specifically the intestinal mucosa visualization ratio, which can be the intestinal mucosa visualization ratio of a colonoscopy image, or the cumulative intestinal mucosa visualization ratio of a scene video, wherein the intestinal mucosa visualization ratio is: after the colonoscope reaches the ileocecal valve and starts to withdraw the scope, the area of ​​the intestinal mucosa on each frame of the colonoscopy image accounts for the percentage of the entire image area, the cumulative intestinal mucosa visualization ratio is: after the colonoscope reaches the ileocecal valve and starts to withdraw the scope, after filtering out the images of biopsy, flushing, blur and poor cleanliness, at a fixed frame rate (such as 10 frames per second), the cumulative sum of the intestinal mucosa visualization ratios of all colonoscopy images (the cumulative sum can be greater than 1).

[0074] Therefore, when the colonoscopy image evaluation model is used, it includes: obtaining the colonoscopy image collected by the colonoscope; inputting the colonoscopy image into a pre-trained colonoscopy image evaluation model, and outputting an image evaluation result corresponding to the colonoscopy image; obtaining a preset image evaluation standard, and generating evaluation display information corresponding to the colonoscopy image based on the image evaluation result, the image evaluation standard and the colonoscopy image.

[0075] Exemplarily, the collected colonoscopic images are input into the trained colonoscopic image evaluation model obtained above, and unqualified images in the colonoscopic images are filtered out by filtering and segmenting the colonoscopic images. At the same time, the boundaries of the qualified colonoscopic images are segmented to obtain the intestinal mucosa visualization ratio of each image in the qualified colonoscopic images. At the same time, the intestinal mucosa visualization ratios of all qualified colonoscopic images can be added up to obtain the cumulative ratio, that is, to obtain the image evaluation result of the colonoscopic images.

[0076] Moreover, after obtaining the image evaluation results, the results can be displayed. At the same time, the actual filtering situation can be displayed. When filtering, the unqualified features in the colonoscopy image are identified to determine whether the colonoscopy image is qualified. The unqualified labels include biopsy, water absorption, blur and poor cleanliness. Therefore, during the evaluation process, the filtered colonoscopy images can also be classified based on specific unqualified labels to obtain specific classification results for display.

[0077] In summary, the above embodiment provides a method for training a colonoscopy image evaluation model, which obtains a model for quality evaluation of colonoscopy images by acquiring relevant historical data for training. Specifically, historical inspection videos collected by colonoscopy are acquired, and historical colonoscopy images are obtained through corresponding preprocessing for data training of the model. First, the image filtering model is trained using historical colonoscopy images, and the trained image filtering model is used to filter the historical colonoscopy images to obtain a qualified set of colonoscopy images that can be used for intestinal mucosal visualization evaluation. Then, the qualified colonoscopy image set is used to train the image segmentation model, and finally the trained image filtering model and image segmentation model are fine-tuned to obtain a colonoscopy image evaluation model. The colonoscopy image evaluation model is trained based on the image's eligibility features and boundary features, thereby automatically filtering and screening colonoscopy videos and images of colonoscopy examinations, and evaluating the degree of intestinal mucosal visualization of qualified colonoscopy images, obtaining an accurate proportion of intestinal mucosal visualization, etc., thereby improving the quality and accuracy of colonoscopy examinations.

[0078] According to the method described in the above embodiment, this embodiment will be further described from the perspective of a training device for a colonoscopy image evaluation model. The training device for the colonoscopy image evaluation model can be implemented as an independent entity or integrated into an electronic device, such as a terminal. The terminal may include a mobile phone, a tablet computer, etc.

[0079] See also Figure 5 , Figure 5 is a structural schematic diagram of a training device for a colonoscopy image evaluation model provided in an embodiment of the present application, such as Figure 5 As shown, the training device 500 of the colonoscopy image evaluation model provided in the embodiment of the present application includes:

[0080] The data acquisition module 501 is used to acquire the historical examination videos collected by the colonoscopy, and pre-process the historical examination videos to obtain historical colonoscopy images;

[0081] The first training module 502 is used to train the constructed image filtering model based on the historical colonoscopy images, and use the trained image filtering model to perform image filtering processing on the historical colonoscopy images to obtain a set of qualified colonoscopy images;

[0082] The second training module 503 is used to annotate the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and train the constructed image segmentation model using the annotated qualified colonoscopy image set to obtain a trained image segmentation model;

[0083] The fine-tuning processing module 504 is used to perform model fine-tuning processing on the trained image filtering model and the trained image segmentation model, and obtain a colonoscopy image evaluation model after the fine-tuning processing is completed.

[0084] In one embodiment, the data acquisition module 501 is further used to:

[0085] Extracting image frames from the historical inspection video according to the set frame rate to obtain a number of image frames;

[0086] Performing image screening on the image frames based on the time sequence relationship to obtain a number of screened image frames;

[0087] Image cropping is performed on each of the screened image frames to obtain a historical colonoscopy image.

[0088] In one embodiment, the data acquisition module 501 is further used to:

[0089] Calculating the image similarity value between two adjacent image frames in the image frames based on the time sequence relationship, and obtaining a pre-set similarity threshold;

[0090] Performing image screening processing on the image frames according to the image similarity value and the similarity threshold, to obtain a number of screened image frames;

[0091] The plurality of image frames include a first image and a second image, the first image being an image frame whose similarity value is less than a similarity threshold, and the second image being an image frame obtained by screening an image set consisting of images whose similarity values ​​are greater than or equal to the similarity threshold.

[0092] In one embodiment, the first training module 502 is further configured to:

[0093] Selecting a first training set from historical colonoscopy images, and labeling each image in the first training set based on a filter label to obtain a labeled first training set, wherein the filter label includes a qualified label and an unqualified label;

[0094] Using the annotated first training set to train the constructed image filtering model to obtain a trained image filtering model;

[0095] Using the trained image filtering model to label each image in the historical colonoscopy images, the output label of each image output by the trained image filtering model is obtained;

[0096] The historical colonoscopy images are filtered according to the output labels to obtain a set of qualified colonoscopy images.

[0097] In one embodiment, the second training module 503 is further used for:

[0098] Annotating the boundary of each image in the qualified colonoscopy image set based on the intestinal mucosa to obtain the annotated qualified colonoscopy image set, and extracting features from the boundary of each image in the annotated qualified colonoscopy image set to obtain boundary features of each image;

[0099] A training set is obtained from the labeled qualified colonoscopy image set, and the constructed image segmentation model is trained based on the training set to obtain an image segmentation model to be verified;

[0100] A validation set is obtained from the annotated qualified colonoscopy image set, and the image segmentation model to be validated is validated based on the validation set to determine whether the validation is passed;

[0101] If the verification is successful, the image segmentation model to be verified is used as the trained image segmentation model;

[0102] If the verification fails, the following steps are executed periodically: the boundaries of each image in the qualified colonoscopy image set are annotated based on the intestinal mucosa to obtain the annotated qualified colonoscopy image set, and the boundaries of each image in the annotated qualified colonoscopy image set are extracted to obtain the boundary features of each image.

[0103] In one embodiment, the second training module 503 is also used to

[0104] If the training cycle of the constructed image segmentation model using the labeled qualified colonoscopy image set is greater than the cycle threshold, the following steps are executed: the constructed image filtering model is trained based on historical colonoscopy images, and the historical colonoscopy images are filtered using the trained image filtering model to obtain a qualified colonoscopy image set.

[0105] In one embodiment, the fine-tuning processing module 504 is further configured to:

[0106] Perform model fusion processing on the trained image filtering model and the trained image segmentation model to obtain a corresponding fusion model;

[0107] Select a fine-tuning set from the historical colonoscopy images, and perform filtering label and boundary annotation processing on each image in the fine-tuning set;

[0108] The fine-tuning set after annotation is used to fine-tune the parameters of the fusion model to obtain the corresponding colonoscopy image evaluation model.

[0109] Also, see Figure 6 , Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device may be a mobile terminal such as a smart phone, a tablet computer, or the like. Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 602. The processor 601 is electrically connected to the memory 602.

[0110] The processor 601 is the control center of the electronic device 600. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device 600 and processes data by running or loading applications stored in the memory 602 and calling data stored in the memory 602, thereby monitoring the electronic device 600 as a whole.

[0111] In this embodiment, the processor 601 in the electronic device 600 will load the instructions corresponding to the processes of one or more applications into the memory 602 according to the following steps, and the processor 601 will run the application stored in the memory 602, thereby implementing any step in the training method of the colonoscopy image evaluation model provided in the above embodiment.

[0112] The electronic device 600 can implement the steps in any embodiment of the training method for the colonoscopy image evaluation model provided in the embodiments of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the training method for any colonoscopy image evaluation model provided in the embodiments of the present application. Please refer to the previous embodiments for details and will not be repeated here.

[0113] See also Figure 7 , Figure 7 is another structural schematic diagram of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, Figure 7The specific structural block diagram of the electronic device provided in the embodiment of the present application is shown, and the electronic device can be used to implement the training method of the colonoscopy image evaluation model provided in the above embodiment. The electronic device 700 can be a mobile terminal such as a smart phone or a laptop computer.

[0114] The RF circuit 710 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other devices. The RF circuit 710 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity module (SIM) cards, memories, etc. The RF circuit 710 can communicate with various networks such as the Internet, corporate intranets, wireless networks, or communicate with other devices through wireless networks. The above-mentioned wireless networks may include cellular telephone networks, wireless local area networks, or metropolitan area networks. The above-mentioned wireless networks can use various communication standards, protocols and technologies, including but not limited to Global System for Mobile Communication (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (WCDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Wireless Fidelity (Wi-Fi) (such as Institute of Electrical and Electronics Engineers standards IEEE 802.11a, IEEE 802.11b, IEEE802.11g and / or IEEE802.11n), Voice over Internet Protocol (VoIP), Worldwide Interoperability for Microwave Access (Wi-Max), other protocols for email, instant messaging and short messages, and any other suitable communication protocols, even those that have not yet been developed.

[0115] The memory 720 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method of the colonoscopy image evaluation model in the above-mentioned embodiment. The processor 780 executes various functional applications and the training method of the colonoscopy image evaluation model by running the software programs and modules stored in the memory 720.

[0116] The memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 720 may further include a memory remotely arranged relative to the processor 780, and these remote memories may be connected to the electronic device 700 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0117] The input unit 730 can be used to receive uploaded digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 730 may include a touch-sensitive surface 731 and other input devices 732. The touch-sensitive surface 731, also known as a touch display screen or touchpad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 731), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 731 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch-sensitive surface 731 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0118] The display unit 740 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 700, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 740 may include a display panel 741, and optionally, the display panel 741 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 731 may cover the display panel 741, and when the touch-sensitive surface 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of the touch event, and then the processor 780 provides corresponding visual output on the display panel 741 according to the type of the touch event. Although in the figure, the touch-sensitive surface 731 and the display panel 741 are implemented as two independent components to implement input and output functions, in some embodiments, the touch-sensitive surface 731 and the display panel 741 can be integrated to implement input and output functions.

[0119] The electronic device 700 may also include at least one sensor 750, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor may generate an interrupt when the flip cover is closed or closed. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 700, they will not be repeated here.

[0120] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the electronic device 700. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, which is converted into a sound signal for output; on the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data, and then the audio data is output to the processor 780 for processing, and then sent to another terminal through the RF circuit 710, or the audio data is output to the memory 720 for further processing. The audio circuit 760 may also include an earplug jack to provide communication between an external headset and the electronic device 700.

[0121] The electronic device 700 can help the user receive requests, send information, etc. through the transmission module 770 (such as a Wi-Fi module), which provides the user with wireless broadband Internet access. Although the transmission module 770 is shown in the figure, it is understandable that it is not a necessary component of the electronic device 700 and can be omitted as needed without changing the essence of the invention.

[0122] The processor 780 is the control center of the electronic device 700. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720, it executes various functions of the electronic device 700 and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 780 may include one or more processing cores; in some embodiments, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It can be understood that the above-mentioned modem processor may not be integrated into the processor 780.

[0123] The electronic device 700 also includes a power supply 790 (such as a battery) for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 780 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The power supply 790 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0124] Although not shown, the electronic device 700 also includes a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory, and one or more programs, wherein one or more programs are stored in the memory, and are configured to be executed by one or more processors to implement any step in the training method of the colonoscopy image evaluation model provided in the above embodiment.

[0125] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above modules can be found in the previous method embodiments, which will not be repeated here.

[0126] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by controlling related hardware through instructions, and the instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium, which stores a plurality of instructions, and when the instructions are executed by the processor, any step in the training method of the colonoscopy image evaluation model provided in the above embodiments can be implemented.

[0127] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0128] Since the instructions stored in the storage medium can execute the steps in any embodiment of the training method for the colonoscopy image evaluation model provided in the embodiments of the present application, the beneficial effects that can be achieved by the training method for any colonoscopy image evaluation model provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0129] The above is a detailed introduction to the training method, device, electronic device and storage medium of a colonoscopy image evaluation model provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and scope of application. In summary, the content of this specification should not be understood as a limitation on the present application. Moreover, for those of ordinary skill in the art, without departing from the principles of the present application, several improvements and modifications can be made, and these improvements and modifications are also regarded as the scope of protection of the present application.

Claims

1. A method for training a colonoscopy image evaluation model, characterized in that: include: Obtain historical examination videos collected by colonoscopy, and pre-process the historical examination videos to obtain historical colonoscopy images; Based on the historical colonoscopy images, the constructed image filtering model is trained, and the historical colonoscopy images are subjected to image filtering processing using the trained image filtering model to obtain a set of qualified colonoscopy images; Annotating the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and using the annotated qualified colonoscopy image set to train the constructed image segmentation model to obtain a trained image segmentation model; The trained image filtering model and the trained image segmentation model are fine-tuned, and a colonoscopy image evaluation model is obtained after the fine-tuning is completed.

2. The method according to claim 1, characterized in that The preprocessing of the historical examination video to obtain the historical colonoscopy image includes: Extracting image frames from the historical inspection video according to the set frame rate to obtain a plurality of image frames; Performing image screening on the image frames based on a time sequence relationship to obtain a plurality of screened image frames; Image cropping is performed on each of the screened image frames to obtain a historical colonoscopy image.

3. The method according to claim 2, characterized in that The image frames are screened based on the time sequence relationship to obtain a plurality of screened image frames, including: Calculating the image similarity value between two adjacent image frames in the image frames based on the time sequence relationship, and obtaining a pre-set similarity threshold; Performing image screening processing on the image frame according to the image similarity value and the similarity threshold to obtain a plurality of screened image frames; The plurality of image frames include a first image and a second image, the first image being an image frame whose similarity value is less than the similarity threshold, and the second image being an image frame obtained by screening an image set consisting of images whose similarity value is greater than or equal to the similarity threshold.

4. The method according to claim 1, characterized in that The constructed image filtering model is trained based on the historical colonoscopy images, and the trained image filtering model is used to perform image filtering processing on the historical colonoscopy images to obtain a set of qualified colonoscopy images, including: Selecting a first training set from the historical colonoscopy images, and labeling each image in the first training set based on a filter label to obtain a labeled first training set, wherein the filter label includes a qualified label and an unqualified label; Using the annotated first training set to train the constructed image filtering model to obtain a trained image filtering model; Using the trained image filtering model to perform labeling processing on each image in the historical colonoscopy images, to obtain an output label of each image output by the trained image filtering model; The historical colonoscopy images are subjected to image filtering processing according to the output labels to obtain a set of qualified colonoscopy images.

5. The method according to claim 1, characterized in that The step of annotating the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and training the constructed image segmentation model using the annotated qualified colonoscopy image set to obtain a trained image segmentation model includes: Annotating the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa to obtain a marked qualified colonoscopy image set, and extracting features from the boundaries of each image in the marked qualified colonoscopy image set to obtain boundary features of each image; A training set is obtained from the annotated qualified colonoscopy image set, and the constructed image segmentation model is trained based on the training set to obtain an image segmentation model to be verified; A verification set is obtained from the annotated qualified colonoscopy image set, and the image segmentation model to be verified is verified based on the verification set to determine whether the verification is passed; If the verification is successful, the image segmentation model to be verified is used as the trained image segmentation model; If the verification fails, the following steps are executed periodically: performing boundary annotation on each image in the qualified colonoscopy image set based on the intestinal mucosa to obtain a set of annotated qualified colonoscopy images, and performing feature extraction on the boundary of each image in the annotated set of qualified colonoscopy images to obtain boundary features of each image.

6. The method according to claim 5, characterized in that The method further comprises: If the training cycle of the constructed image segmentation model using the labeled qualified colonoscopy image set is greater than the cycle threshold, the steps are executed: training the constructed image filtering model based on the historical colonoscopy images, and performing image filtering processing on the historical colonoscopy images using the trained image filtering model to obtain a qualified colonoscopy image set.

7. The method according to claim 1, characterized in that The method of fine-tuning the trained image filtering model and the trained image segmentation model, and obtaining a colonoscopy image evaluation model after completing the fine-tuning, includes: Perform model fusion processing on the trained image filtering model and the trained image segmentation model to obtain a corresponding fusion model; Selecting a fine-tuning set from the historical colonoscopy images, and performing a filtering label and boundary annotation process on each image in the fine-tuning set; The fine-tuning set after the annotation process is used to fine-tune the parameters of the fusion model to obtain the corresponding colonoscopy image evaluation model.

8. A training device for a colonoscopy image evaluation model, characterized in that: include: A data acquisition module is used to acquire historical examination videos collected by colonoscopy and pre-process the historical examination videos to obtain historical colonoscopy images; A first training module is used to train the constructed image filtering model based on the historical colonoscopy images, and use the trained image filtering model to perform image filtering processing on the historical colonoscopy images to obtain a set of qualified colonoscopy images; A second training module is used to annotate the boundaries of each image in the qualified colonoscopy image set based on the intestinal mucosa, and train the constructed image segmentation model using the annotated qualified colonoscopy image set to obtain a trained image segmentation model; The fine-tuning processing module is used to perform model fine-tuning processing on the trained image filtering model and the trained image segmentation model, and obtain the colonoscopy image evaluation model after the fine-tuning processing is completed.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method according to any one of claims 1 to 7 are implemented.

Images