Evaluation methods, apparatus, electronic equipment, and computer programs for evaluating cleanliness.

Abstract

To provide technical solution means for evaluating cleanliness of an examination object by analyzing an endoscopic image.SOLUTION: A method includes selection of a plurality of frames in an endoscopic image on the basis of interframe positioning of the endoscopic image. The method further includes determination of a pollution degree of each frame in the plurality of frames, and evaluation of cleanliness of an object in the endoscopic image on the basis of the pollution degree of each frame. According to the embodiment of the present disclosure, key frames distributed uniformly in different parts of the examination object are extracted from the endoscopic image that is likely to contain a pollutant, and the cleanliness of the object in the image can be evaluated more accurately and entirely, which helps to improve precision and reliability of an endoscopic examination, and reduce a workload on an operator.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate to the field of image processing technology, and more particularly, to methods, apparatus, electronic devices, computer readable storage media, and computer program products for assessing cleanliness. [Background technology]

[0002] Endoscopy is the examination of the body using optical instruments that are passed from outside the body through the body's natural cavities to examine the body for diseases. An endoscope can consist of a curved section, a light source, and a set of lenses. In use, an operator introduces the endoscope into the organ to be examined and controls the movement of the endoscope to directly view the relevant part of the organ and record images or videos.

[0003] In actual use, various contaminants may be present in the organ to be examined, which may hinder the examination. Therefore, comprehensive and accurate evaluation and scoring of cleanliness in endoscopic examination contributes to efficient and accurate quality control of the examination. Evaluation is usually performed by the endoscope operator after the examination is performed. However, quantitative or qualitative scoring of cleanliness by relying only on memory increases fatigue, and is subjective and inaccurate. It is necessary to introduce an accurate cleanliness evaluation mechanism into the endoscope system to improve the quality of endoscopic examination and reduce the workload of the operator. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of this, an embodiment of the present disclosure puts forward a technical solution for evaluating the cleanliness of an inspection object by analyzing an endoscopic image. [Means for solving the problem]

[0005] In a first aspect of the present disclosure, a method for assessing cleanliness is provided, the method including: selecting a plurality of frames in an endoscopic video based on frame-to-frame registration of the endoscopic video, determining a degree of contamination for each frame in the plurality of frames, and assessing a degree of cleanliness of an object in the endoscopic video based on the degree of contamination for each frame.

[0006] In a second aspect of the present disclosure, an apparatus for assessing cleanliness is provided, the apparatus comprising: a frame selection unit configured to select a plurality of frames in an endoscopic video based on frame-to-frame registration of the endoscopic video, a contamination degree determination unit configured to determine a contamination degree of each frame in the plurality of frames, and a cleanliness evaluation unit configured to determine a cleanliness degree of an object in the endoscopic video based on the contamination degree of each frame.

[0007] In a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising at least one processor unit and at least one memory coupled to the at least one processor unit and storing instructions for execution by the at least one processor unit, the instructions, when executed by the at least one processor unit, causing the electronic device to perform the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, there is provided a computer-readable storage medium comprising machine-readable instructions which, when executed by an apparatus, cause the apparatus to perform a method according to the first aspect of the present disclosure.

[0009] In a fifth aspect of the present disclosure, there is provided a computer program product comprising machine readable instructions which, when executed by an apparatus, cause the apparatus to perform a method according to the first aspect of the present disclosure.

[0010] The Summary of the Invention is intended to introduce a selection of concepts in a simplified manner, which will be further described in the following embodiments. The descriptions in the Summary of the Invention are not intended to identify key features or required features of the disclosure, nor are they intended to limit the scope of the disclosure. [Brief description of the drawings]

[0011] The above and other objects, features and advantages of the present disclosure will become more apparent by describing in more detail exemplary embodiments of the present disclosure with reference to the drawings, in which like reference numerals generally represent like elements. [Figure 1A] 1 illustrates a schematic diagram of an exemplary environment in which embodiments of the present disclosure can be implemented; [Figure 1B] 1 shows an example of bowel cleanliness scoring applicable to embodiments of the present disclosure. [Diagram 2] 1 shows a schematic flow chart of a method for assessing cleanliness according to an embodiment of the present disclosure. [Diagram 3] 1 shows a schematic diagram of a cleanliness evaluation system according to an embodiment of the present disclosure. [Figure 4A] 1 illustrates an example of a low quality frame according to an embodiment of the present disclosure. [Figure 4B] 1 shows a schematic diagram of applying frame-to-frame registration to video according to an embodiment of the present disclosure; [Diagram 5] 1 shows a schematic diagram of inter-frame spatial offset according to an embodiment of the present disclosure. [Figure 6] 1 illustrates a schematic flowchart of a process for selecting key frames in a video according to an embodiment of the present disclosure. [Figure 7] 1 illustrates exemplary cleanliness evaluation results obtained in accordance with embodiments of the present disclosure. [Figure 8] FIG. 1 shows a schematic block diagram of an apparatus for assessing cleanliness, according to an embodiment of the present disclosure. [Figure 9] 1 shows a schematic block diagram of an exemplary device in which embodiments of the present disclosed subject matter can be implemented; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the preferred embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be realized in various forms and should not be limited to the embodiments discussed herein. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0013] As used herein, the term "comprises" and variations thereof refer to the open "including, but not limited to." Unless otherwise specified, the term "or" refers to "and / or." The term "based on" refers to "based at least in part on." The terms "one exemplary embodiment" and "one embodiment" refer to "at least one exemplary embodiment." The term "another embodiment" refers to "at least one other embodiment." The terms "first," "second," etc. can refer to different or the same object. Other explicit and implicit definitions may be included in the following text.

[0014] Organs (e.g., the intestinal tract) examined by endoscopy contain contaminants that may interfere with the examination. Depending on the type, form, and amount of contaminants, the impact on the examination is not the same. Qualitative or quantitative scoring of the cleanliness of the organ usually needs to rely on the memory of the operator (e.g., a doctor) after the examination. This method is more subjective and depends on the operator's experience, which causes a problem of reduced accuracy in the quality control of endoscopy. It is necessary to introduce an accurate cleanliness evaluation mechanism into the endoscopy system to improve the quality of endoscopy and reduce the workload of the operator.

[0015] In the conventional solution, deep neural networks are used to directly analyze intestinal images collected at fixed time intervals (e.g., 30 seconds) to produce cleanliness evaluation results. The problem with this method is, firstly, that it cannot evaluate the cleanliness comprehensively, on the one hand, it does not take into account the quality of the images being analyzed, and on the other hand, it does not acquire images of different parts evenly. Another problem is that the evaluation method relies on deep neural networks to classify the cleanliness of the entire image, which is not precise enough to judge the cleanliness from the pixel level. These problems lead to practical performance degradation.

[0016] In view of this, an embodiment of the present disclosure provides a cleanliness evaluation method based on key frame extraction, which can obtain the cleanliness of an inspection object in an endoscope video more accurately and comprehensively. In this method, frame-to-frame registration is performed on the endoscope video, and multiple frames in the endoscope video are selected as key frames based on the result of the frame-to-frame registration. The cleanliness of these key frames is then evaluated, and the cleanliness of the object in the endoscope video is evaluated based on the cleanliness of each key frame. It should be noted that although the present specification describes the cleanliness evaluation of the intestinal tract based on an enteroscope video as an example, the embodiment of the present disclosure may be applied to other types of endoscope video (e.g., gastroscope, etc.) to evaluate the cleanliness of the corresponding object in the video, and is not limited to the intestinal tract. The present disclosure is not limited in this respect. The following describes the details of the implementation of the embodiment of the present disclosure in detail with reference to Figures 1 to 9.

[0017] FIG. 1 illustrates a schematic diagram of an exemplary environment 100 in which embodiments of the present disclosure may be implemented. As illustrated, the environment 100 relates to an endoscopic system and includes a control device 101 operable by an operator. The operator may operate the control device 101 to control an endoscopic lens 102 to move within the human body and collect images and video. The collected images and video may be transmitted to a computing device 105 for storage. A display 103 may be coupled to the computing device 105 to display images and video collected through the lens 102 in real time. Although FIG. 1 illustrates an exemplary enteroscope system, it should be understood that embodiments of the present disclosure may be applied to other endoscopic systems, including, but not limited to, gastroscopes, ENT endoscopes, oral endoscopes, dental endoscopes, neuroendoscopes, urethrocystoscopes, resecting endoscopes, laparoscopes, arthroscopes, sinusoscopes, laryngoscopes, and the like.

[0018] The computing device 105 may be a general-purpose computing device or a dedicated device for endoscopy, and is configured to apply to image processing or video processing, particularly medical image processing. The computing device 105 may be a terminal or a server device. If the computing device 105 is a server, it may be an independent server, a server network of servers, or a cluster of servers. This includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or multiple servers to build a cloud server. The computing device 105 may include, but is not limited to, a desktop terminal or a mobile terminal, such as a desktop computer, a mobile phone, a tablet computer, a laptop, a medical auxiliary device, etc.

[0019] An exemplary environment in which embodiments of the present disclosure may be implemented has been described above with reference to Fig. 1A. It should be understood that Fig. 1A is merely schematic, and the environment may include more modules or systems, or may omit some modules or systems, or may incorporate new combinations of the modules or systems shown. Embodiments of the present disclosure may be implemented in environments other than those shown in Fig. 1, and the present disclosure is not limited in this respect.

[0020] Images collected inside the human body may contain human body contaminants such as feces, which may interfere with the examination. The form and amount of contaminants reflect different cleanliness levels, and the impact on endoscopic examination is not the same. Taking intestinal cleanliness as an example, the contaminants are mainly solid feces and liquid feces. The internationally commonly used standard is the Boston scoring. FIG. 1B shows an example of the scoring of intestinal cleanliness applicable to the embodiment of the present disclosure. The intestinal image in FIG. 1B corresponds to 0 to 3 points of the Boston scoring.

[0021] As shown in Figure 1B, a score of 0 indicates that there is solid stool that cannot be expelled in the intestinal tract and the intestinal mucosa is not clearly visible; a score of 1 indicates that some of the intestinal mucosa is clearly visible and some is unclear due to residual stool and opaque liquid; a score of 2 indicates that small clumps of stool and opaque liquid remain and the mucosa is clearly visible; and a score of 3 indicates that all of the mucosa is clearly visible and no stool or opaque liquid remains in the intestinal tract.

[0022] In order to guide the preparation of the endoscopy and improve the examination effect, the endoscopy video can be analyzed to determine the contamination degree at different positions of the examination object, such as the intestine, and assist the operator in determining the cleanliness. According to an embodiment of the present disclosure, the computing device 105 can perform frame-to-frame registration of the endoscopy video to obtain the spatial offset between each frame. By utilizing the spatial offset, the computing device 105 can determine the spatial distance between each frame in the examination object and select a number of evenly distributed frames as key frames. Then, the computing device 105 performs frame-by-frame analysis on the selected key frames to determine the contamination degree of each frame, and further combines the contamination degrees of all the key frames to determine the cleanliness of the examination object.

[0023] FIG 2 shows a schematic flow chart of a method 200 for assessing cleanliness according to an embodiment of the present disclosure. The method 200 may be implemented by the computing device 105 shown in FIG 1. It should be understood that the method 200 may further comprise additional operations not shown and / or may omit operations shown, and the scope of the present disclosure is not limited in this respect. The method 200 is described in detail below with reference to FIG 1A.

[0024] In block 210, the computing device 105 selects a number of frames in the endoscopic video based on the frame-to-frame registration of the endoscopic video. In some embodiments, the computing device 105 selects high-quality frames that are evenly distributed in different parts of the object. These frames may be referred to as key frames. To select high-quality frames, the computing device 105 may perform quality scoring for each frame of the video, and filter out low-quality frames and not select them as key frames. For the selected evenly distributed key frames, the computing device 105 performs frame-to-frame registration of the video and obtains distance information between the frames.

[0025] Based on the inter-frame alignment, the computing device 105 may obtain an offset amount between corresponding pixels of two adjacent frames (e.g., the difference between the abscissa and ordinate of corresponding pixels). The offset amount may be expressed in the form of a two-dimensional vector (e.g., {dx, dy}). The computing device 105 may determine an average value of the offset amounts of some or all pixels as the spatial offset between the two frames. Furthermore, the computing device 105 may use some measure of the spatial offset (e.g., Euclidean distance) as the distance between the two frames. The computing device 105 may take the distance between the frames into consideration when selecting key frames, and control the distance between adjacent key frames to be within a certain range so that the adjacent key frames are evenly distributed.

[0026] After obtaining the multiple frames considered as key frames, in block 220, the computing device 105 determines the contamination degree of each frame in the multiple frames. Considering that most of the contamination materials in the organs of the human body are a mixture of solid and liquid and tend to have a certain transparency, the higher the transparency, the clearer the inspected part is. Therefore, the computing device 105 may obtain the transparency (alpha) of pixels in the region of interest of each frame. The transparency value may be, for example, in the interval [0,1], where 0 represents opaqueness and 1 represents complete transparency. In some embodiments, the computing device 105 may obtain the transparency of pixels by performing soft segmentation of each frame, and then determine the contamination degree of the frame based on the accumulation of transparency or other methods.

[0027] In block 230, computing device 105 evaluates the cleanliness of an object in the endoscopic image based on the contamination level of each frame. Computing device 105 classifies the frames based on the contamination level of each frame. The classification criteria may refer to the international standard Boston Classification. In some embodiments, the frames may be classified by a method based on thresholds. For example, based on experience, a set of contamination thresholds (e.g., H1 < H2 < H3, etc.) may be set by manual labeling and regression labeling, and accordingly a set of contamination intervals may be obtained. Computing device 105 performs classification according to the interval in which the contamination level of a single frame is located.

[0028] Next, computing device 105 may perform overall cleanliness statistics based on the classification results of all key frames. The cleanliness statistics may include the total number of frames in each classification shown in histogram form, or the percentage of each classification.

[0029] According to the cleanliness evaluation method shown in FIG. 2, computing device 105 can extract key frames evenly distributed in different parts from the endoscopic image that may contain contaminants, and can evaluate the cleanliness of the object in the image more accurately and comprehensively. Therefore, method 200 can improve the accuracy and reliability of endoscopic examinations and reduce the workload of the operator.

[0030] FIG. 3 shows a schematic diagram of a cleanliness evaluation system 300 according to an embodiment of the present disclosure. The cleanliness evaluation system 300 can be implemented in computing device 105 in the form of software or hardware, and can be applied to the implementation of method 200 described with reference to FIG. 2. It should be understood that the cleanliness evaluation system 300 shown in FIG. 3 is merely exemplary, and may include more or fewer modules or units.

[0031] As shown, the cleanliness assessment system 300 includes a quality scoring module 305, a frame-to-frame alignment module 310, a key frame extraction module 315, and a cleanliness assessment module 320. As shown, frames 301, 302, 303, etc. of a video are provided to the cleanliness assessment system 300, where frames 301, 302, 303 are a sequence of frames in time order, i.e., frame 302 is the immediately subsequent frame of frame 301, frame 303 is the immediately subsequent frame of frame 302, and so on.

[0032] The quality scoring module 305 and the inter-frame registration module 310 of the cleanliness assessment system 300 may collectively be referred to as pre-processing modules. The quality scoring module 305 and the inter-frame registration module 310 may operate independently of each other, and thus may operate in any order, either sequentially or in parallel.

[0033] The quality scoring module 305 is for scoring the quality of each frame of the video to ensure the validity of the key frames and obtaining a corresponding quality level, which may be expressed as a number in an interval, for example in the range [0,1], where 0 represents low quality and 1 represents high quality.

[0034] The quality scoring module 305 is for filtering out low-quality frames that are not suitable for use in cleanliness assessment, such as low-light, high-light, blurred, water-submerged, and close-up. FIG. 4A illustrates an example of low-quality frames according to an embodiment of the present disclosure, including water-submerged, low-light, blurred, and close-up images. The quality scoring module 305 may score the quality based on feature operator calculation and template matching. This method is computationally efficient and can meet real-time requirements. Optionally, the quality scoring module 305 may further score the quality using a learning method based on deep learning supplemented with manual labeling. Optionally, the quality scoring module 305 may further score the quality using a learning method that combines feature extraction with a classifier (e.g., support vector machine).

[0035] The inter-frame alignment module 310 is for aligning adjacent frames of a video, and determines the mapping relationship between some or all pixels of two frames and the offset amount between the pixels mapped to each other, thereby calculating the spatial offset between the two frames, for example, the spatial offset between frames 301 and 302 may be the average value of the offset amounts between the matched pixels.

[0036] The registration method used by the inter-frame registration module 310 may include a sparse optical flow field, e.g., a KLT method that uses local gradient changes to calculate pixel mapping points after focusing on edge points. In this case, the inter-frame spatial offset may be an average value of the offset amounts of the sparse optical flow points or dense matching points. Optionally, the registration method may further include a deep learning method, e.g., FlowNet, which uses a deep learning network to automatically obtain pixel spatial offsets. Optionally, the registration method may further include a region matching method, e.g., LF-Net. The embodiments of the present disclosure are not limited to the registration method.

[0037] 4B illustrates a capture of applying frame-to-frame registration to video according to an embodiment of the present disclosure. The frame-to-frame registration module 310 sequentially performs registration on a series of adjacent frames on the left side of FIG. 4B to obtain a corresponding pixel in the next frame for a given pixel, and determines an offset between the two pixels. As shown in FIG. 4B, the obtained offset can be used to indicate the motion of the given pixel between frames (as indicated by the arrow).

[0038] The key frame extraction module 315 selects key frames based on the quality level of each frame obtained from the quality scoring module 305 and the spatial offset between adjacent frames obtained from the inter-frame alignment module 310. The key frame extraction module 315 may filter out frames with low quality and not select frames whose quality level is below a quality threshold. The key frame extraction module 315 may also calculate the spatial offset between any two frames based on the spatial offset between adjacent frames and determine the distance between the frames based on the spatial offset. Since this distance may reflect the degree of perspective between the parts where the frames are located, the key frame extraction module 315 may select evenly distributed frames as key frames based on the distance.

[0039] 5 shows a schematic diagram of inter-frame spatial offset according to an embodiment of the present disclosure. FIG 5 shows a series of frames in chronological order, including a first frame 501, frames 502-1, 502-2, ..., 502-i, etc. that follow the first frame. The inter-frame alignment module 310 can be used to determine the spatial offsets {dx1, dy1}, {dx2, dy2}... between any two adjacent frames of these frames. 、 {dx i ,dy i}, where {dx1, dy1} is the spatial offset between the first frame 501 and the adjacent frame 502-1, {dx2, dy2} is the spatial offset between frames 502-1 and 502-2, and so on. Thus, the keyframe extraction module 315 calculates the spatial offset between the first frame 501 and frame 502-i as the cumulative sum of all spatial offsets between the first frame 501 and frame 502-i, i.e., {Σ i dx i ,Σ i dy i} is determined.

[0040] 6 shows a schematic flow chart of a process 600 for selecting a key frame in a video according to an embodiment of the present disclosure. The process 600 may be an exemplary implementation of block 210 shown in FIG. 2, and may be implemented by the key frame extraction module 315. The process 600 is for selecting a representative high-quality video frame from a video, and the previous and subsequent key frames can keep a certain distance to evenly reflect the cleanliness of different parts.

[0041] At block 610, a frame in the video whose quality level exceeds a quality threshold is selected as a first frame, such as frame 501 shown in FIG. 5. The first frame is the earliest key frame selected from the video, i.e., the initial key frame. In some embodiments, the frame whose first quality level exceeds a quality threshold (e.g., 0 or other value) is determined to be the initial key frame.

[0042] Subsequently, process 600 proceeds to block 620, where the selected frame is set as the current frame. When first executing block 620, the first frame is set as the current frame. Subsequently, the next key frame is searched for among the subsequent frames of the first frame.

[0043] In block 630, based on the spatial offset from the current frame and a predetermined range, at least one frame among the subsequent frames of the current frame is determined. As described with reference to FIG. 5, the spatial offset {fx, fy} between the current frame and the subsequent frame can be obtained by accumulating the spatial offsets of adjacent frames, that is, {Σ i dx i , Σ i dy i}, where i is the frame number of the subsequent frame with respect to the current frame. In some embodiments, some measure of the spatial offset (e.g., Euclidean distance) between the current frame and the subsequent frame may be defined as the distance between the current frame and the subsequent frame. For example, the distance

Equation

[0044] In some embodiments, to select the next key frame of the current frame, at least one candidate frame whose spatial distance from the current frame is within a predetermined range may be searched or determined from a set of subsequent frames after the current frame. In this case, the key frames may be distributed evenly. The set of subsequent frames of the current frame may be a certain number of subsequent frames after the current frame, or subsequent frames within a certain time, so that it is not necessary to search all subsequent frames, and the amount of calculation and time is reduced. Among the set of subsequent frames, a frame whose distance from the current frame is within an interval (T1, T2) may be determined as a candidate frame. The frames whose distance falls within a predetermined interval may be considered to keep a certain distance from the current key frame, and they reflect different parts.

[0045] At block 640, it is determined whether any candidate frames have a quality level that exceeds the quality threshold. If any of the candidate frames have a quality level that exceeds the quality threshold, e.g., if there are multiple such candidate frames, process 600 proceeds to block 650 and selects the frame with the highest quality level as the next key frame.

[0046] If there is no candidate frame whose quality level exceeds the quality threshold among the candidate frames, the process 600 proceeds to block 660 and selects a frame whose spatial offset from the current frame is greater than a predetermined range and whose quality level exceeds the quality threshold (e.g., greater than 0 or other value). That is, if there is no frame whose distance from the current frame is within the interval (T1, T2) and whose quality meets the standard, a frame whose quality meets the standard from a location further away from the current frame is selected as the next key frame.

[0047] After either block 650 or block 660, process 600 proceeds to block 670 to determine whether the end of the video has been reached. If the end of the video has not been reached, process 600 returns to block 620 and repeats blocks 620 to 670 to continue the iterative search for key frames. For example, after selecting the next key frame after the first frame, i.e., the second frame, the second frame is set as the current frame. Then, based on the inter-frame alignment, a spatial offset between the second frame and another set of subsequent frames after the second frame is determined, and a third frame is selected from the other set of subsequent frames as the next key frame based on the spatial offset between the second frame and the other set of subsequent frames. The process may be repeated until all key frames in the video have been determined. That is, if it is determined in block 670 that the end of the video has been reached, then in block 680, all selected frames are output.

[0048] 3, the key frame extraction module 315 outputs the selected frames to the cleanliness evaluation module 320. For example, the key frame extraction module 315 provides frame numbers to the cleanliness evaluation module 320, or provides image data of the selected frames directly to the cleanliness evaluation module 320. The cleanliness evaluation module 320 is for determining the contamination level of each frame and classifying the frames based on the contamination level. For example, when the endoscopic video is an enteroscope, the classification standard may be, for example, the Boston classification.

[0049] To determine the contamination level, the cleanliness evaluation module 320 may perform soft segmentation of contaminants for each key frame. Compared with general image segmentation (also referred to as "hard segmentation"), in soft segmentation, not only the foreground segmentation result is obtained, but also the transparency of the segmented pixels is obtained. The transparency (alpha) may be represented by a numerical value between 0 and 1, for example, where 0 represents opacity and 1 represents complete transparency. Since most contaminants are mixtures of solids and liquids and tend to have a certain transparency, and the higher the transparency, the more favorable it is for observation, it is beneficial to use the transparency obtained by soft segmentation to evaluate the contamination level of a single frame. The method for obtaining transparency may include, for example, soft segmentation based on empirical and constrained cues, or alternatively, soft segmentation based on deep learning.

[0050] Next, the cleanliness evaluation module 320 calculates the contamination level for each key frame based on the soft segmentation region mask and the transparency alpha. An exemplary calculation formula may be as follows.

Equation

[0051] The cleanliness evaluation module 320 then classifies the key frames (e.g., Boston classification) based on the contamination level and evaluates the cleanliness of the frames. In some embodiments, one or more thresholds may be set for the contamination level, and the key frames are classified based on the one or more thresholds. For example, thresholds H1, H2, H3 (H1 < H2 < H3) may be set and classified according to the following relationship. · When the contamination level W <= H1, the category is defined as 3 and the evaluation score is S = 3. · When the contamination level H1 < W <= H2, the category is defined as 2, and the evaluation score is S = 2. · When the contamination level H2 < W <= H3, the category is defined as 1, and the evaluation score is S = 1. · When the contamination level W > H3, the category is defined as 0, and the evaluation score is S = 0.

[0052] The threshold values H1, H2, and H3 may be obtained based on experience. Optionally, the threshold values H1, H2, and H3 may be obtained by manual labeling or regression labeling. Specifically, in the learning stage, a set of image frames is selected, the contamination level W is obtained according to the above method, the score of S for Boston is manually scored, and then in the learning stage, a mapping model between W and S is learned using linear regression or logistic regression. Then, in the test and prediction stages, based on the learned model, classification and scoring are performed based on the contamination level W to obtain the cleanliness of the frame.

[0053] Thereby, the cleanliness evaluation module 320 obtains the frame-level cleanliness of all key frames, and further, the cleanliness evaluation module 320 can obtain the cleanliness of the entire endoscope video. In some embodiments, the cleanliness may include the statistical results of the classification for these key frames. For example, the cleanliness evaluation module 320 may obtain the overall cleanliness by the statistical method of the histogram for all key frames. The overall cleanliness may be the distribution of each classification based on the statistics of the number of frames and the percentage.

[0054] 7 shows an exemplary cleanliness evaluation result obtained according to an embodiment of the present disclosure. The cleanliness evaluation result in FIG. 7 was obtained based on Boston scoring. In the figure, S=0 represents a frame where the intestinal mucosa is not clearly visible due to the presence of solid stool that cannot be expelled, S=1 represents a frame where some of the mucosa is clearly visible and some of the mucosa is unclear due to residual stool and opaque liquid, S=2 represents a frame where a small amount of small stool clumps and opaque liquid remain and the mucosa is clearly visible, and S=3 represents a frame where all the mucosa is clearly visible and there is no stool or opaque liquid remaining in the intestine.

[0055] Above, a method for evaluating the cleanliness of an inspection target in an endoscopic video according to an embodiment of the present disclosure has been described with reference to Figures 1 to 7. Compared with the conventional solution, the embodiment of the present disclosure can extract key frames that are evenly distributed in different parts from an endoscopic video that may contain contaminants, and can evaluate the cleanliness of the target more accurately and comprehensively, which is helpful in improving the accuracy and reliability of endoscopic examination and can reduce the workload of the operator.

[0056] 8 shows a schematic block diagram of an apparatus 800 for assessing cleanliness according to an embodiment of the present disclosure. The apparatus 800 may be implemented by the computing device 105, and may be implemented as software, hardware, or a combination of hardware and software.

[0057] As shown, the apparatus 800 includes a frame selection unit 810, a contamination degree determination unit 820, and a cleanliness evaluation unit 830. The frame selection unit 810 is configured to select a plurality of frames in the endoscopic video based on frame-to-frame registration of the endoscopic video. The contamination degree determination unit 820 is configured to determine a contamination degree of each frame in the plurality of frames. The cleanliness evaluation unit 830 is configured to evaluate a cleanliness of an object in the endoscopic video based on the contamination degree of each frame.

[0058] In some embodiments, the frame selection unit 810 may be further configured to determine a spatial offset between a first frame in the endoscopic video and a set of subsequent frames after the first frame based on the inter-frame alignment, and select a second frame from the set of subsequent frames based on the spatial offset between the first frame and the set of subsequent frames.

[0059] In some embodiments, the frame selection unit 810 may be further configured to determine a spatial offset between a first frame and a set of subsequent frames of the first frame in the endoscopic video by determining a spatial offset between adjacent frames of the endoscopic video and determining a spatial offset between the first frame and a subsequent frame in the set of subsequent frames based on an accumulation of the spatial offsets between adjacent frames.

[0060] In some embodiments, the frame selection unit 810 may be further configured to determine a spatial offset between adjacent frames of the endoscopic video by determining, for two adjacent frames of the endoscopic video, a plurality of offset amounts from a plurality of pixels of a previous frame of the two adjacent frames to corresponding pixels of a later frame based on the inter-frame alignment, and determining a spatial offset between the previous frame and the later frame based on an average value of the plurality of offset amounts.

[0061] The apparatus 800 may further comprise a quality scoring unit, which is configured to determine a quality level of each frame in the endoscopic video, and the frame selection unit 810 may further be configured to determine one frame whose quality level exceeds a quality threshold as a first frame.

[0062] In some embodiments, the frame selection unit 810 may be further configured to select the second frame from the set of subsequent frames by: determining at least one candidate frame from the set of subsequent frames, the at least one candidate frame having a spatial offset from the first frame within a predetermined range; determining whether a candidate frame exists among the at least one candidate frame having a quality level higher than a quality threshold; and in response to determining that a candidate frame exists among the at least one candidate frame having a quality level higher than the quality threshold, selecting a candidate frame having the highest quality level among the at least one candidate frame as the second frame.

[0063] In some embodiments, the frame selection unit 810 may be further configured to, in response to determining that there is no subsequent frame among the at least one candidate frame having a quality level higher than the quality threshold, select a second frame from the set of subsequent frames by selecting as the second frame a frame having a quality level above the quality threshold from among frames having a spatial offset from the first frame greater than a predetermined range.

[0064] In some embodiments, frame selection unit 810 may be further configured to determine a spatial offset between the second frame and another set of subsequent frames after the second frame based on the inter-frame alignment, and select a third frame from the another set of subsequent frames based on the spatial offset between the second frame and the another set of subsequent frames.

[0065] In some embodiments, the contamination degree determination unit 820 may be further configured to obtain a transparency of a target region in the frame based on the image segmentation, and determine a contamination degree of the frame based on the transparency of the target region.

[0066] In some embodiments, the cleanliness assessment unit 830 may be further configured to classify the plurality of frames based on one or more thresholds for contamination level, and determine the cleanliness of the object based on the classification of the plurality of frames.

[0067] In some embodiments, the cleanliness may include classification statistics for multiple frames. In some embodiments, the classification may include Boston scoring for bowel cleanliness.

[0068] FIG. 9 shows a schematic block diagram of an exemplary apparatus 900 capable of implementing embodiments of the present disclosure. For example, the computing device 105 according to the embodiments of the present disclosure may be realized by the apparatus 900. As shown in the figure, the apparatus 900 includes a central processor unit (CPU) 901. The CPU 901 may perform various appropriate operations and processes based on computer program instructions stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 may further store various programs and data required for the operation of the apparatus 900. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0069] A number of components in the device 900 are connected to an I / O interface 905. The components include an input unit 906 such as a keyboard, a mouse, etc., an output unit 907 such as various types of displays, speakers, etc., a storage unit 908 such as a magnetic disk, an optical disk, etc., and a communication unit 909 such as a network interface card, a modem, a wireless communication transceiver, etc. The communication unit 909 enables the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0070] Each of the methods or processes described above may be executed by the processor unit 901. For example, in some embodiments, these methods or processes may be embodied as a computer software program and tangibly stored in a machine-readable medium, such as the storage unit 908. In some embodiments, some or all of the computer program may be loaded and / or installed in the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the CPU 901, it may perform one or more operations of the methods described above.

[0071] The present disclosure may be a method, an apparatus, a system, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions stored thereon for carrying out aspects of the present disclosure.

[0072] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples of computer readable storage media include (but are not limited to) portable computer diskettes, hard disks, random access memories (RAM), read-only memories (ROM), erasable-programmable read-only memories (EPROM or flash memory), static random access memories (SRAM), portable compact disk read-only memories (CD-ROM), digital versatile disks (DVD), memory sticks, floppy disks, mechanical encoder devices, such as punch cards or protruding structures in grooves on which instructions are stored, and any suitable combination of the above. A computer readable storage medium as used herein is not to be construed as being a momentary signal itself, such as, for example, radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a wave guide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0073] The computer readable program instructions described herein may be downloaded from a computer readable storage medium into each computing / processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network interface card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions to be stored in the computer readable storage medium of each computing / processing device.

[0074] The computer program instructions for carrying out the operations of the present disclosure may be assembler directives, Instruction Set Architecture (ISA) instructions, machine language instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and general procedural programming languages ​​such as "C" or similar programming languages. The computer readable program instructions may run entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the context of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, the status information of the computer readable program instructions is used to personalize electronic circuitry, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), that may execute the computer readable program instructions to implement aspects of the present disclosure.

[0075] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of each block in the flowcharts and / or block diagrams, may be implemented by computer readable program instructions.

[0076] These computer readable program instructions may be provided to a processor unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, which, when executed by the processor unit of the computer or other programmable data processing apparatus, produces an apparatus implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may be stored on a computer readable storage medium. These instructions cause the computer, programmable data processing apparatus, and / or other apparatus to operate in a particular manner. Thus, a computer readable medium having instructions stored thereon includes an article of manufacture containing instructions for each aspect of implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0077] The computer readable program instructions may be loaded into a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable data processing apparatus, or other device to perform a series of operational steps to generate a computer implemented process, such that the instructions executing on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0078] The flowcharts and block diagrams in the figures represent possible architectures, functions and operations of systems, methods and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program segment or part of instructions, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions depicted in the blocks may occur in a different order than depicted in the figures. For example, two consecutive blocks may actually be essentially executed in parallel, or may be executed in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts may be implemented in a dedicated hardware-based system that performs the specified functions or operations, or may be implemented in a combination of dedicated hardware and computer instructions.

[0079] Although each embodiment of the present disclosure has been described above, the above description is illustrative and not exhaustive, and is not limited to each of the disclosed embodiments. It is clear that a person skilled in the art can make many modifications and changes without departing from the scope and spirit of each of the described embodiments. The terms used herein are selected with the intention of optimally explaining the principles, actual applications, or technical improvements in the market of each embodiment, or allowing a person skilled in the art to understand each embodiment disclosed in the present specification.

Claims

1. A method for evaluating cleanliness, Selecting multiple frames within the endoscopic image based on inter-frame alignment of the endoscopic image, Determining the degree of contamination of each frame in the aforementioned plurality of frames, Based on the degree of contamination of each frame, the cleanliness of the object in the endoscopic image is evaluated. including, Evaluation method.

2. Selecting multiple frames within the aforementioned endoscopic image means Based on the inter-frame alignment, the spatial offset between the first frame in the endoscopic image and the subsequent set of frames after the first frame is determined. Selecting a second frame from the set of subsequent frames based on the spatial offset between the first frame and the set of subsequent frames, including, The evaluation method according to claim 1.

3. Determining the spatial offset between the first frame in the endoscopic image and the set of frames following the first frame is: Determining the spatial offset between adjacent frames of the endoscopic image, Based on the accumulation of the spatial offsets between adjacent frames, the spatial offset between the first frame and one subsequent frame in the set of subsequent frames is determined. including, The evaluation method according to claim 2.

4. Determining the spatial offset between adjacent frames of the endoscopic image is: With respect to two adjacent frames of the endoscopic image, multiple offset amounts are determined based on the inter-frame alignment, from multiple pixels in the earlier frame to corresponding pixels in the later frame. The spatial offset between the previous frame and the subsequent frame is determined based on the average value of the plurality of offset amounts, including, The evaluation method described in claim 3.

5. Determining the quality level of each frame in the aforementioned endoscopic image, The first frame is determined to be one frame in which the quality level exceeds the quality threshold. Further including, The evaluation method according to claim 2.

6. Selecting a second frame from the aforementioned set of subsequent frames means From the set of subsequent frames, determine at least one candidate frame whose spatial offset from the first frame is within a predetermined range. Determining whether there is a candidate frame among the at least one candidate frame whose quality level is higher than the quality threshold, In response to the determination that there is a candidate frame among the at least one candidate frame whose quality level is higher than the quality threshold, the candidate frame having the highest quality level among the at least one candidate frame is selected as the second frame. including, The evaluation method according to claim 5.

7. Selecting a second frame from the aforementioned set of subsequent frames means In response to determining that there are no subsequent frames among the at least one candidate frame whose quality level is higher than the quality threshold, the method includes selecting a frame as the second frame from among frames whose spatial offset from the first frame is greater than the predetermined range, and whose quality level exceeds the quality threshold. The evaluation method according to claim 6.

8. Based on the inter-frame alignment, determine the spatial offset between the second frame and another set of subsequent frames after the second frame. Selecting a third frame from the set of other subsequent frames based on the spatial offset between the second frame and the set of other subsequent frames, Further including, The evaluation method according to claim 2.

9. Determining the degree of contamination of each frame in the aforementioned plurality of frames is: Based on image segmentation, the transparency of the target region within each frame is obtained, Based on the transparency of the target area within each frame, the degree of contamination of each frame is determined. including, The evaluation method according to claim 1.

10. Evaluating the cleanliness of the object in the endoscopic image is, Classifying the multiple frames based on one or more thresholds for the degree of contamination, The cleanliness of the target is determined based on the classification of the plurality of frames, including, The evaluation method according to claim 1.

11. The cleanliness level includes statistical data of the classification for the plurality of frames, The evaluation method according to claim 10.

12. The aforementioned classification includes Boston scoring for bowel cleanliness. The evaluation method according to claim 10.

13. A device for evaluating cleanliness, A frame selection unit configured to select multiple frames within an endoscopic image based on inter-frame alignment of the endoscopic image, A contamination determination unit configured to determine the contamination level of each frame in the plurality of frames, A cleanliness evaluation unit configured to determine the cleanliness of an object in the endoscopic image based on the degree of contamination of each frame, Equipped with, Device.

14. At least one processor unit, A memory connected to the at least one processor unit and storing instructions to be executed by the at least one processor unit, Electronic equipment equipped with, If the instruction is executed by the at least one processor unit, the evaluation method according to any one of claims 1 to 12 is performed. electronic equipment.

15. A computer program that includes machine-readable instructions, When the machine-readable instruction is executed by the device, the device will cause the device to execute the evaluation method described in any one of claims 1 to 12. Computer program.

Images