Quality assessment in video endoscopy

Abstract

The analyzer analyzes a video image signal including consecutive frames of an endoscopy procedure. The machine learning block analyzes the video image signal using machine learning techniques to classify regions of the frames as belonging to one of a plurality of classes corresponding to respective types of image artifacts. The classes include a motion blur class corresponding to motion blur in the image, at least one misexposure class corresponding to types of misexposure in the image, and at least one noise artifact class corresponding to types of image artifacts that are noise. The quality score block derives quality scores representing the image quality of the consecutive frames based on the classified regions.

Description

[Technical field]

[0001] The present invention relates to video endoscopy. [Background technology]

[0002] Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Originally used to image the esophagus, stomach, and colon, miniaturization of hardware and improvements in imaging sensors now allow endoscopic examination of the ear, nose, throat, heart, urinary tract, joints, and abdomen. Endoscopy provides monitoring of disease progression in any such tract, surgical planning, and control of surgical treatment. An endoscopy procedure involves the capture of a video image signal by an endoscope. The video image signal includes successive frames that image the endoscopic procedure.

[0003] Common to these endoscopy applications, a variety of different imaging artifacts corrupt the video image and impede its visual interpretation, resulting in significant clinical challenges in the use of endoscopy. The camera in the endoscope is embedded in a long flexible tube. Any small hand movement can cause severe motion artifacts in the recorded video. The light required for illumination can interact with tissue and surrounding fluids, generating very bright pixel areas (either due to specularity or pixel saturation). Different viewing angles and occlusions can result in contrast problems due to underexposure. In addition, as with any other complex real-world imaging application, visual clutter such as debris, liquid, air bubbles, etc. can limit the visual understanding of the underlying tissue.

[0004] Such artifacts limit the effectiveness of endoscopy. For example, artifacts affect the physician's ability to use video images, e.g. during diagnosis and treatment. It is part of the reason for high operator variability between endoscopists, which directly affects the quality of diagnosis. As an example, this is particularly relevant for luminal GI cancers, which can only be effectively treated if they are detected at an early stage, as disclosed in (References 48 and 49). Similarly, artifacts adversely affect any computer-assisted endoscopic methods performed on the video image signal, e.g. video mosaicing for follow-up and archiving, video frame retrieval for reporting, etc. Various methods of analyzing video image signals are known, which address the detection and restoration of selected artifacts. However, endoscopic video images typically contain a large number of artifacts, and such known analysis methods do not provide solutions that are effective in typical clinical applications.

[0005] By way of example, some known analytical methods are as follows:

[0006] (References 1 and 2) disclose studies of video frame quality assessment methods. Although very useful global video quality metrics are introduced and discussed, they do not provide information on the causes of frame quality degradation or identify degraded areas for frame restoration. In general, utilizing these quality scores only allows the removal of frames corrupted by artifacts without considering the severity of each artifact type. Such simple removal of corrupted frames can significantly reduce the information content of videos and also affect their overall temporal smoothness. One negative consequence of this can be, for example, mosaicing methods, as disclosed in (Reference 3), which require at least 60% overlap in consecutive temporal frames to be successful. Hence, artifacts are a major obstacle in developing effective and reliable computer-aided endoscopy tools. Accurate identification, classification, and, where possible, restoration are essential to perform downstream analysis of video data.

[0007] Detecting multiple artifacts and providing an appropriate restoration is very challenging. To date, most research groups have only studied specific artifacts in endoscopic imaging, e.g., in (Refs. 4-7). For example, deblurring wireless capsule endoscopy images using a Total Variation (TV) technique was proposed in (Ref. 4). However, TV-based deblurring is parameter sensitive and requires geometric features to be implemented successfully. Endoscopic images have very sparse features and lack geometrically salient structures.

[0008] Both handcrafted features as disclosed in (References 5-8) and neural networks as disclosed in (Reference 9) have been used to restore specular reflections. The main drawback of these existing restoration techniques is that heuristically chosen image intensities are compared to nearby (local) image pixels. In general, both local and global information are required for realistic frame restoration. One common limitation of almost all methods is that they address only one specific artifact class, although of course a variety of different effects corrupt endoscopy videos. For example, both “specularity” and water “bubbles” may be present in the same frame. Endoscopists also dynamically switch between different modalities during acquisition (e.g., typically brightfield (BF), acetic acid, narrow-band imaging (NBI), or fluorescence (FL)) to better highlight certain pathological features. Finally, interpatient variability is significant even when observed with the same modality. Existing methods cannot adequately address all of these challenges.

[0009] In addition to addressing one type of imaging artifact, only one imaging modality, and a single patient video sequence, has been considered in the majority of the endoscopy-based image analysis literature, e.g., (References 5-9). Summary of the Invention [Means for solving the problem]

[0010] According to a first aspect of the present invention, there is provided a method for analysing a video image signal comprising successive frames imaging an endoscopic procedure, the method comprising the steps of analysing the video image signal using machine learning techniques to classify regions of the frames as belonging to one of a plurality of classes corresponding to different types of image artefacts, said classes including a motion blur class corresponding to motion blur in the image, at least one misexposure class corresponding to types of misexposure of the image, and at least one noise artefact class corresponding to types of image artefacts introducing noise into the frames, and deriving at least one quality score representative of image quality of the successive frames based on the classified regions.

[0011] Thus, the method provides, first, a fully automatic framework that can detect and classify six different main artifacts and, second, provide a frame-by-frame quality score. Machine learning techniques provide multi-class detection for each of the motion blur, misexposure, and at least one noise artifact, providing simultaneous detection of all types of artifacts likely to be encountered in practice. Unlike previous methods (e.g., in (references 4-9) above), which require manual adjustment of parameter settings or the use of handcrafted features that are only suitable for certain artifacts, a multi-class artifact detection is applied that can utilize multi-patient and multi-modality video frames.

[0012] The classified regions are used to derive a quality score that represents the image quality of successive frames. Such a quality score offers several advantages: since it is derived from the classified regions, the quality score is derived systematically and is a reliable measure of quality. Such an approach reduces the false classification rate and generalizes better the detection method, and also the frame restoration method if applied subsequently.

[0013] The quality score can be used in many ways, for example: The quality score can be provided to the endoscopist in real time, allowing the endoscopist to adapt the endoscopy procedure, for example, by repeating the endoscopy of the imaged ductal section or by modifying the endoscopy device or procedure settings. Similarly, the quality score can be used during the training of endoscopists, since it provides reliable information about the quality of the video image signal being captured. In other situations, the quality score can be stored together with the video image signal as a quality record, which can be used to evaluate the clinical usefulness of a particular video image signal or the performance of the endoscopist. The system can also be used for the training of endoscopists. The detection of motion blur can be used directly, for example, to provide the trainee with information about where to slow down or where to re-examine a particular anatomical region.

[0014] The at least one misexposure class may include two misexposure classes corresponding to overexposure of an image and underexposure of an image, respectively.

[0015] The at least one noise artifact class includes one or more of a noise artifact class corresponding to occlusion of an image by air bubbles, a noise artifact class corresponding to specular reflection, and a noise artifact class corresponding to noise artifacts other than air bubbles or specular reflection, in which case the noise artifacts other than air bubbles or specular reflection may include any one or more of chromatic aberration or occlusion by debris.

[0016] Advantageously, the quality score may have the following characteristics:

[0017] At least one quality score may be based on the area of ​​the classified region, with the quality score representing quality decreasing with area.

[0018] At least one quality score may be based on the location of the classified region, with the quality score representing a lower quality for a central location within the image than a non-central location.

[0019] The at least one quality score may include a combined quality score based on all types of classified regions. This is not required, and quality scores may alternatively be presented for each class corresponding to a type of image artifact. However, the combined quality score is particularly useful because it provides an overall measure of quality.

[0020] Such a combined quality score may be based on the area of ​​the classified regions of each class weighted by class-dependent weights, with the quality score representing quality decreasing with area.

[0021] Such a combined quality score may be based on the location of the classified regions of each class weighted by a class-dependent weight, with the quality score representing lower quality for central locations in the image than for non-central locations.

[0022] The weights may further depend on the number of classified regions.

[0023] Advantageously, the method may further comprise a step of restoring at least some of the image frames by processing the image frames so as to reduce the type of image artifact corresponding to the class of image artifact to which the classified region belongs.

[0024] The image frames to be restored may be selected based on a quality score. This is advantageous because the quality score provides a measure of the severity of the artifacts and therefore whether a reliable restoration is possible. Image frames that are not selected for restoration may be discarded on the assumption that they will not be of sufficient quality even after restoration.

[0025] The restoring step may include processing the image frames with a separate process for each class to reduce the corresponding type of image artifact.

[0026] The separate processes may be performed in a predefined order of the classes to which the regions belong. It is understood that due to the high probability of the presence of multiple artifacts in a single frame, unordered restoration of these artifacts may not be effective and may risk a significant degradation of the frame quality. Therefore, a sequential restoration process is used that takes into account the nature of the individual artifact types.

[0027] A generative adversarial network with carefully chosen regularization is finally used to restore the corrupted frames.

[0028] Any or all of the processes may use generative adversarial networks, which may be conditioned on prior information.

[0029] Such a generative adversarial network may embed a context loss, e.g., l2-context loss and high-frequency context loss in case of processing for the motion blur class, l2-context loss in case of processing for at least one misexposure class, or l1-context loss in case of processing for at least one misexposure class. By choosing the type of regularization depending on the type of artifact, the quality of the restoration is improved.

[0030] The default order may include a motion blur class, followed by at least one misexposure class, followed by at least one noise artifact class.

[0031] A process for the motion blur class may have the following characteristics:

[0032] The process for the motion blur class may include a blind frame deblurring process that may deconvolve frames with an optimized kernel that represents the motion blur.

[0033] The process for at least one erroneous exposure class may have the following characteristics:

[0034] The process for at least one erroneous exposure class may include a gamma correction process, which may be non-linear and / or may include correction with an optimized gamma function.

[0035] The process for at least one incorrect exposure class may additionally include color transfer.

[0036] The process for at least one noise artifact class includes an inpainting process.

[0037] It has been demonstrated that such carefully chosen models can yield both high quality and highly realistic frame reconstructions.

[0038] The machine learning technique can be a Convolutional Neural Network (CNN). Reliable multi-class detection is enabled through multi-scale and deep convolutional neural network based object detection that can efficiently generalize cross-patient and cross-modality multi-class artifact detection present in endoscopic data.

[0039] A CNN can be configured to simultaneously predict the region and the class of the region in a single stage. Such a single stage of a CNN can use spatial pyramid pooling.

[0040] According to a further aspect of the present invention there is provided a computer program executable by a computing device and configured, when executed, to cause the computing device to carry out the method according to any one of claims 1 to 23. The computer program may be stored on a computer readable storage medium.

[0041] According to a still further aspect of the present invention there is provided an analysis apparatus for analysing a video image signal comprising successive frames of an endoscopy procedure, the analysis apparatus configured to perform a method similar to the method of the first aspect of the present invention.

[0042] To enable a better understanding, embodiments of the invention will now be described, by way of non-limiting example, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0043] [Figure 1] 1 is a schematic diagram of an endoscopic device and an analysis device for analyzing a video image signal generated by the endoscopic device. [Diagram 2] FIG. 2 shows four example frames of a video image signal with classified regions within the frames superimposed thereon. [Diagram 3] FIG. 13 shows example frames for which quality scores have been determined, with classified regions overlaid, along with a pie chart of the area of ​​the regions within each frame. [Figure 4] FIG. 13 shows example frames for which quality scores have been determined, with classified regions overlaid, along with a pie chart of the area of ​​the regions within each frame. [Diagram 5] FIG. 13 is a schematic diagram of the scoring process implemented in the CGAN implemented in the motion blur correction block. [Figure 6] FIG. 2 is a schematic diagram showing the process performed in the Exposure Compensation block, together with some example frames processed by it; [Figure 7] FIG. 13 shows a set of class-specific precision-recall curves for artifact detection. [Figure 8] 1A-1C show a set of frame pairs before and after motion blur correction performed using different deblurring methods. [Figure 9] FIG. 1 shows a set of frames and associated RGB histograms illustrating the effect of the exposure correction process. [Figure 10] A set of frames illustrating the effect of the inpainting process. [Figure 11] FIG. 2 shows a temporal map of three video image signals showing the frames selected for restoration, a graph of the ratios selected, and a pie chart of the proportion of each type of artifact present. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] The original video images shown in Figures 2 to 6 and 8 to 10 were color images.

[0045] FIG. 1 illustrates an endoscopy device 1 and an analysis device 10 configured as follows.

[0046] The endoscopic apparatus 1 comprises an endoscope 2 connected to a camera 3. During endoscopy, the endoscope 2 is inserted into a canal 4 of a human (or animal) body and the camera 3 captures a video image signal, comprising a number of successive frames, that represent a video image of the endoscopy procedure. The video image signal is provided from the endoscopy apparatus 2 to an analysis device 10 for analysis. The endoscopy apparatus 1 is shown diagrammatically and may in general be of any known type, and the analysis performed by the analysis device 10 is not dependent on the nature of the endoscopy apparatus 1.

[0047] The analysis device 10 may be a computing device that executes a computer program that, when executed, causes the analysis device 10 to perform a method of analyzing a video image signal. FIG. 1 illustrates functional blocks that perform the steps of the method. The computing device may be any type of computing device, but is typically of conventional construction. The computer program may be written in any suitable programming language that is executable by the computing device. The computer program may be stored on a computer readable storage medium, which may be of any type, for example a storage medium that can be inserted into a drive of a computing system and that can store information magnetically, optically, or magneto-optically, a fixed storage medium of a computer system such as a hard drive, or a computer memory.

[0048] A storage device 11 of the analysis device 10 stores input video data 12 representing the video image signal provided by the endoscopy device 1 .

[0049] The analysis device 10 performs the analysis method as follows to provide a systematic and general approach to handling cross-modality and cross-patient video image signals for both automatic detection of multiple artifacts present in the endoscopic data and subsequent reconstruction of the detected artifacts. Such artifacts may be embodied in a frame linearly, non-linearly, or both.

[0050] The CNN block 13 implements a convolutional neural network (CNN) that is used to analyze the video image signal. A CNN is an example of a machine learning technique. A CNN classifies regions of a frame of the video image signal as belonging to one of a number of classes. The classes correspond to respective types of image artifacts, as described below. In one example, six classes are illustrated corresponding to six types of image artifacts.

[0051] Recent research in computer vision provides object detectors that are robust and suitable for real-time applications, where CNNs use multi-scale deep object detection models to discriminate between different artifacts in real-time.

[0052] A general discussion of the machine learning techniques implemented in CNN follows.

[0053] Deep learning makes it possible to build object detectors that generalize traditional handcrafted “sliding window” object classification methods (e.g., as disclosed in (Ref. 11)).

[0054] Early attempts, including OverFeat (Ref. 12) and R-CNN (Ref. 13), demonstrated the ability of convolutional neural networks (CNNs) to learn relevant features and detect objects using a fixed number of pre-generated candidate object region proposals (Ref. 14). Faster R-CNN (Ref. 15) first introduced a fully trainable end-to-end network that yields an initial region proposal network and successive classification of proposed regions without intermediate processing. Because region proposal generation occurs sequentially prior to bounding box detection, this architecture is known as a two-stage detector. Although highly accurate, its main drawbacks are its slow inference and extensive training.

[0055] Alternatively, a CNN may simultaneously predict regions and region classes in a single stage. For example, You Only Look Once (YOLO, Ref. 16) simplified Faster R-CNN to simultaneously predict classes and bounding box coordinates using a single CNN and a single loss function, with superior performance and significantly faster inference time. This simultaneous detection is known as a one-stage detector. Compared to two-stage detectors, single-stage detectors suffer from two main problems: 1) high false positives due to the presence of objects of various sizes, and 2) a high initial number of anchor boxes requirement, which requires more accurate positive box mining. The former is corrected by predicting bounding boxes at multiple scales using feature pyramids 17-18. To solve the latter, RetinaNet 19 introduced a novel focal loss that adjusts the propagation loss to better focus on difficult misclassified samples. Recently, YOLOv3 20 simplified the RetinaNet architecture with further speed improvements. Bounding boxes are predicted at just three different scales (instead of five in RetinaNet) utilizing an objectness score and independent logistic regression to enable the detection of objects belonging to multiple classes, unlike the focal loss in RetinaNet. Taken together, Faster R-CNN, RetinaNet, and YOLOv3 define the current prior art detection envelope of accuracy vs. speed on the popular natural image benchmark COCO dataset (Ref. 21).

[0056] Due to these advantages, the CNN implemented in the CNN block 13 is advantageously of the type that predicts the region and the region class simultaneously in a single stage, including any of the techniques described above, but preferably spatial pyramid pooling is used to allow a single stage to capture both large and small sized artifacts. This allows real-time artifact detection, which is of practical value, and allows the detection results to be used, for example, to provide feedback to the endoscopist during data acquisition.

[0057] The classes corresponding to the different types of image artifacts that are detected are as follows: The classes are divided into three groups: 1. Motion blur class, which corresponds to the motion blur of the image; 2. At least one misexposure class corresponding to a type of misexposure of the frame; and 3. At least one noise artifact class corresponding to a type of image artifact that introduces noise into the frame. It can be divided into:

[0058] We now turn to these points.

[0059] The Motion Blur class corresponds to motion blur in the image, i.e. blur caused by the motion of the endoscope relative to the object being imaged. This is common in endoscopy and minimizing blur is a skill of the endoscopist. Often the blur is manifested as streaking in the image. Motion blur reduces the quality of the image by distorting the object being imaged.

[0060] The at least one misexposure class may be either or both of a class corresponding to overexposure of the image and a class corresponding to underexposure of the image. The endoscopy device 1 may typically perform exposure control using various techniques commonly known for cameras, such as changing the overall exposure settings of the image sensor that take into account the brightness of the captured image. However, despite this, it is still common for endoscopic video images to have overexposed and / or underexposed regions. Overexposure is observed as regions of bright pixels, also referred to below as "saturation". Underexposure is observed as regions of low contrast, also referred to below as "low contrast". Both of these types of misexposure reduce the quality of the image by making it difficult to discern object features.

[0061] At least one noise artifact class may generally correspond to any image artifact that introduces noise into the frame. Such image artifacts may be considered as additive noise to the video image, but are typically caused by physical obstructions or optical effects rather than by electrical or other noise occurring in the camera 3.

[0062] At least one noise artifact class may include a noise artifact class corresponding to image occlusion by air bubbles. Typically, water bubbles are present in endoscopy video images that distort the appearance of underlying tissue.

[0063] The at least one noise artifact class may include a noise artifact class corresponding to specular reflections. Typically, in endoscopy video images, mirror-like surface reflections cause artifacts. These are referred to hereafter in this specification as "specularity."

[0064] The at least one noise artifact class may include a noise artifact class corresponding to noise artifacts other than air bubbles or specular reflections, such as chromatic aberration or occlusion by debris, which are referred to hereinafter as "miscellaneous" artifacts.

[0065] The CNN block 13 outputs artifact data 14 that identifies classified regions of a frame of the video image signal. For each classified region, the artifact data 14 identifies both the region itself, i.e. the pixels that make up the region, and the class to which the region belongs, typically by means of a bounding box.

[0066] As an example, FIG. 2 illustrates some example frames of a video image signal overlaid with bounding boxes and labels identifying regions within those frames that were classified by CNN block 13 in this example where the CNN implemented was YOLOv3-SPP.

[0067] The quality score block 15 derives quality scores representative of the image quality of successive frames based on the classified regions identified by the artifact data 14. The quality score block 15 outputs quality data 16 representative of the quality score for each frame.

[0068] The quality score block 15 derives the quality score as follows:

[0069] Quality assessment is important in video endoscopy because image corruption greatly affects image analysis methods. However, it is likely that not all frames are corrupted to the same degree. Depending on the amount and type of artifacts present in a frame, realistic frame reconstruction may be possible. In this example, the quality score is a combined quality score based on all types of classified regions in the frame.

[0070] The quality score QS represents the quality of the frame and is based on a) the class (i.e., type of image artifact), b) the area, and c) the location within the frame of each classified region in the frame. A weight is assigned to each of these categories and the average weight is calculated as the quality score. Thus, the quality score may be calculated as follows: QS = [1-Σ(λ A W C W A +λ L W C W L )]0(1) where the summation is over the set of regions associated with each detected artifact in the image, W C are the class weights associated with the classes of the region, and W A is the area weight associated with the area of ​​the region, and W L is the location weight associated with the image location, and λ A , λ L is a relative weight that weights the relative contribution of area and location to each other. Thus, the quality score QS decreases from a value of 1 depending on the nature of the classified region, but is bounded by a lower value of zero.

[0071] Class weight W C , area weight W A , and the location weight W L The value of each of depends on the ease and importance of restoration associated with the class, area, and location of the region.

[0072] Class weight W C The value of depends on the ease of restoring each type of image artifact. For example, motion blur is easier to restore than types of image artifacts that introduce noise into a frame, even across the entire frame, so the class weight for the noise artifact class is higher than the class weight for the motion blur class. Similarly, underexposure is easier to restore than overexposure, so the class weight for the overexposed class is higher than the class weight for the underexposed class.

[0073] As an example of a suitable weighting scheme, the class weights W C The values ​​can be: motion blur 0.05, saturation 0.10, low contrast 0.05, air bubbles 0.10, specularity 0.20, and miscellaneous artifacts 0.50.

[0074] In equation (1), the term λ A W C W A Let the quality score QS represent a quality that decreases with the area of ​​the classified region in the frame.

[0075] Area weight W A The value of depends on the area and the ease of recovery associated with it. Therefore, the area weight W A The value of increases with area since the ease of restoration decreases with area.

[0076] As an example of a suitable weighting scheme, the area weight W A The value of W may simply scale linearly with the area of ​​the region, e.g., a constant fraction or percentage of the total area of ​​the frame. A The value of may increase with the area of ​​the region in a non-linear manner.

[0077] In equation (1), the term λ L W C W L makes the quality score QS based on the location of the classified region within the frame.

[0078] Location weight W L The value of W depends on the location of the region in the frame and the importance of the restoration associated with it. That is, restoration is of higher importance for regions in central locations for visual consistency. Thus, the location weight W L represents a quality that increases with proximity to the center of the frame and thereby decreases with the area of ​​the classified region within the frame; in particular, the quality will be lower for locations that are central in the image than for locations that are not central.

[0079] As an example of a suitable weighting scheme, the location weights W for different locations within a region are L The values ​​can be 0.5 in the center of the region, 0.25 on the left of the region, 0.25 on the right of the region, 0.25 on the top of the region, 0.25 on the bottom of the region, 0.125 on the top left of the region, 0.125 on the top right of the region, 0.125 on the bottom left of the region, and 0.125 on the bottom right of the region.

[0080] In a simple example, the relative weights λ A , λ L is a constant value, e.g., λ A = 0.5, λ L = 0.5. Alternatively, the relative weight λ A , λ L may further depend on the number of classified regions. For example, the relative weights λ A , λ L A weighting scheme where λ is a constant value emphasizes frames with few detected artifacts (e.g., less than five), especially when large area artifacts are present. Hence, in more sophisticated weighting schemes, such a constant λ A , λ L may be increased for a reduced number of regions, e.g., the value λ when the number of regions is 5 or more. A = 0.5, λ L = 0.5, and if the number of regions is less than 5, the value λ A = 1, λ L =1.

[0081] As an example, Figures 3 and 4 show two example frames that demonstrate how the quality score depends on the class, area, and location of the classified region. In the example where the CNN implemented was YOLOv3-SPP, the bounding boxes and labels identifying the regions in those frames that were classified by CNN block 13 are overlaid on the images, with a pie chart of the area of ​​the regions in the different classes shown next to them.

[0082] The example in Figure 3 is a frame with a large area of ​​the region classified as underexposed (labeled "contrast"). However, the quality score is significantly lower than the class weight W for the underexposed class despite its central location and large area. C Since ,is low, a relatively high value of 0.75 is taken, which is appropriate since the frame strength can be easily restored.

[0083] The example in FIG. 4 shows a frame with multiple regions of various artifacts and specularity. The quality score is calculated as the various artifacts and specularity have a high class weight W C , and is located in the center, which inhibits realistic frame reconstruction, and hence takes a relatively low value of 0.23.

[0084] Thus, in the above example, each quality score is a combined quality score based on all types of classified regions in the corresponding frame.

[0085] However, the quality score block 15 may alternatively or additionally derive a quality score for each type of artifact, which augments the information provided to the endoscopist and allows, for example, the endoscopy procedure to be adapted to reduce certain types of artifacts.

[0086] The quality score can be used in many ways, for example: The quality score can be provided to the endoscopist in real time, allowing the endoscopist to adapt the endoscopy procedure, for example, by repeating the endoscopy of the imaged portion of the vessel or by modifying the endoscopy device or procedure settings. Similarly, the quality score can be used during the training of endoscopists, as it provides reliable information regarding the quality of the video image signal being captured. In other situations, the quality score can be stored together with the video image signal as a record of quality, which can be used to evaluate the clinical usefulness of a particular video image signal or the performance of the endoscopist.

[0087] The image restoration block 20 of the analysis device 10 performs a restoration process on the video image signal represented by the video data 11. The image restoration block 20 provides a high quality restoration of many frames of the video image signal, thereby improving the overall quality and usefulness of the video image signal for diagnostic purposes. However, use of the image restoration block 20 is not essential and the quality scores derived above are useful in their own right.

[0088] The image restoration block 20 includes the following functional blocks:

[0089] The selection block 21 selects frames of the video image signal for restoration based on their quality scores, represented by the quality score data 16. It recognizes that the degree of corruption of a frame affects the degree to which restoration is worthwhile or possible. Thus, the selection block 21 classifies frames based on their quality score QS, which is compared to a threshold, for example as follows: QS<0.5: Frame classified as severely damaged; 0.5≦OS≦0.95: Frames classified as lightly damaged, and QS>0.95: Frame classified as high quality.

[0090] The threshold level may be user specified.

[0091] Frames classified as severely corrupted are not selected, but instead discarded without any further processing, on the basis that the frame cannot be restored to an acceptable level and therefore restoration is not worthwhile.

[0092] Frames classified as high quality are not selected because they are already of sufficient quality without the need for reconstruction. Alternatively, frames classified as high quality may be selected.

[0093] Frames that are classified as lightly corrupted are selected for reconstruction.

[0094] The selected frame is provided to an extension block 22, which extends the classified region in the selected frame to provide an extended region. For the motion blur class, the extended region may be the entire frame. For other classes, the extended region is extended from the classified region, but generally not the entire frame, unless the classified region is of a large enough area that the extension extends beyond the frame boundaries. The extension block 22 outputs extended data 23 representing the extended region. The extended data 23 is provided to subsequent blocks for further processing.

[0095] Three further blocks are present for carrying out the actual restoration process by processing the image frames so as to reduce the type of image artifact corresponding to the class of image artifact to which the classified region belongs and outputting restored video data 27 representing the restored video image signal.

[0096] These further blocks are the motion blur correction block 24, which implements a process for restoring areas classified in the motion blur class, the exposure correction block 25, which implements a process for restoring areas classified in one of the incorrect exposure classes, and the noise artifact correction block 26, which implements a process for restoring areas classified in one of the noise artifact classes. Thus, a separate process for each class is implemented in these three blocks to reduce the corresponding type of image artifact. Since a single frame may be corrupted by multiple artifacts and each artifact class may affect the endoscopic frame differently, the separate processes can provide an effective restoration for all types of image artifacts encountered in practice.

[0097] The restoration process is particularly effective due to the fact that it performs processing in a predefined order of the classes to which the regions belong: a motion blur class, followed by at least one misexposure class, followed by at least one noise artifact class. This is because:

[0098] Image restoration is the process of generating real and noise-free image pixels from corrupted image pixels. In endoscopic frame restoration, depending on the artifact type, the goal is either to generate a completely noise-free image or to pixel inpaint unwanted pixels using surrounding pixel information (Reference 22). In multi-class endoscopic artifact restoration, the restoration block performs the following tasks: 1) using frame deblurring in the motion blur correction block 24 when h(.) is unknown, i.e., a blind deblurring task; 2) minimizing the effect of contract imbalance (correction of over- and under-exposed regions in the frame, i.e., gamma correction in the exposure correction block 25; and 3) replacing specular pixels and pixels with imaging artifacts or debris by inpainting, i.e., correction of additive noise η(.) or a combined nonlinear function F(.) in the noise and artifact correction block 26.

[0099] Due to the high probability of multiple artifacts being present within a single frame, unordered restoration of these artifacts may further devastate the frame quality, therefore a sequential restoration process is used that takes into account the nature of each type of artifact.

[0100] Formulating the reconstruction of the true signal given a noisy and corrupted input image I as an optimization or estimation problem requires a well-motivated mathematical model. The variety of different types of artifacts induces a level of complexity that requires complex solutions to achieve high quality results. It is reasonable to assume that image noise is additive and to approximate motion blur as a linear convolution with an unknown kernel. In addition, contrast and pixel saturation problems can be formulated as a nonlinear gamma correction. Other remaining artifacts resulting from the combined process of these phenomena (e.g. specularity, bubbles, and imaging artifacts) can be assumed to be functions of the overall process.

[0101] If the video image signal is f(t), the corrupted noisy video frame can therefore be approximated as, I(t)=F[(h*f(t)+η) γ ] (2) where η is the additive noise induced by the imaging system, an approximation to the convolution-induced motion blur in h, γ captures over- and under-exposed regions, and F is a general nonlinear function that models capturing other artifacts (including specularity, air bubbles, and imaging artifacts), or combinations thereof, as well.

[0102] This model explains why the reconstruction of a video frame is structured into separate processing steps that are performed in a predefined order: the predefined order of corrections for motion blur, then misexposure, and then noise artifacts corresponds to the order in which the mathematical operations corresponding to the components of noise operate on the video image signal f(t) in equation (2), i.e., h, then γ, and then F. If the reconstruction were performed in a different order, the individual processes could corrupt each other, resulting in a less effective reconstruction or even a deterioration of the frame quality.

[0103] The processes performed in each of the motion blur correction block 24, the exposure correction block 25, and the noise and artifact correction block 26 are performed using deep learning models, in particular Generative Adversarial Networks (GANs), conditioning on prior information and embedding contextual losses.

[0104] Recently, GANs (Ref. 10) have been successfully applied to the image-to-image translation problem with limited training data. In a GAN as applied here, a generator G “generates” samples G(z) from a random noise distribution, z ∼ N(0, σ 2 I) having p noise (z) Meanwhile, a separate classifier network is used to model the real target images (x~p, assuming a non-zero mean Gaussian). data (x)) from fake images generated by the generator. The objective function V is therefore a min-max problem in this case. minG maxD V(D,G)=E x~pdata (x)[log D(x)]+E z~pnoise (z)[log(1-D(G(z)))] (3)

[0105] In practice, the generator model in equation (3) is highly non-convex, unstable, and slow to train because the samples are generated from random input noise. Various groups (23-26) have provided schemes to solve this problem, achieving improved reconstruction quality and numerical stability, as well as reduced computation time.

[0106] Therefore, one popular method to ensure the stability of the generator output applied here is to make prior information a prerequisite for the GAN (e.g., the class label “y” in CGAN (Reference 23)). The objective function Vcond for CGAN can be written as follows: minG maxD V cond (D,G)=E x,y~pdata (x,y)[logD(x|y)]+Ey~py ,z~ pz [log(1-D(G(z|y),y))] (4)

[0107] Another efficient method that can be applied here is to regularize the generator using a context loss (e.g., pix2pix (Reference 25), deblurGAN (Reference 27)). In (Reference 28), regularizing the discriminator and generator helped a lot to improve the visual quality.

[0108] Such a conditional generative adversarial model (CGAN), which embeds artifact-class-dependent contextual losses, is trained for effective restoration. Table 1 summarizes the restoration methods applied in each block and are described in further detail below. [Table 1]

[0109] The restoration process performed in the motion blur correction block 24 for the classified regions within the motion blur class shall now be considered.

[0110] Motion blur is a common problem in endoscopy videos. Unlike still images, motion blur is often non-uniform with unknown kernel h(.) in the video frame data (see equation (2)). Several blind deconvolution methods have been applied for motion blur removal. These range from classical optimization methods (Refs. 29-31) to neural network-based methods (Refs. 32, 33). Despite the superior performance of convolutional neural networks (CNNs) over classical methods, the main drawback of CNNs is that they require tuning a large number of hyper-parameters and large training datasets. Blind deconvolution can be posed as an image-to-image translation problem, where a blurred image is transformed into its corresponding unblurred image. Hence, the CGAN implemented in the motion blur correction block 24 uses a blind frame deblurring process that deconvolves a frame with an optimized kernel that represents the motion blur.

[0111] CGAN is applied with an l2-context loss (the squared difference between the generated image and the target / sharp image) and an additional l2 high-frequency loss as regularization. This is motivated by the fact that motion blur mainly affects image edges, which are few characteristic image pixels compared to the whole image. High-frequency images are first computed for both blurred and sharp images in the training data using iterative low-pass-high-pass filtering at four different scales (Ref. 34). These images are then used to provide additional information to the classifier regarding the generator behavior.

[0112] Thus, the CGAN implemented in the motion blur correction block 24 is trained by adapting equation (4) to become: minG maxD V' cond (D,G)=V cond +Σλ||x real i -G(z i |y i )|| i (5) where the summation is over the original and high frequency image pairs, λ=50, l=., and x real is the ground truth image for restoration (i.e., the sharp image in this case). Figure 5 graphically illustrates the operation of equation (5) and illustrates the context high frequency feature loss.

[0113] Minimizing equation (4) using the Jensen-Shannon (JS) divergence as in (Reference 10) can lead to problems such as mode collapse and vanishing gradients. As a result, (Reference 35) proposed using the Wasserstein distance with gradient penalty (WGAN-GP). The CGAN implemented in the motion deblurring block 24 uses a critique network based on WGAN-GP (Reference 27).

[0114] The restoration process performed in the exposure correction block 25 for the regions classified in one of the incorrect exposure classes shall now be considered. In the case of saturation or low contrast, a small or large distance between the light source and the imaged tissue may result in a large illumination change, which may result in saturation or low contrast. This motivates the role of the variable γ in equation (2). Therefore, the restoration process performed in the exposure correction block 25 performs a gamma correction process that is non-linear. This gamma correction process includes correction with an optimized gamma function.

[0115] Saturated or low contrast image pixels often occur over a large image area compared to specularity and affect the entire image globally. In addition, these illumination changes are usually observed more prominently in bright field (BF) modalities compared to other modalities. Compensation of the affected image pixels is a challenging problem depending on the size of the affected image area. The saturation restoration task is posed as an image-to-image translation problem, and the exposure compensation block 25 applies the same end-to-end CGAN technique used for motion blur removal described above, only with the l2-context loss to train the generator-discriminator network for saturation removal. Here, the l2-context loss is more preferred since it is desirable to capture the deviation between normal illumination and low contrast conditions with respect to saturation.

[0116] Due to the lack of any ground truth data for two different lighting conditions, the exposure compensation block 25 can be trained on a fusion dataset that includes natural scene images with diffuse (scattered light) and ambient (additional lighting to natural light that imparts areas with pixel saturation) illumination 1, as well as an endoscopic image pair simulated using cycleGAN-based style transfer (reference 24) (which can be trained separately on the other images using saturated and normal BF images).

[0117] To correct for color shifts due to the incorporation of natural images in the training set, the process implemented in the Exposure Compensation block 25 may further include a Color Transfer (CRT) applied to the generated frames. Given a source image Is, and a target image It for recoloring, the average (μ s ,μ t ) and the covariance matrix (Σ s ,Σ t ) can be matched through a linear transformation, for example as disclosed in [Reference 36]. A suitable linear transformation is given by: I' t =Σ s 1 / 2 Σ t-1 / 2 (I t -μ t )+μ s (6) In the formula, I' t is the recolored output. To avoid color retransfer from saturated pixel areas in the source, the mean and covariance matrix are calculated from image intensities <90% of the maximum intensity value.

[0118] To illustrate, Figure 6 shows an example of the results from the exposure correction block 25 using a trained GAN-based network (right) and after color shift correction (bottom), which show very close to the ground truth results.

[0119] To recover the low-contrast frames, a CGAN saturation network was trained on inverted image pairs of the same training dataset, i.e., the overexposed class (saturated) and the underexposed class (low contrast) are treated in the same manner, except that the training set is inverted.

[0120] The restoration process performed in the noise artifact correction block 26 for regions classified in one of the noise artifact classes shall now be considered.

[0121] Illumination inconsistencies and viewpoint changes can cause strong bright spots due to reflections from air bubbles and shiny organ surfaces, and watery materials can create multi-colored chromatic artifacts (referred to herein as "imaging or blending artifacts"). These inconsistencies appear as a combination of linear (e.g., additive noise η) and non-linear noise (function F(.)) in equation (2). Thus, the restoration process performed in noise and artifact correction block 26 includes inpainting. Such inpainting uses information from surrounding pixels, since prior information is used to replace saturated pixels in the affected regions.

[0122] The inpainting methods that can be applied are as follows: TV inpainting methods are popular for restoring images with geometric structure (Ref. 37), and patch-based methods for texture synthesis (Ref. 38). However, these methods are computationally expensive. Recent advances in deep neural networks have proven to recover visually plausible image structures and textures (Ref. 39) with near real-time performance. However, they are limited in size of the mask, or the number of unknown pixels in the image. In this context, GANs (Refs. 40-42) have shown success by providing faster and more coherent reconstructions, even with larger masks. Both contextual and generative losses have been used in these methods. Iizuka et al. (Ref. 41) and Yu et al. (Ref. 42) used local and global classifiers to improve the reconstruction quality. To enlarge the receptive field of the network, (Ref. 42) further used a coarse-fine network architecture using WGAN-GP instead of DCGAN in (Ref. 41). Additionally, an l1-discounted context (reconstruction) loss using a distance-based weight mask was used for added regularization

[42] . Due to the reduced training time and better reconstruction quality compared to

[41] , the noise and artifact correction block 26 may use the network proposed in

[42] for inpainting.

[0123] By way of example, an experiment using the analytical device 10 was carried out as follows.

[0124] Seven unique patient videos (gastroesophageal, selected from a large cohort of 200 videos) were used for training and 10 different videos were used for extensive validation. Experiments utilized well-established video quality assessment metrics to illustrate the effectiveness of the technique. In addition, the quality of the reconstructed frames was also evaluated by two experienced endoscopists. Scores based on visual improvement, importance, and the presence or absence of any artificially introduced artifacts in the reconstructed frames were provided by these experts.

[0125] The endoscopy dataset for artifact detection was as follows:

[0126] The artifact detection dataset consisted of a total of 1290 endoscopic images (resized to 512x512 pixels) from two operating modalities: normal bright field (BF) and narrow band imaging (NBI), sampled from seven unique patient videos selected from a cohort of 200 endoscopic videos for training data. Selection was based on the number of representative artifacts present in these videos and the texture variability of the underlying esophagus. Two experts annotated a total of 6504 artifacts using bounding boxes within which each annotation was classified.

[0127] A 90%-10% split was used to construct the training and testing sets for object detection, resulting in 1161 and 129 images and 5860 and 644 bounding boxes, respectively. In general, the training and testing data exhibit the same class distribution and similar bounding boxes (approximately square), but are either small with an average width less than 0.2, or large with a width greater than 0.5. If a given region contains multiple artifacts, multiple annotations are used.

[0128] We investigated Faster R-CNN, RetinaNet, and YOLOv3 architectures for artifact detection. Validated open source code is available for all of these architectures. Experimentally, we chose to incorporate YOLOv3 with spatial pyramid pooling (YOLOv3-SPP) due to its robust detection and improved inference time for endoscopic artifact detection. Spatial pyramid pooling allowed us to pool features from subimage regions utilizing single-stage CNN features computed at multiple scales from the YOLOv3 architecture. In addition to increasing the inference speed, incorporating spatial pyramid pooling reduced false positive detections compared to the classical YOLOv3 method (see Section IV-B). YOLOv3-SPP offered outstanding features for the accuracy-speed tradeoff, which is the main requirement for use in clinical environments.

[0129] Regarding quality assessment metrics, the standard mean average precision (mAP) and Intersection-over-Union (IoU) metrics were used to evaluate artifact detection. 25 , and mAP 50We quantitatively compared the detection results of all architectures using the mAP at IoU thresholds for 5%, 25%, and 50% positive matches, denoted as mAP, the average IoU between positive matches, the number of predicted boxes over the number of annotated boxes, and the average inference time for one image. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) measures were used to assess the quality of the deblurring methods. Further included to overcome the limitations of PSNR for the quantification of saturation and specularity restoration tasks are the more advanced Visual Information Fidelity (VIF, Ref. 44) and Relative Edge Coherence (RECO, Ref. 45) quality assessment metrics, which are independent of the type of distortion.

[0130] Table 2 shows artifact detection results on the test set using different neural network architectures. All timings are reported on a single 6GB NVIDIA GTX Titan Black GPU and are average times for a single 512x512 image (possibly rescaled relative to the input as shown) evaluated across all 129 test images. Total number of ground truth boxes = 644 boxes.

[0131] Table 2 shows that YOLOv3 variants outperform both Fast R-CNN and Retinanet. YOLOv3-spp (proposed) yields the best mAP of 49.0 and 45.7 at IoU thresholds of 0.05 and 0.25, respectively, with a detection rate 6 faster than Fast R-CNN (ref. 15). Retinanet exhibits the best IoU of 38.9, but note that IoU is subject to annotator variance in bounding box annotations, which may not be analogous to detector performance. [Table 2]

[0132] Fig. 7 shows the class-specific precision-recall curves for artifact detection, and Table 3 shows the class-specific average precision (AP) of different object detection networks. Fig. 7 and Table 2 show that in terms of class-specific performance, YOLOv3-spp is the best across detecting various artifacts and air bubbles (both of which are predominantly present in endoscopy videos) with average accuracies of 48.0 and 55.9, respectively. Faster R-CNN yielded the highest average accuracy for saturation (71.0) and blur (14.5), while RetinaNet and YOLOv3 outperformed for contrast (73.6) and specularity detection (40.0), respectively. It is worth noting that the proposed YOLOv3-spp yielded the second-best average precision scores for specularity (34.7), saturation (55.7), and contrast (72.1). [Table 3]

[0133] Thus, in summary, CNN block 13 yields the highest mean accuracy (mAP at 5% threshold) of 49.0 and the lowest computation time of 88 ms, enabling accurate real-time processing.

[0134] The restoration was carried out as follows:

[0135] In the motion blur correction block 24, the model was trained for 300 epochs on a paired blur-sharp dataset consisting of 10,710 multi-patient and multi-modal images (715 unique sharp images) with 15 different simulated motion trajectories for blurring (see reference 27).

[0136] A conditional generative adversarial network with added context and high-frequency feature losses was compared against deblurGAN (Ref. 27), the scale regression network-based SRN-DeblurNet (Ref. 46), and traditional TV-based methods (Ref. 31).

[0137] The TV regularization weight λ and the blur kernel r affect the quality of the recovered deblurred image (Ref. 31). λ = 103 and r = 2.3 were chosen after several iterative parameter setting experiments for the dataset. Retraining was performed for SRN-DeblurNet (Ref. 46) and deblurGAN (Ref. 27) on the same dataset used by the deblurring models. The frame deblurring method was quantitatively evaluated using five images and simulated test trajectories with visually large blur (shown in Table 4), as well as for three different test sequences with 30 images each (simulated motion blur trajectories, shown in Table 5). Table 4 shows the peak signal-to-noise ratio (PSNR) and structural similarity measure (SSIM) for randomly selected images with different motion blur. Table 5 shows the average PSNR and average SSIM for image sequences in the test trajectory with both the high frequency (HF) feature loss (as proposed) and the context loss only (as disclosed in (27)) added in the conditional GAN ​​model. Figure 8 illustrates the qualitative results of different deblurring methods on WL and NBI frames.

[0138] Table 4 shows that CGAN with l2-context loss and added high-frequency (HF) feature loss scored the highest PSNR and SSIM values ​​for all blurred frames, while the TV-based deconvolution method (ref. 31) resulted in the lowest PSNR and SSIM values ​​across all frames. An increase of almost 1 dB can be seen for the deblurGAN method (ref. 27) for frames #80, #99, and #103, while a 2 dB gain can be seen for #102, #116 for SRN-DeblurNet (ref. 46) using the proposed model. Overall, the proposed model provides the best results compared to the second best deblurGAN for the blurred image sequences in Table 5. This can also be seen qualitatively in Figure 8. It can be observed that SRN-DeblurNet distorts the image in the top right corner for both WL and NBI frames. [Table 4] [Table 5]

[0139] In exposure correction block 25, due to the lack of any ground truth data for the two different lighting conditions, the model was trained on a fusion dataset including 200 natural landscape images with diffuse (scattered light) and ambient (additional lighting to natural light giving areas with pixel saturation) illumination 1, as well as 200 endoscopic image pairs simulated using cycleGAN-based style transfer (reference 24) (which could be trained separately on another 200 images with saturated and normal BF images from 7 unique patients).

[0140] Results of treating saturation removal as a global problem and correcting the entire frame for overexposure as discussed above. Table 6 shows the average PSNR and average SSIM for 19 randomly selected saturated images in the simulated dataset using CycleGAN, l2-contexual CGAN, and post-processing using color retransfer (CRT) methods. These show that the restoration model demonstrates increased average values ​​across all tested metrics (PSNR, SSIM, VIF, and RECO). [Table 6]

[0141] The improvements after color transformation for visual quality metrics such as RECO (from 1.313 to 1.512) and VIF (from 0.810 to 0.818) illustrate the increased visual quality.

[0142] This is also evident in the qualitative results presented in Figure 9, which illustrates saturation and specularity correction. In particular, Figure 9 shows the frames and the associated RGB histograms top to the right of each frame. In the histograms in Figure 9, the vertical axis is frequency (0.00-0.10) and the horizontal axis is intensity (0-255).

[0143] The frames in the first column are three different original corrupted frames with contours of saturated regions in the first two frames, and contour specularity in the third frame. The frames in the second column are restored images restored by the exposure compensation block 25 using the trained end-to-end generator, presented in the second column. It shows that the heavily saturated image patches in the left and center frames are visibly removed by the trained generator while preserving the underlying image details.

[0144] The frames in the third column are the result of a simple rescaling of the corrected image intensities. This shows that simple contrast stretching as shown in Figure 9 (third column) by rescaling the CGAN reconstructed frames is not able to recover the original tones.

[0145] The frames in the fourth column are the results using color correction instead, which shows that the color transformation successfully restores the original color consistency in the CGAN reconstructed image without introducing additional saturation.

[0146] In the noise and artifact correction block 26, the bottleneck method was used to retrain the model initialized with pre-trained weights on the places2 dataset (ref. 43). To capture the large visual variability present in endoscopy images, 1000 images from 7 different patient endoscopy videos with quality scores >95% were used as “clean” images. 172 images were used as the validation set during training. Both the training and validation sets contained multimodal endoscopy video frames. During training and validation, masks of different patch sizes {(5 × 5), (7 × 7), (11 × 11), (13 × 13), ..., (33 × 33)} were randomly generated and used for restoration. A single image can have one or multiple generated masks for restoration.

[0147] Specularity and other local artifacts are removed based on inpainting as discussed above. To validate the inpainting method, a set of 25 images (clean) was used with randomly selected patches covering 5% and 12% of the total pixels of a 512 × 512 image size. Our CGAN-based model with l1-context loss model was compared with the widely used conventional TV-based and patch-based inpainting methods. Table 7 shows the average values ​​for PSNR, VIF (Ref. 44), and RECO (Ref. 45) metrics for restoration of missing pixels for masks covering 5% and 12% of the total image pixels (512 × 512 pixels) with 21 randomly sampled rectangular boxes on 20 randomly selected images from three different patient videos. The results in Table 7 show that the l1-context CGAN method has the best quality assurance values ​​for both VIF and RECO measures (VIF: 0.95, RECO: 0.992 for 5% mask pixels, and VIF: 0.883, RECO: 0.983 for 12% mask pixels). [Table 7]

[0148] Even though the TV-based inpainting method scores higher PSNR values ​​in both cases, it scores the smallest RECO values ​​(0.984 and 0.975 for the 5% and 12% cases, respectively) and has the highest computational cost (392 seconds). In contrast, the l1-contexual CGAN has the smallest computation time (2 seconds to load the trained model and apply it to an image on a GeForce GTX1080 Ti).

[0149] The qualitative results of the specularity and local artifact removal of the present invention on a real problematic gastro-esophageal endoscope frame are shown in FIG. 10 as follows.

[0150] The first row is the original corrupted frame including corrupted regions (specularity, imaging artifacts) detected by analysis device 10. As can be seen, the imaging artifacts (first and fourth columns) and specularity (second and third columns) cause large deviations in pixel intensity both locally relative to nearby pixels and globally relative to the uncorrupted image appearance.

[0151] The frames in the second row show the detected bounding boxes and the reconstruction areas marked with rectangles.

[0152] The frames in the third row are the same frames subjected to the inpainting process using a recent TV-based method. The frames in the fourth row are the same frames subjected to the inpainting process using the l1-contexual CGAN. As can be seen from the frames in the third and fourth rows, using the inpainting method the image is restored based on bounding box detection. The second best TV-based method in the third row produces blurred and non-smooth patches during the reconstruction of unknown pixels compared to the CGAN generative model shown in the fourth row.

[0153] The frames in the fifth row show the reconstructed areas using the TV-based process and the l1-contexual CGAN. A closer look at these regions shows that the local image structure is well preserved and there is a smooth transition from the reconstructed pixels to the surrounding pixels. An immediately noticeable ghosting effect can be observed in the frames in the second column using the TV-based method.

[0154] The artifact detection and restoration framework was evaluated on 10 gastroesophageal videos, each containing nearly 10,000 frames. For artifact detection, an object-likeness threshold of 0.25 was used to reduce overlaps within the detection boxes, and the QS value for restoring frames was set to 0.5. As a baseline, we trained a sequential 6-layer convolutional neural network (layers with 64 filters of size 3, 5 × 5, ReLU activation function, and batch normalization) with a fully connected final layer for binary classification on a set of 6000 manually labeled positive and negative images to decide whether to discard or keep a given input video frame. A threshold of 0.75 was set for the binary classifier to keep only frames of sufficient quality.

[0155] Thus, restoration of classified regions of each type of image artifact shows quantitative and qualitative improvements for the frame restoration task, achieving, in particular, improvements in both PSNR and SSIM metrics for blur and saturation, as well as significant improvements to visual similarity metrics for specularity and various other artifact removal.

[0156] Fig. 11 illustrates frame restoration of three clinical endoscopy video image signals. The temporal maps show the frames of the video image signals selected for restoration by the binary deep classifier and selection block 21. The bar charts show the proportion of frames of the video image signals selected for restoration by the two methods. The pie charts show the proportion of each artifact type present in each video image signal.

[0157] As shown in FIG. 11, analysis apparatus 10 is successful in retaining the majority of the frames compared to a binary decision.

[0158] The enhanced video was again fed to a CNN-based binary classifier which resulted in fewer frame rejections than the raw video. As a result, the resulting video is more continuous compared to the equivalent binary cleaned video using the raw video. For example, in the video image signal 3, the video after frame rejection based on the binary classifier directly results in many obvious abrupt transitions that can be detrimental to post-processing algorithms, since only 30% are retained. In comparison, the analysis device 10 preserves 70% of the frames, i.e., a frame restoration of nearly 40%. Qualitatively across all 10 endoscopic videos tested, 25% more video frames were restored, preserving an average of 68.7% of the 10 videos considered.

[0159] The clinical relevance of the reconstructed video image signal was tested as follows.

[0160] Twenty high-quality frames selected from the ten test videos were corrupted by blur, specularity, saturation, and various artifacts. The restoration process described above was then applied to these frames. Two expert endoscopists were asked to independently score these restorations in comparison with the original high-quality images and the corresponding videos. The scores, ranging from 0 to 10, were based on the following: 1) the addition of unnatural distortions was assigned a negative score, and 2) the removal of distortions was assigned a positive score. The average scores obtained were: blur 7.87, specularity or various artifacts 7.7, and saturation 1.5. Significant restorations were obtained for blur and specularity or various artifacts. However, saturation correction was not satisfactory for the experts (according to feedback comments), mostly due to the loss of 3D information, even though the visual coherence was improved. (References) (Reference 1) S.Chikkerur, V.Sundaram, M.Reisslein, and びLJKaram, "Objective video quality assessment methods: A classification, review, and performance comparison", IEEE Trans.Broadcast., vol.57, no.2, pages 165~182, 2011 (Reference 2) DPMenor, CAMello, and C. Zanchettin, "Objective video quality assessment based on neural networks", Procedia Comput.Sci., vol.96, pages 1551~1559, 2016 (Reference 3) S.Ali, C.Daul, E.Galbrun, F.Guillemin, and W.Blondel, "Anisotropic motion estimation on edge preserving Riesz wavelets for robust video mosaicing", Patt.Recog., vol.51, pp.425-442, 2016 (Reference 4) H. Liu, WS Lu, and MQH Meng, "De-blurring wireless capsule endoscopy images by total variation minimization", PACRIM. IEEE, August 2011, pp. 102-106 (Reference 5) T. Stehle, "Removal of specular reflections in endoscopic images", Acta Polytechnica, vol. 46, no. 4, 2006 (Reference 6) S. Tchoulack, J.P. Langlois, and F. Cheriet, “A video stream processor for real-time detection and correction of specular reflections in endoscopic images,” Workshop on Circuit and Syst. and TAISA Conf. IEEE, 2008, pp. 49-52. (Reference 7) M. Akbari, M. Mohrekesh, S. Soroushmehr, N. Karimi, S. Samavi, and K. Najarian, “Adaptive specular reflection detection and inpainting in colonoscopy video frames,” arXiv preprint arXiv:1802.08402, 2018. (Reference 8) A. Mohammed, I. Farup, M. Pedersen, O. Hovde, and S. Yildirim Yayilgan, "Stochastic capsule endoscopy image enhancement", J. of Imag., vol.4, no.6, 2018 (Reference 9) F. Isabel, B. Sebastian, R. Carina, W. J. We, S. Stefanie, I. Funke, S. Bodenstedt, C., J. Weitz, S. Speidel, "Generative adversarial networks for specular highlight removal in endoscopic images," Proc. SPIE, vol. 10576, 2018, pp. 10 576-10 576 - 9. (Reference 10) IJ Goodfellow, J. Pouget-Abadie, M. Mirza, B. (Reference 11) P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," CVPR. IEEE, 2001, pp. 511-518. (Reference 12) P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. Le-Cun, "Overfeat: Integrated recognition, localization and detection using convolutional networks," arXiv preprint arXiv:1312.6229, 2013. (Reference 13) R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation", CVPR.IEEE, 2014, pp. 580-587 (Reference 14) JRUijlings, KEVan De Sande, T.Gevers, and AWSmeulders, "Selective search for object recognition", Int.J.of Comput.vision, vol.104, no.2, pp. 154-171, 2013. (Reference 15) S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” NIPS, 2015, pp. 91–99. (Reference 16) J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection", CVPR.IEEE, 2016, pp. 779-788. (Reference 17) K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition", ECCV. Springer, 2014, pp. 346 - 361 (Reference 18) T.-Y. Lin, P. Dollar, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie, "Feature pyramid networks for object detection", CVPR. IEEE, 2017, pp. 936 - 944 (Reference 19) T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dolla’r, "Focal loss for dense object detection", arXiv preprint arXiv:1708.02002, 2017 (Reference 20) J. Redmon and A. Farhadi, "Yolov3: An incremental improvement", arXiv preprint arXiv:1804.02767, 2018 (Reference 21) T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick, "Microsoft coco: Common objects in context", ECCV. Springer, 2014, pp. 740 - 755 (Reference 22) C. A. Z. Barcelos and M. A. Batista, "Image restoration using digital inpainting and noise removal", Image and Vision Comput., vol. 25, no. 1, pp. 61 - 69, 2007 (Reference 23) M. Mirza and S. Osindero, "Conditional generative adversarial nets", CoRR, vol. abs / 1411.1784, 2014 (Reference 24) J.-Y. Zhu, T. Park, P. Isola, and A.A.E. Fros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," ICCV. IEEE, 2017, pp. 2242-2251. (Reference 25) P. Isola, J.-Y. Zhu, T. Zhou, and A.A.E. Fros, “Image-to-image translation with conditional adversarial networks,” CVPR, 2017. (Reference 26) T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation”, CoRR, 2017 (Reference 27) O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin, and J. Matas, “Deblurgan: Blind motion deblurring using conditional adversarial networks,” CoRR, vol. abs / 1711.07064, 2017. (Reference 28) B. Duhyeon and S. Hyunjung, "Improved training of generative adversarial networks using representative features," CoRR, vol. abs / 1801.09195, 2018. (Reference 29) H. Tong, M. Li, H. Zhang, and C. Zhang, “Blur detection for digital images using wavelet transform,” ICME. IEEE, 2004, pp. 17-20. (Reference 30) L. Xu, S. Zheng, and J. Jia, "Unnatural l0 sparse representation for natural image deblurring", CVPR. IEEE, 2013, pp. 1107 - 1114 (Reference 31) P. Getreuer, "Total Variation Deconvolution using Split Bregman", Image Process. On Line, vol. 2, pp. 158 - 174, 2012 (Reference 32) A. Chakrabarti, "A neural approach to blind motion deblurring", ECCV. Springer, 2016, pp. 221 - 235 (Reference 33) S. Nah, T. H. Kim, and K. M. Lee, "Deep multi - scale convolutional neural network for dynamic scene deblurring", CVPR. IEEE, 2017, pp. 257 - 265 (Reference 34) A. Buades, T. Le, J.-M. Morel, and L. Vese, "Cartoon + Texture Image Decomposition", Image Process. On Line, vol. 1, 2011 (Reference 35) M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks", ICML, 2017, pp. 214 - 223 (Reference 36) A. P. Hertzmann, "Algorithms for rendering in artistic styles", Ph.D. dissertation, New York University, Graduate School of Arts and Science, 2001 (Reference 37) J. Shen and T. F. Chan, "Mathematical models for local nontexture inpaintings", SIAM J. of Appl. Math., vol. 62, no. 3, pp. 1019 - 1043, 2002 (Reference 38) A. A. Efros and W. T. Freeman, "Image quilting for texture synthesis and transfer", SIGGRAPH.ACM, 2001, pp. 341 - 346 (Reference 39) R. Kohler, C. Schuler, B. Scholkopf, and S. Harmeling, "Mask - specific inpainting with deep neural networks", GCPR.Springer, 2014, pp. 523 - 534, lNCS (Reference 40) D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting", CVPR.IEEE, 2016, pp. 2536 - 2544 (Reference 41) S. Iizuka, E. Simo - Serra, and H. Ishikawa, "Globally and Locally Consistent Image Completion", SIGGRAPH, vol. 36, no. 4, 107:1 - 107:14, 2017 (Reference 42) J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, "Generative image inpainting with contextual attention", CoRR, vol. abs / 1801.07892, 2018 (available online: http: / / arxiv.org / abs / 1801.07892) (Reference 43) B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, "Places: A 10 million image database for scene recognition", IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 6, pp. 1452 - 1464, 2018 (Reference 44) H. R. Sheikh and A. C. Bovik, "Image information and visual quality", IEEE Trans. Image Process., vol. 15, no. 2, pp. 430 - 444, 2006 (Reference 45) V. Baroncini, L. Capodiferro, E. D. D. Claudio, and G. Jacovitti, "The polar edge coherence: A quasi blind metric for video quality assessment", EUSIPCO, August 2009, pp. 564 - 568 (Reference 46) X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, "Scale - recurrent network for deep image deblurring", CVPR. IEEE, 2018, pp. 8174 - 8182 (Reference 47) A. Newson, A. Almansa, Y. Gousseau, and P. Pérez, "Non - Local Patch - Based Image Inpainting", Image Process. On Line, vol. 7, pp. 373 - 385, 2017 (Reference 48) G. R. Cameron et al., "Detection and staging of esophageal cancers within Barrett’s esophagus is improved by assessment in specialized Barrett’s units", Gastrointestinal endoscopy 80.6 (2014), pp. 971 - 983 (Reference 49) DW Scholvinck et al., "Detection of lesions in dysplastic Barretta Az's esophagus by community and expert endoscopists," Endoscopy 49.02 (2017), pp. 113-120

Claims

1. 1. A method for analyzing a video image signal comprising successive frames of imaging an endoscopy procedure, comprising: Analysing the video image signal using machine learning techniques to classify regions of the frames as belonging to one of a number of classes corresponding to different types of image artifacts, the classes comprising: a motion blur class corresponding to the motion blur of the image; at least one mis-exposure class corresponding to a type of mis-exposure of the image; and at least one noise artifact class corresponding to a type of image artifact that introduces noise into the frame; and deriving at least one quality score representative of an image quality of the successive frames based on the classified regions.

2. The method of claim 1 , wherein the at least one misexposure class includes two misexposure classes corresponding to overexposure of the image and underexposure of the image, respectively.

3. The at least one noise artifact class may include: a noise artifact class corresponding to occlusions of said image by air bubbles; a noise artifact class corresponding to specular reflections, The method of claim 1 or 2, including one or more of the noise artifact classes corresponding to noise artifacts other than air bubbles or specular reflections.

4. The method of claim 1 , wherein the noise artifacts other than air bubbles or specular reflections include any one or more of the following: chromatic aberration or occlusion by debris.

5. The method of claim 1 , wherein the at least one quality score is based on an area of ​​the classified region, the quality score representing a quality that decreases with area.

6. 6. The method of claim 1 , wherein the at least one quality score is based on a location of the classified region, the quality score representing a lower quality for central locations in the image than for non-central locations.

7. The method of claim 1 , wherein the at least one quality score comprises a combined quality score based on all types of the classified regions.

8. The method of claim 7 , wherein the combined quality score is based on the area of ​​the classified regions of each class weighted by a weight that is class dependent, the quality score representing quality that decreases with area.

9. 9. The method of claim 7 or 8, wherein the combined quality score is based on the location of the classified regions of each class weighted by a weight that is dependent on the class, the quality score representing lower quality for central locations in the image than for non-central locations.

10. The method according to claim 8 or 9, wherein the weights are further dependent on the number of classified regions.

11. 11. The method of claim 1, further comprising the step of restoring at least some of the image frames by processing the image frames to reduce the type of image artifact that corresponds to the class of image artifact to which the classified region belongs.

12. The method of claim 11 , wherein the restoring step includes processing the image frames with a separate process for each class to reduce the corresponding type of image artifact.

13. The method of claim 12 , wherein the separate processes are performed in a predefined order of the class to which the region belongs.

14. The method of claim 13 , wherein the predefined order includes the motion blur class, followed by the at least one misexposure class, followed by the at least one noise artifact class.

15. 15. The method of claim 12, wherein the process for the motion blur class includes a blind frame deblurring process that can deconvolve the frame with an optimized kernel that represents the motion blur.

16. The method of claim 15 , wherein the blind frame deblurring process uses a generative adversarial network.

17. The generative adversarial network is conditioned on prior information and / or ２ The method of claim 16, further comprising embedding a context loss, which may include a context loss and a high frequency context loss.

18. The method of claim 12 , wherein the process for at least one erroneous exposure class includes a gamma correction process.

19. 19. The method of any one of claims 12 to 18, wherein the gamma correction process is non-linear and / or comprises correction with an optimized gamma function.

20. The method of claim 19 , wherein the gamma correction process uses a generative adversarial network.

21. The generative adversarial network is conditioned on prior information and / or ２ The method of claim 20, further comprising embedding a context loss, which may be a context loss.

22. 22. The method of claim 18, wherein the process for at least one misexposure class further comprises color transfer.

23. 23. The method of claim 12, wherein the process for at least one noise artifact class comprises an inpainting process.

24. The method of claim 23 , wherein the inpainting process uses a generative adversarial network.

25. The generative adversarial network is conditioned on prior information and / or １ The method of claim 24, further comprising embedding a context loss, which may be a context loss.

26. The method of claim 11 , further comprising the step of selecting the image frames to be reconstructed based on the quality scores.

27. 27. The method of claim 1, wherein the machine learning technique is a convolutional neural network.

28. 28. The method of claim 27, wherein the convolutional neural network is configured to simultaneously predict the region and the class of the region in a single stage, which may use spatial pyramid pooling.

29. A computer program executable by a computing device and configured, when executed, to cause the computing device to carry out a method according to any one of claims 1 to 28.

30. 30. A computer readable storage medium storing the computer program of claim 29.

31. 1. An analysis apparatus for analyzing a video image signal comprising successive frames of an endoscopy procedure, comprising: a machine learning block configured to analyze the video image signal using machine learning techniques to classify regions of the frame as belonging to one of a plurality of classes corresponding to respective types of image artifacts, the classes comprising: a motion blur class corresponding to the motion blur of the image; at least one mis-exposure class corresponding to a type of mis-exposure of the image; and a machine learning block including at least one noise artifact class corresponding to a type of image artifact that is noise; a quality score block configured to derive a quality score representative of an image quality of the successive frames based on the classified regions.

Images