Automatic selection of endoscopy images

Abstract

According to an aspect of the invention, it is provided a computer implemented method for selecting a final image that best shows an abnormality of a part of a human or animal body, from a plurality of endoscopy images of the abnormality.

Description

[0001] AUTOMATIC SELECTION OF ENDOSCOPY IMAGES

[0002] TECHNICAL FIELD

[0003] The technical field of the invention is the one of medical imaging

[0004] The present invention concerns a method for selecting, from a plurality of images, a final image of an abnormality of an organ of a human or animal body.

[0005] BACKGROUND OF THE INVENTION

[0006] Endoscopy is widely used in medical imaging for inspecting the inside of an animal or human body. This imaging technique relies on acquiring with a camera many images of the cavity of the body that is under examination. However, depending on the part of the body that is being examined, the acquisition produces several thousand images, or even several tens of thousands of images.

[0007] Browsing and analysing such a large number of images is tedious for the practitioner. This is especially the case when an anomaly, such as an abnormality, is looked for into these many images, for example in order to establish the cause of a given symptom. Moreover, when browsing many images, the attention level significantly decreases with the number of images that have already been analysed, consequently increasing the risk of missing a relevant information.

[0008] Approaches are known from the art that reduce the number of images to be browsed by the practitioner, by discarding irrelevant images that do not content any abnormality, i.e., that do not show any abnormality. These approaches are, however, insufficient as there still remains lots of images, sometimes more than a thousand, to be browsed, each showing relevant information on abnormalities. The attention level then still significantly decreases with the number already analysed, all the more when several different abnormalities are present.

[0009] Therefore, there exists a need for further reducing the number of images to be browsed by the practitioner when analysing an endoscopy investigation.

[0010] SUMMARY OF THE INVENTION

[0011] An object of the invention is to provide a method that automatically selects a best image of an abnormality, from a plurality of images of said abnormality, where this best image is an image from the plurality of images that shows best the abnormality under investigation.

[0012] To this end, according to a first aspect of the invention, it is provided a computer implemented method for selecting a final image of an abnormality of a part of a human or animal body, from a plurality of images of the abnormality, each having been acquired with an endoscopy device, the method comprising:

[0013] Determining a plurality of intermediate images from the plurality of images, each image of the plurality of intermediate images being an image of the plurality of images that corresponds to a relevant view of the abnormality;

[0014] From the plurality of intermediate images, selecting the final image based on at least one property determined for each image of the plurality of intermediate images.

[0015] By “endoscopy” is meant any kind of endoscopy technique, applied in any medical specialty, for example digestive endoscopy, ENT (ear, nose, throat) endoscopy, broncho-pulmonary endoscopy, urogenital endoscopy, Laparoscopic endoscopy, etc, and for example any kind of endoscope, such as rigid endoscope, flexible endoscope and / or capsule endoscope.

[0016] By “part” of the human or animal body is meant any area of a human or animal body that can be inspected via endoscopy. For example, a part of the body can be an organ, a tissue, a bone, etc., or a part of it. Examples of such parts are: the oesophagus, the uterus, an ear, whole a part of the digestive tract, etc.

[0017] By “abnormality” is meant a morphological alteration of a tissue, an organ or the content of an organ or tissue. Such alteration appears as an anomaly in the images acquired via an endoscope. The abnormality can be of any nature and / or type, depending on the part of the body that is inspected. Examples of abnormality can be: a tumour, an ulceration, a polyp, an, an erythematous patch, a vascular abnormality (such as an angiectasia, a lymphangiectasia, an angiodysplasia or a phlebectasia), a red spot, a cyst, an active bleeding, a stenosis, fresh blood, clots, melena, a foreign body, a cancer, etc.

[0018] By “corresponding to a relevant view of the abnormality” is meant that a considered image is in agreement with one or more conditions. Those conditions can be based on one or more morphological and / or physiological property of the abnormality, and / or based on one or more features of the abnormality that is present in the considered image. Otherwise said, the “relevant view of an abnormality” is a class, from a classification predefined by the conditions. This class corresponds to images in which the abnormality is seen and for which the view from which the abnormality is observed allows to reliably analyse the abnormality. For example, a relevant view, in the case of capsule endoscopy to investigate a suspected small bowel bleeding, corresponds to the P1 and P2 classes of the Saurin’s method classification (see Saurin, J. C. & Pioche, M., Why should we systematically specify the clinical relevance of images observed at capsule endoscopy? Endosc. Int. Open 2, E88-89 (2014)). Another classification, for example when investigating the large intestine can be the Paris classification (see Endoscopic Classification Review Group, Update on the Paris classification of superficial neoplastic lesions in the digestive tract, Endoscopy, vol. 37(6), 2005).

[0019] By “property” is meant a metric, or a feature or a characteristic of the images of the plurality of intermediate images. The property is the same of each image. The determination of the property then provides a value of said property, such as an alphanumerical value, for each image of the plurality of intermediate images.

[0020] Thanks to the invention, it is possible to extract from a plurality of images of an abnormality, the image that best shows the abnormality, based on predefined relevance criteria. The practitioner now only has one image, that contains the abnormality, to analyse.

[0021] Advantageously, several sequences of images of one or more abnormalities can be processed according to the present method. The practitioner would then only need to analyse one image of each abnormality per sequence of images, therefore reducing the time needed to analyse them and reducing the risk a missing a relevant information. For example, the thousands or tens of thousands of images can be reduced to less than three hundreds, or less than two hundreds, or less than a hundred of even less than fifty images to be analysed by the practitioner.

[0022] The final image selection is divided into two main steps. The first one applies a filtering among the sequence of images to discard images that do not contain relevant information concerning the abnormality, notably due to the viewing angle from which the abnormality is observed. The second one applies another filtering that selects the final image according to preselected image properties.

[0023] This method is also compatible with almost real time image processing.

[0024] Apart from the characteristics mentioned above in the previous paragraph, the method according to a first aspect of the invention may have one or several complementary characteristics among the following characteristics considered individually or in any technically possible combinations.

[0025] According to an embodiment, the selected final image is sent and / or added to a medical report.

[0026] By “medical report” is meant the medical report related to the endoscopy of the patient (human or animal) whose part of the body has been the subject of the investigation.

[0027] In this embodiment, the best image of the abnormality is automatically added to the medical report of the patient without the need of the practitioner to do this by himself. Advantageously, the medical report can be filed with this best image while the practitioner or the operator is conducting the endoscopy.

[0028] According to an embodiment, the plurality of images of the abnormality is obtained by:

[0029] Extracting the plurality of images of the abnormality from a plurality of endoscopy images that have been acquired with the endoscopy device, each image of the plurality of images of the abnormality being an image of the plurality of endoscopy images that is an image of the abnormality.

[0030] By “an image being an image of the abnormality” is meant that the abnormality can be seen in the image, among other elements of the body. This image then at least comprises a picture of the abnormality. In other words, this image comprises a representation of the abnormality.

[0031] Thanks to this embodiment, the plurality of images of the abnormality is formed from endoscopy images that also comprise images that are not images of the abnormality. According to an embodiment, the plurality of images of the abnormality is a sequence of images that comprises at least two images of the plurality of endoscopy images that are consecutive in time, respectively to the acquisition of the plurality of endoscopy images.

[0032] By “consecutive in time” is meant that the images of the sequence of images have been acquired successively during the investigation of the part of the body. This means that there is no other image that has been acquired in-between the acquisition instants of two consecutive images of the sequence of images.

[0033] According to an embodiment, an intermediate image of the plurality of intermediate images correspond to a relevant view of the abnormality when it satisfies one or more conditions related to a classification standard.

[0034] The best image of the abnormality is then chosen among images that are considered as relevant for this abnormality, according to widely used standards. This makes the method according to the invention applicable wherever the endoscopy has been conducted.

[0035] According to an embodiment, the determination of the plurality of intermediate images is carried out via a classification algorithm.

[0036] According to an embodiment, the classification algorithm has been trained from a classification training database comprising a plurality of classification training images, each classification training image of the plurality of classification training images being associated with a label that indicates whether said image corresponds to a relevant view of the abnormality.

[0037] By “classification algorithm” is meant an automatic algorithm that is adapted for image classification and that is trained to classify the images of the sequence of images. It is noted that the classification algorithm can either be trained before carrying out the steps of the method according to the first aspect of as a preliminary step of said method.

[0038] By “classification training database” is meant a database of images that have been acquired and / or numerically constructed (for example, via a simulation software) and labelled in order to train the classification algorithm from it.

[0039] By "learning" or "training" is meant a mechanism adapted to modify, the parameters of a learning model. This modification can be achieved by implementing an iterative process that aims at minimizing a cost function, defining a distance between data produced by the learning model and data used as a reference.

[0040] By “label" is meant an annotation assigned to each image of the classification training database. The annotation can be, for example, a text, another image, a symbol, a number, etc. The annotation can correspond to a characteristic of the image and can be determined automatically determined or can be manually determined by an operator. The annotation is used to give the learning model the characteristic(s) of the images that must be learned and generated by the model when a new image is presented to it for processing.

[0041] The use of such a classification algorithm allows for almost instant classification of the images of the sequence of images, with a low error rate.

[0042] According to an embodiment, the at least one property is a visual property.

[0043] By “visual property” is meant a property of the image that can be perceived with the eye. In other words, the property is predefined so that it reliably corresponds to one or more psychovisual selection criteria that would use a practitioner to analyse the images using his own eyes.

[0044] According to an embodiment, the at least one property is one more of:

[0045] - A confidence indicator;

[0046] - A size of a frame around the abnormality;

[0047] - A position of the centre of the frame in the image;

[0048] - A colorimetric indicator.

[0049] By “confidence indicator” is meant one or more alphanumerical characters associated with each image of the plurality of images of the abnormality. The confidence indicator indicates how reliable an image is, regarding an abnormality that can be seen in it. The confidence indicator can be obtained through various techniques known from the art. For example, the confidence indicator can be driven from a probability of the abnormality in the considered image to belong to one or more abnormality classes, according to a predefined abnormality classification or disease classification

[0050] By “size of a frame” is meant the area of a frame that is artificially drawn onto the image and that fits to the perimeter of the abnormality in the image. The frame can be of any geometrical form. For example, it can be a simple shape such as a rectangle or a circle, or it can be of more complex shape, for example comprising five or more sides. The frame can also be the contour of the abnormality in the image. The frame can be constructed via any known technique from the art.

[0051] By “position of the centre of the frame” is meant the coordinates of the centre of the shape of the frame in the reference frame of the considered image. Alternatively, the property can be the distance between the centre of the frame and the centre of the considered image.

[0052] By “colorimetric indicator” is meant a metric that relates to a colour, a brightness, a contrast or any other colour-related feature of the abnormality in the image.

[0053] According to an embodiment, the final image is the image from the plurality of intermediate images for which the at least one property is in accordance with a selection criterion related to said property.

[0054] By “selection criterion” is meant a condition that the property has to be in accordance with, in order to the considered image being selected as the final image. This selection criterion depends on the property that is considered. For example, the selection criterion can be based on a distance of the property to a predefined value, can be based on a threshold that the value of the property must be above or under, can be that the value of the property of a given image is the highest among the property values of the other images, etc. The selection criterion corresponding to a property can also be defined depending on the number of properties, when several properties are considered. In this case, one property corresponds to one selection criterion, and vice-versa. A weighting can be applied to the selection criteria, in order to determine the final image, when several properties are considered.

[0055] According to an embodiment, the method further comprises:

[0056] Updating the selection criterion based on an input provided by an operator. By “updating” is meant to modify the selection criterion based on the input provided by the operator. This modification depends on the type of selection criterion that is concerned. For example, the modification can be the increase or decrease of a threshold, the increase or decrease of the weighting associated with one or more of the several selection criteria, etc.

[0057] By “input” is meant any kind of data provided by the operator that can be used to modify the selection criterion.

[0058] By “operator” is meant the individual that has conducted the endoscopy, which can be the practitioner himself or a qualified endoscopy individual.

[0059] Therefore, it is possible to adapt the outcome of the method, depending on preferences of the operator, indicated via the input.

[0060] According to an embodiment, the input is an image selected by the operator from the plurality of intermediate images that is different from the final image.

[0061] By “image selected by the operator” is meant an image that is chosen by the operator when reviewing the other images of the plurality of intermediate images.

[0062] According to an embodiment, the selection of the final image is carried out via a machine learning model.

[0063] According to an embodiment, the machine learning model is trained from a selection training database comprising several pluralities of selection training images, each selection training image of each plurality of selection training images being associated with a label that indicates whether or not said image is the final image of the plurality of selection training images it belongs to.

[0064] By “machine learning model” is meant an automatic algorithm that is adapted for image processing and that is trained to determine the final image among a plurality of intermediate images. It is noted that the machine learning model can either be trained before carrying out the steps of the method according to the first aspect of as a preliminary step of said method.

[0065] By “selection training database” is meant a database of images that have been acquired and / or numerically constructed (for example, via a simulation software) and labelled in order to train the machine learning model from it. The use of such a machine learning model allows for almost instant determination of the final image, with a low error rate.

[0066] According to a second aspect of the invention, it is provided a processing module adapted to carry out the steps of the method according to the first aspect

[0067] According to a third aspect of the invention, it is provided a device comprising a processing module according to the second aspect.

[0068] According to a fourth aspect of the invention, it is provided a computer program product comprising instructions which, when the program is executed by a computer, causes the computer to carry out the method according to the first aspect.

[0069] According to a fifth aspect of the invention, it is provided a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to first aspect.

[0070] The invention and its various applications will be better understood by reading the following description and examining the accompanying figures.

[0071] BRIEF DESCRIPTION OF THE FIGURES

[0072] The figures are presented for information purposes only and in no way limit the invention.

[0073] - Figure 1 is a flow chart of a method according to an embodiment.

[0074] - Figure 2 represents images of an abnormality from a sequence of images, according to an embodiment.

[0075] - Figure 3 represents a device for implementing a method according to the invention, according to an embodiment.

[0076] - Figure 4 represents capsule endoscopy images, according to an exemplary embodiment, processed by carrying out the steps of a method according to the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0077] Some embodiments of devices and methods in accordance with embodiments of the present invention are now described, byway of example only, and with reference to the accompanying drawings. The description is to be regarded as illustrative in nature and not as restrictive.

[0078] The invention relates to an endoscopy images processing mechanism that allows to obtain the best image of an abnormality that is detected in the images that have been acquired. This processing mechanism is based on carrying out two main processing steps: first for determining the relevant images of an abnormality from a sequence of images of an abnormality, and second to determine which image is the best image of the abnormality, among the relevant images.

[0079] As such, a first aspect of the invention relates to a computer implemented method, for selecting a best image, called “final image”, of an abnormality of a part of a human or animal body. The final image is selected from a plurality of images of the abnormality that have been acquired with an endoscopy device. A flowchart of the steps of this method 100 is represented in Figure 1.

[0080] In the following, the implementation of this method 100 is exemplified with a capsule endoscopy application. The plurality of images of the abnormality has been acquired with a capsule endoscopy device. The plurality of images of the abnormality herein corresponds to images acquired when the device was investigating the small bowel. In this example, the investigated abnormality is a gastrointestinal angiodysplasia or AGD.

[0081] The method 100 comprises a step 110 of extracting a sequence of images from a plurality of endoscopy images that have been acquired by the endoscopy device. Indeed, the plurality of endoscopy images can comprise unwanted images, such as images of another abnormality and / or images that do not show an abnormality. This step 110 is carried out to apply a filtering of the plurality of endoscopy images to discard these unwanted images. Said differently, carrying out this step 110 produces a sequence of images, wherein each image of the sequence is an image of the abnormality under investigation, i.e. , the AGD here. The extraction of the sequence of images can be carried out by using known techniques from the art. For example, the sequence of images can be extracted so that it only comprises the images of the abnormality, here the AGD, that are successive in the time scale of the acquisition of the images of the plurality of endoscopy images. A condition, in this example, for determining the start and the end of the sequence of images can be that a current image belongs to the sequence when there is at least one of:

[0082] - The abnormality under inspection is seeable in the previous consecutive image from the current image, in successive time order; and

[0083] - The abnormality under inspection is seeable in the next consecutive image from the current image, in successive time order.

[0084] In other words, the sequence of images comprises at least two images of the plurality of endoscopy images that are consecutive in time, respectively to the acquisition of the plurality of endoscopy images. In some embodiments, the sequence of images comprises at least three images, or at least four images, or at least five images, or even at least ten images of the plurality of endoscopy images that are consecutive in time.

[0085] In another example, the images of the sequence of images are not consecutive in time but are close to each other regarding the acquisition time. In other words, the sequence of images can comprise images from the plurality of images that at least show the same abnormality, even if other images that do not show said abnormality have been acquired in between these images. Put another way, the sequence of images comprises the images of the plurality of images that show the same abnormality. To reduce the number of images of the plurality of images, the images of the sequence of images can be images showing the same abnormality within a predetermined duration. The predetermined duration can be automatically set, for example based on the total acquisition time, or can be set by the practitioner and / or the operator.

[0086] In an embodiment, the sequence of images is built previously to the implementation of the method 100. In such embodiment, the method 110 does not comprise the step 110 for extracting the sequence of images. For example, in some cases, the sequence of images can be constructed on the fly, by the operator or automatically by a processing mechanism, during the acquisition of the images. The step 110 is then optional, depending on the application.

[0087] The plurality of endoscopy images can comprise images that originate from different endoscopy sensors of the same endoscopy device. Indeed, some endoscopy device, such as for capsule endoscopy comprise, one, two, three, four or more sensors adapted to acquire images.

[0088] The plurality of endoscopy images can be obtained in a step prior to this step 110 of extracting the sequence of images.

[0089] The method 100 also comprises a step 120 of determining a plurality of intermediate images. The intermediate images are taken from the sequence of images (which is either obtained from implementing step 110 or from another process, prior to implementing the method 100). The final image is therefore selected, in the next step of the method 100, from the intermediate images. The intermediate images correspond to images of the sequence of images where the view of the abnormality is a relevant view.

[0090] In particular, each image of the plurality of intermediate images is an image of the sequence of images that corresponds to a relevant view of the abnormality.

[0091] The relevance of the view can be defined by any classification standard known from the art that indicates whether the view of the abnormality is relevant or not. In other terms, the intermediate image corresponds to a relevant view of the abnormality when it satisfies one or more conditions related to a classification standard.

[0092] This approach notably depends on the application, i.e. , the part of the body that is inspected, the type of the endoscopy that is used, the abnormality that is considered, etc.

[0093] For example, in the case of the AGD inspected via capsule endoscopy, the Saurin method (see Saurin, J. C. & Pioche, M., Why should we systematically specify the clinical relevance of images observed at capsule endoscopy? Endosc. Int. Open 2, E88-89 (2014)) can be used to determine the relance of the view of the abnormality in the intermediate images.

[0094] Alternatively or concurrently, the relevance of the view of the abnormality in the image can be determined based on elements masking the abnormality in the images. A relevant image can also or alternatively be an image in which there is no element (such as gas bubbles, a blood clot, a bleeding, a natural secretion, a foreign body such as a food, etc.) masking totally or partially the abnormality.

[0095] The determination of the plurality of intermediate images can be achieved by using a classification algorithm. The classification algorithm can be based on classification techniques known from the art, such as supervised learning techniques, for example a support vector machine, a random forest, an ensemble learning method (such as a gradient boosting classifier, an Adaboost method, a Hasard method, etc.), a neural network, etc.

[0096] The classification algorithm is configured to classify the images of the sequence of images between relevant and non-relevant images.

[0097] The classification algorithm can be trained on a classification training database. This database comprises a plurality of classification training images. The classification training images are images of an abnormality of the same type as the investigated abnormality.

[0098] Each classification training image is associated with a label that indicates whether said image corresponds to a relevant view of the abnormality. The label is established on the basis of the classification standard related to the currently investigated abnormality. The label, therefore, refers to a class of the classification standard that is used. The label of each image of the classification training database can be predetermined by the practitioner, and / or by several practitioners, possibly including the practitioner currently considered. The images of the database have then already been analysed by a practitioner. Alternatively, the label of each image can be predetermined using another automatic image processing algorithm.

[0099] The images of the classification training database originate from an acquisition by a same endoscopy device as the one used to obtain the plurality of endoscopy images. The images of this database can originate from the same human or animal body as the one currently considered or can originate from acquisitions performed on one or more other human or animal bodies, respectively.

[0100] The training on the classification training database enables the classification algorithm to assign a label related to the classification standard to each image of the sequence of images. The method 100 can thus comprise a preliminary step 105a for training the classification algorithm on the classification training database, before the step 120 of determining the plurality of intermediate images, or even before the step 110 of extracting the sequence of images. In some embodiments, the training of the classification algorithm is carried out before the implementation of the method 100 and is not a step carried out by implementing the method 100.

[0101] Data augmentation techniques, known per se, can be used to enhance the quality of the training.

[0102] In the current example, the classification algorithm is a random decision tree. Each image of the plurality of classification training images is an image of an AGD in the small intestine of the human or animal body that is considered, for example from a previous endoscopy, and / or from one or more other human or animal bodies, respectively. The images of the classification training database have been previously acquired with a similar capsule endoscopy device and have each previously been labelled by a practitioner with a label that indicate the class of Saurin’s classification standard corresponding to the considered image.

[0103] The method 100 also comprises a step 130 of selecting the final image from the plurality of intermediate images. In other words, the intermediate images are browsed automatically in order to identify the image that shows the best the abnormality under investigation.

[0104] The selection is carried out by evaluating at least one property of the images. This property is then determined for each image of the plurality of intermediate images. This determination provides an alphanumeric value of the property for each of the intermediate images.

[0105] The property can be any kind of feature of an image that is representative of the quality of the view of the image.

[0106] In particular, the property can be a visual property in connection with the visual perception of a human eye. Said differently, the property can be representative of the way a human eye would analyse the images to select the final image, i.e. , the image that best shows the abnormality, among the other intermediate images. This means that the property can be a visual property, relating to a psychovisual feature that a human eye would analyse to determine the final image.

[0107] The property can, for example, be one or a combination of:

[0108] - A confidence indicator;

[0109] - A size of a frame around the abnormality;

[0110] - A position of the centre of the frame in the image;

[0111] - A colorimetric indicator.

[0112] A weighting of each property can be applied when several properties are analysed. For each intermediate image, the property that is determined is compared to at least one selection criterion. In other words, the determined value of the property is evaluated against the selection criterion.

[0113] The selection criterion depends on the type of the property to which it corresponds. The selection criterion is predefined based on a previous analysis of intermediate images acquired from a similar application (i.e. , similar endoscopy technique, similar part of the human or animal body and for a similar abnormality type), that has been conducted via another automatic intermediate images analysis approach or manually by a practitioner, who can be the same or another practitioner as the one currently using the method 100.

[0114] The final image is the image of the plurality of intermediate images for which the determined property is in accordance with the selection criterion, related to said property.

[0115] When several properties are considered, the finale image can be the intermediate image for which at least two of the determined properties satisfy their respective at least one selection criterion.

[0116] In the presented example, as schematically illustrated in Figure 2, three intermediate images 10a, 10b and 10c of the same AGD 11 are analysed. A confidence indicator Cia, Cib and Cic is provided for each imageWa, 10b and 10c, respectively. In each image, a frame 12a, 12b, 12c of rectangular shape delimits the abnormality 11 . Each side of the frame 12a, 12b, 12c fits to a portion of the contour of the shape of the abnormality 11. In each image 10a, 10b and 10c, is represented by a vertical cross the centre 13 of the image 10a, 10b and 10c. In the presented example, three properties of the intermediate images are analysed: the confidence indicator Cia, Cib and Cic, the size of the frame 12a, 12b, 12c and the position of the centre of the frame 12a, 12b, 12c. The final image is selected among the intermediate images as the one whose confidence indicator is above a predefined confidence threshold, whose frame is the largest (i.e., has the largest area) compared to the frames of the other intermediate images, and whose frame centre is the closest to the centre 13 of its corresponding intermediate image.

[0117] In some alternatives, it is possible to carry out a preselection among the intermediate images by determining at least one first property to be determined, corresponding selection criterion must be fulfilled. For example, this can be achieved by preselecting intermediate images whose confidence indicator is above the confidence threshold and for which a distance of the frame centre to the image centre is inferior to a predefined distance threshold. The selection is then carried out by selecting at least one second property to be determined, which corresponding selection criterion must be fulfilled. For example, following the preceding preselection example, the final image can be selected from the preselected intermediate images as being the one whose frame size is the largest.

[0118] It can be appreciated that these examples are mere illustration of the implementation of the method 100 and that the selection criteria and the way they are used in the calculation (i.e., their definition, which selection criterion or criteria prevail compared to others and / or which one are mandatory to be fulfilled, etc.) highly depend on the application of this method 100.

[0119] In some embodiments, the selection of the final image can be achieved by using a machine learning model. The machine learning model can be based on automatic image analysis techniques known from the art, such as supervised learning techniques, for example a support vector machine, a random forest, an ensemble learning method, a neural network (a convolutional neural network, a generative adversarial network, parallel neural networks or others), etc.

[0120] The machine learning model is configured to analyse the images of the plurality of intermediate images and to indicate which one is the final image, i.e., the best image that shows the abnormality. The machine learning model can be trained on a selection training database. This database comprises several pluralities of selection training images. Each plurality of selection training images comprises selection training images that are images of an abnormality of the same type as the investigated abnormality. Each plurality of selection training images has been obtained by an image processing similar to the one of step 120 of determining the plurality of intermediate images from a sequence of images. Each plurality of selection training images has then been determined from a sequence of images previously formed and that is different for each plurality of selection training images.

[0121] Each image of each plurality of selection training images is associated with a label that indicates whether said image is the final image of the plurality of selection training image it belongs to. The label is predefined by determining the at least one property and by comparing it against the corresponding at least one selection criterion. The label of each image of the classification training database can be predetermined by the practitioner, and / or by several practitioners, possibly including the practitioner currently considered. The images of the database have then already been analysed by a practitioner. Alternatively, the label of each image can be predetermined using another automatic image processing algorithm.

[0122] The images of the selection training database originate from an acquisition by a same endoscopy device as the one used to obtain the plurality of images. The images of this database can originate from the same human or animal body as the one currently considered or can originate from acquisitions performed on one or more other human or animal bodies, respectively.

[0123] The training on the selection training database enables the machine learning model to assign a label to each image of the plurality of intermediate images and that indicates whether the image is the final image or not.

[0124] The method 100 can thus comprise a preliminary step 105b for training the machine learning model on the selection training database, before the step 130 of selecting the final image, or before the step 120 of determining the plurality of intermediate images, or even before the step 110 of extracting the sequence of images. In some embodiments, the training of the machine learning model is carried out before the implementation of the method 100 and is not a step carried out by implementing the method 100.

[0125] Data augmentation techniques, known per se, can be used to enhance the quality of the training.

[0126] The method 100 can also comprise a step 140 of automatically sending and / or adding the selected final image to the medical report corresponding to the human or animal under examination.

[0127] The method 100 can also comprise a step 150 of updating the selection criterion based on an input provided by an operator.

[0128] Indeed, automatically selecting the final image does not mean that the practitioner cannot browse the other intermediate images.

[0129] The practitioner can, therefore, determine that another image suits best as final image. The practitioner can, then, provides the image he selected as input to step 150 of updating the selection criterion.

[0130] The updating of the selection criterion is then automatically achieved by analysing the input image and modifying the selection criterion based on this analysis. The analysis is performed regarding the properties that are determined for the intermediate images, i.e., the properties are determined for this input image and the selection criteria are adapted so that the input image is the final image according to these modified criteria.

[0131] The updating can be performed by an updating machine learning model adapted and trained to update said selection criterion from the input image. This model can use a supervised or unsupervised mechanism to carry out the update.

[0132] Alternative, the input can be an update value that corresponds to one or more of the determined properties and / or to one or more of the selection criteria. For example, the practitioner can indicate, thanks to this value, that a property is not relevant for him, and / or that the weighting assigned to a property is not suitable for him, and / or that the selection criterion is not appropriate for him. The practitioner can thus directly inform the method 100 that the designated property and / or selection criterion must be modified, for example must be tuned, to fit his preferences. Another aspect of the invention relates to a device 20, as illustrated in Figure 3, adapted to carry out the steps of the method 100. More precisely, the device 20 comprises a processing module 21 adapted to carry out the steps of the method 100. The device 20, more precisely the processing module 21 are then configured to carry out the steps of the method 100.

[0133] The device 20 also comprises a volatile or non-volatile memory 22 that comprises instructions that, when executed by the processing module 21 , cause the processing module 21 to carry out the steps of the method 100.

[0134] Such a device 20 can be a computer comprising a circuit comprising the processing module 21 and the memory 22.

[0135] Alternatively, the circuit can comprise an electronic board on which the steps of the method 100 are described in silicon, or a programmable electronic chip such as an FPGA (Field-Programmable Gate Array) chip.

[0136] The device 20 is also configured to obtain the plurality of endoscopy images or the sequence of images, depending on whether step 110 needs to be carried out. In this case, the device 20 also comprises a connection module 23 adapted to receive these images, such as a wired (USB, SATA, Ethernet, Firewire, etc.) or wireless (WIFI, Bluetooth, etc.) connection module 23. The connection module 23 can also be used to emit the final image to another device, for example in order to fill the medical report of the human or animal, which can be located on another device, with this final image, or to display the final image on a screen. In some embodiments, the medical report is built on the device 20 and is afterwards sent, if need, to another device for the practitioner to analyse the medical report.

[0137] The device 20 can be the endoscopy device itself. The device 20 then comprise an endoscopy sensor adapted to acquire the images of the plurality of endoscopy images. The endoscopy device can be a flexible endoscopy device, a rigid endoscopy device or a capsule endoscopy. The endoscopy device can comprise one, two, three, four or more endoscopy sensors.

[0138] The device 20 can alternatively be a device different to the endoscopy device that is connected to the device 20 via the connection module 23. The device 20 can be connected to the endoscopy device during the acquisition of the images, in order to analyse the acquired images in almost real time. Otherwise, the device 20 can be connected to the endoscopy device after the acquisition is finished.

[0139] The device 20 can comprise a display (not represented) or can be connected to a display via its connection module 23. The display is adapted to display the final image to the practitioner.

[0140] An illustration of implementation of the method 100, the based on in situ acquired endoscopy images, is given in Figure 4. These images were acquired by capsule endoscopy.

[0141] The Figure 4(a) shows a sequence of images of an abnormality comprising six images, numbered A to F. These images were then previously extracted as a sequence according to a process similar to the process of step 110 of extracting the sequence of images. Each image A-F is then an image of the abnormality under investigation, namely an AGD here. Each image A-F also comprises a frame 12 that delimits the abnormality in the image A-F.

[0142] The Figure 4(b) shows a plurality of intermediate images comprising three intermediate images that were selected from the sequence of images as satisfying the relevance scale of Saurin’s method. These images were obtained by carrying out step 120 of the method 100. The intermediate images are images B, E and F from the sequence images.

[0143] The Figure 4(c) shows the final image that is selected from the intermediate images of Figure 4(b), by carrying out the step 130 of the method 100. The final image is herein selected by evaluating the size of the frame 12 of each of the intermediate images and its position regarding the centre of the corresponding image.

[0144] The final image is then, thanks to the method 100, automatically selected from the sequence of images of the abnormality under investigation. The practitioner can then avoid the burden of browsing all the other images of the sequence to find the best view of the abnormality and thus can directly analyse the abnormality.

[0145] Although exemplified with capsule endoscopy of the small intestine, it is noted that the above-described method 100 can be carried out to select a final image for any kind of endoscopy, any kind of endoscopy device and for any kind of abnormality, in any organ (digestive or else) without needing particular adaptation.

[0146] Although presented for selecting the final image among the sequence of images, it can be appreciated that this method 100 also works for any plurality of images of the abnormality, even if they do not form a sequence, as aforementioned. In other words, the application to a sequence of images is a particular embodiment of the method to a specific kind of plurality of images that agrees with predetermined conditions, i.e., images that form the sequence of images.

[0147] Therefore, in some embodiments, the steps 120 of determining the plurality of intermediate images and the step 130 of selecting the final image are directly carried out for the plurality of images of the abnormality. Thus, the intermediate images are determined from the plurality of images.

[0148] Each image of the plurality of images of the abnormality is an image of the investigated abnormality. This plurality of images of the abnormality can be extracted from the plurality of endoscopy images that has been acquired.

[0149] This extraction can be carried out before the method 100 is carried out, or during a step, before carrying out step 120 of determining the plurality of intermediate images. The step 110 is then a step of extracting the plurality of images of the abnormality, instead of extracting the sequence of image.

[0150] The plurality of images of the abnormality can comprise all or part of the images of the abnormality comprised in the plurality of endoscopy images. For example, one or more of the images of the plurality of images of the abnormality can have been acquired with the same or another endoscopy device, and / or at an ulterior time during the same acquisition or an ulterior acquisition, than the rest of the images of the plurality of images of the abnormality.

Claims

CLAIMS1. Computer implemented method (100) for selecting a final image of an abnormality (11 ) of a part of a human or animal body, from a plurality of images of the abnormality (11 ), each having been acquired with an endoscopy device, the method comprising:Determining (120) a plurality of intermediate images (10a, 10b, 10c) from the plurality of images, each image of the plurality of intermediate images (10a, 10b, 10c) being an image of the plurality of images that corresponds to a relevant view of the abnormality (11 ) according to a classification standard, the classification standard being:• Saurin’s classification standard when the part of the human or animal body is a small bowel investigated by capsule endoscopy; or• Paris classification when the part of the human or animal body is a large intestine;From the plurality of intermediate images (10a, 10b, 10c), selecting (130) the final image based on at least one property determined for each image of the plurality of intermediate images (10a, 10b, 10c).

2. Method (100) according to any of the preceding claims, wherein the selected final image is sent and / or added (140) to a medical report.

3. Method (100) according to any of the preceding claims wherein said plurality of images of the abnormality (11 ) is obtained by:Extracting (110) the plurality of images of the abnormality (11 ) from a plurality of endoscopy images that have been acquired with the endoscopy device, each image of the plurality of images of the abnormality (11 ) being an image of the plurality of endoscopy images that is an image of the abnormality (11 ).

4. Method (100) according to claim 3, wherein the plurality of images of theabnormality (11 ) is a sequence of images that comprises at least two images of the plurality of endoscopy images that are consecutive in time, respectively to the acquisition of the plurality of endoscopy images.

5. Method (100) according to any of the preceding claims, wherein the determination of the plurality of intermediate images (10a, 10b, 10c) is carried out via a classification algorithm.

6. Method (100) according to claim 5, wherein the classification algorithm has been trained from a classification training database comprising a plurality of classification training images, each classification training image of the plurality of classification training images being associated with a label that indicates whether said image corresponds to a relevant view of the abnormality (11 ).

7. Method (100) according to any of the preceding claims, wherein the at least one property is a visual property.

8. Method (100) according to claim 7, wherein the at least one property is one or more of:- A confidence indicator (Cia, Cib, Cic);- A size of a frame (12a, 12b, 12c) around the abnormality (11 );- A position of the centre (13) of the frame (12a, 12b, 12c) in the image;- A colorimetric indicator.

9. Method (100) according to any of the preceding claims, wherein the final image is the image from the plurality of intermediate images (10a, 10b, 10c) for which the at least one property is in accordance with a selection criterion related to said property.

10. Method (100) according to claim 9, wherein the method further comprises:Updating (150) the selection criterion based on an input provided by an operator.

11. Method (100) according to claim 10, wherein the input is an image selected by the operator from the plurality of intermediate images (10a, 10b, 10c) that is different from the final image.

12. Method (100) according to any of the preceding claims, wherein the selection of the final image is carried out via a machine learning model.

13. Method (100) according to claim 12, wherein the machine learning model is trained from a selection training database comprising several pluralities of selection training images, each selection training image of each plurality of selection training images being associated with a label that indicates whether or not said image is the final image of the plurality of selection training images it belongs to.

14. Device (20) comprising a processing module (21 ) adapted to carry out the steps of the method (100) according to any of the preceding claims.

15. Computer program product comprising instructions which, when the program is executed by a computer, causes the computer to carry out the method (100) according any of claims 1 to 13.

Images