Combined system and method for selection, annotation and training via a shared machine-learning model

Abstract

The invention relates to a combined system for selecting and annotating training data of a machine-learning model, comprising the model (12) and two databases (14, 16) comprising unannotated or annotated data, respectively, characterised in that the model is configured to select a set of unannotated data (20) that are most useful to annotate, to automatically annotate the most useful unannotated data (20), to receive at least one correction and / or validation of data annotation by an oracle (18) in order to provide validated data (24) stored in the second database (16), and to be retrained by the validated data of the second database (16) for the selection of the most useful unannotated data and for the interactive annotation.

Description

[0001] DESCRIPTION

[0002] TITLE OF THE INVENTION: COMBINED SYSTEM AND METHOD FOR SELECTION, ANNOTATION AND TRAINING VIA A

[0003] SHARED MACHINE LEARNING MODEL

[0004] Technical field of the invention

[0005] The invention relates to a combined system for selecting training data for a machine learning model, annotating training data for the model, training the learning model and predicting from the model, a method and an associated computer program product. The invention is particularly intended for the annotation of data related to the segmentation of medical images, for example from magnetic resonance imaging (MRI) or computed tomography (CT) for the identification of organs on each of the section images obtained by FIRM or CT in order to recreate a 3D model of the organ from these segmentations, among other applications.

[0006] Technological background

[0007] Image segmentation is a technique for associating each pixel or voxel of a 2D or 3D image with a label representing the object, texture or state, etc. corresponding to said pixel or voxel.

[0008] Image segmentation is used in particular in a medical context to determine the organs, tissues, fluids, etc. visible on a 2D or 3D image. Such an image is obtained, for example, by MRI and makes it possible to form a 3D model of the targeted organ and, for example, of its environment and the tumor to be operated on. This creation of a 3D model requires the segmentation of the images obtained by FIRM.

[0009] Image segmentation can be based on the use of a machine learning model allowing automatic or semi-automatic segmentation of images, respectively without or with the help of a user to make corrections. The machine learning model acts as a predictor which, by receiving as input an image to be segmented, provides a segmented image as output. The use of a machine learning model requires prior training of the model using a large amount of annotated data, i.e. images in which the different segmentation classes have been determined and annotated / labeled. This annotated data is grouped into a dataset dedicated to the training, validation and testing of the implemented model.

[0010] The availability of annotated data is a key requirement for the supervised training of machine learning systems, which are defined by their ability to learn from this annotated data to make predictions or decisions. At the heart of a machine learning system is the predictor, which is trained by a machine learning model representing what the system has learned from the annotated data.

[0011] The performance and generalization capabilities of supervised machine learning inherently depend on the size and composition of the annotated training dataset. However, data annotation is generally costly: it often requires specific expertise in the target domain, as well as careful attention to detail and precise input from the annotator. This is particularly true in the field of medical imaging, which requires in-depth medical knowledge and often presents interpretation challenges. This need for expert intervention complicates the increase in the size of the dataset: annotating large amounts of data requires specialized labor and considerable time.

[0012] Solutions have been proposed to simplify the annotation task in various ways.

[0013] In particular, solutions to mitigate the data annotation problem have included, for example, semi-supervised, weakly supervised, self-supervised, and unsupervised learning, zero-shot and one-shot learning, transfer learning, and multitask learning. They can limit or completely eliminate the need for human annotation. However, their applicability is limited: they are susceptible to errors, which makes them less suitable for complex, safety-critical, and high-variability problems, typically those related to clinical and medical applications.These solutions offer less control over the quality of the results, generally require more complex algorithms and finer tuning, and can result in less accurate and interpretable models than supervised learning. Supervised learning generally offers a simpler way to achieve better performance when sufficient annotated data is available, and therefore remains relevant and used by many current approaches. Thus, data annotation by a human expert remains useful, and the search for a solution to speed up the annotation process is relevant. The human expert thus acts as an oracle for the machine learning system.

[0014] The basic machine learning paradigm has three main phases: a data annotation phase, a training phase for the predictive machine learning model, and an evaluation phase. The system can then return to the data annotation phase if necessary. In simpler systems, the data annotation phase may use a traditional annotation system such as 3D Slicer, which requires a lot of effort and time for a human oracle. In more advanced systems, this phase can be improved by using active learning to select the most useful unannotated data. This solution reduces the amount of data to be annotated while still providing relevant annotated data for training the machine learning model.

[0015] Solutions propose replacing the classical annotation system with a dedicated annotation predictor. The annotation predictor suggests an annotation that the expert can validate or correct. This raises the question of training this annotation predictor. Existing systems train the annotation predictor once a sufficient amount of data has been annotated by the classical annotation system. They are suboptimal because the annotation predictor and the classical annotation system do not improve as the data is annotated. One objective is therefore to exploit the data as it is annotated to train the annotation predictor and improve the annotation mechanism itself.

[0016] Other solutions propose to supplement the training with self-training (self-learning in English) allowing the use of pseudo-labels to automatically provide data labeled by the model for training the model itself. However, this pseudo-labeled or pseudo-labeled data does not present the same annotation quality as training data that has benefited from correction or validation, for example with a classic annotation system.

[0017] The inventors therefore sought a solution to simplify the annotation task for the oracle by improving existing systems.

[0018] Objectives of the invention

[0019] The invention aims to provide a system, a method and a computer program product for selecting training data of a machine learning model, annotating training data of the model, training the model and predicting from the model.

[0020] The invention also aims to provide, in at least one embodiment, a system for using a shared learning model to perform the selection of training data, and assist an oracle in the annotation of training data, while allowing the deployment of the shared model as a predictor at any time.

[0021] The invention also aims to provide, in at least one embodiment, a system, a method and a computer program product allowing image segmentation, object detection and recognition, and / or image classification, more generally of any type of task related to image processing, medical or not, or to computer vision, including, but not limited to, classification, categorization, segmentation, semantic segmentation, instance segmentation, object detection, localization. Presentation of the invention

[0022] To this end, the invention relates to a combined system for selecting training data for a machine learning model and for annotating training data for the model, comprising: the pre-trained model with initial annotated training data, a first database comprising a set of unannotated data, a second database configured for storing annotated data, characterized in that the model is configured to: select, from the set of unannotated data in the first database, a set of the most useful unannotated data to be annotated via an active learning method, automatically annotate the most useful unannotated data to provide automatically annotated data, receive, via an interactive annotation interface of the system associated with the machine learning model,at least one correction and / or annotation validation of the data automatically annotated by an oracle to provide validated data, said validated data being stored in the second database, being retrained by the validated data of the second database for the selection of the most useful unannotated data and for interactive annotation.,

[0023] A combined system according to the invention therefore makes it possible to manage different functions around a shared learning model so that the learning model benefits from this interaction between the different functions.

[0024] In particular, the system allows on the one hand to select training data by determining which unannotated data are most useful for training thanks to active learning implemented by the machine learning model. This allows to reduce the number of data to be annotated by an oracle and to allow a continuous addition of new data to be annotated by prioritizing the unannotated data that contributes most to improving the system's performance in the next training cycle.

[0025] The system also provides a solution to assist a human oracle in interactively annotating this most useful data using this same machine learning model to simplify data annotation. The annotation is thus controlled and guided by the machine learning model. The more efficient the model, the fewer interactions required by the oracle for data annotation, especially since the number of corrections required by the oracle is reduced. The annotated data is also of better quality than pseudo-labeled data from self-training.

[0026] Finally, the annotated data itself is used for retraining the model, in particular to assist in sorting the most useful unannotated data to annotate, and to assist the oracle in annotating the unannotated data. The machine learning model is better known as a Machine Learning model in English. The time spent annotating the data is thus reduced for the oracle, while maintaining control of the annotation. Retraining broadly refers to several solutions for training the model with the entire annotated data set, including the annotated data already used for training and the validated data, which can consist of: complete retraining, fine tuning or continual learning.

[0027] This retraining is carried out automatically or by a human expert competent to act on the machine learning model.

[0028] The fusion of these functions allows on the one hand to limit the resources required for annotation, and on the other hand to improve the overall performance of the annotation system, by combining the learning and annotation channels in a single system combined by the joint use of active learning by the machine learning model and interactive annotation using the same machine learning model as an annotation assistant. The use of a shared machine learning model therefore also makes it possible to facilitate the management of the model and to reduce technical and logistical interventions and delays, in particular by combining the annotation phase and the training phase of the machine learning model.

[0029] This approach also facilitates the deployment of the machine learning model for its automatic annotation prediction function for unannotated data when the machine learning model is sufficiently trained, especially when the dataset that has been annotated by the oracle is large enough. Pre-training also allows for a rapid approach to a deployable system that plays its role as a predictor during interactive annotation assisting the oracle, while allowing re-training with the new annotated data. Furthermore, pre-training requires only a minimal amount of initial annotated training data because the oracle corrects the first predictions and the model improves quickly. Initial annotated data is also kept as test data to track the progress of re-training.

[0030] Thus, the system allows a strong and crossed interaction between annotation and prediction: the selection of the data to be annotated is carried out with the objective of achieving a better prediction of the machine learning model and the prediction by the learning model allows assistance in the generation of annotations of non-annotated data.

[0031] The active learning method selects data based on specific criteria that can be internal to the data, external to the data (metadata), dependent on the data already annotated, and / or the complexity of annotation, etc.

[0032] The interactive annotation method is for example derived from prior art methods, for example a method derived from the methods described in one of the following publications:

[0033] Amrehn, M., Gaube, S., Unberath, M., Schebesch, F., Horz, T., Strumia, M., Steidl, S., Kowarschik, M., Maier, A., 2017. UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model, in: Eurographics Workshop on Visual Computing for Biology and Medicine, The Eurographics Association.

[0034] Zhou, T., Li, L., Bredell, G., Li, J., Konukoglu, E., 2021. Quality-aware memory network for interactive volumetric image segmentation, in: MICCAI.

[0035] Liao, X., Li, W., Xu, Q., Wang, X., Jin, B., Zhang, X., Zhang, Y., Wang, Y., 2020. Iteratively -refined interactive 3d medical image segmentation with multi-agent reinforcement learning. 2020 IEEE / CVE Conference on Computer Vision and Pattern Recognition (CVPR), 9391-9399.

[0036] Prior art systems use different means for data selection, automatic annotation, and validation / correction by the oracle. While some prior art systems already use machine learning models at each training step, none of these prior art systems combine data selection, automatic annotation, and validation / correction by the oracle for training a single model. Prior art systems typically use a model that is already fully trained at at least one of the steps, particularly for the annotation tool, and this already fully trained model has not benefited from the data improvements that other functions within the scope of the invention would have provided.

[0037] For example, data annotation is typically performed by pre-trained tools, whether based on a machine learning model or not. Prior art machine learning models use an active learning method regardless of the annotation mechanism. If active learning is performed, it is not used for improving the machine learning model that is used for automatic annotation and interactive annotation.

[0038] Advantageously and according to the invention, the selection of the most useful set of unannotated data to be annotated is carried out via the machine learning model using one or more of the following selection criteria: one or more internal properties of the data, one or more external properties of the data, the uncertainty of prediction by the machine learning model, the representativeness of the annotated data and the unannotated data, a criterion of difficulty of annotation of the data.

[0039] Advantageously and according to the invention, the system is configured for the annotation of new non-annotated data for at least one of the following applications: segmentation of new non-segmented or partially segmented images, detection and recognition of objects in new images, classification of new images, categorization of new images.

[0040] According to this aspect of the invention, the system is deployed for its final task and allows the annotation of new data not used for training, i.e. not used for pre-training or re-training.

[0041] According to other variants of the invention, the system is used in other applications, in particular any type of task related to image processing, medical or not, or to computer vision, including, but not limited to classification, categorization, segmentation, semantic segmentation, instance segmentation, object detection, localization, etc.

[0042] The invention also relates to a method for selecting training data of a machine learning model and for annotating training data of the model, said model being pre-trained with initial annotated training data, comprising: a step of selecting by the machine learning model and from among the set of unannotated data, a set of the most useful unannotated data to be annotated via an active learning method, a step of automatically annotating by the machine learning model the most useful unannotated data to provide automatically annotated data, a step of receiving at least one validation and / or an annotation correction of the automatically annotated data by an oracle to provide validated data,a step of training the machine learning model using validated data for the selection of the most useful unannotated data and for interactive annotation.,

[0043] Advantageously, the combined method according to the invention is configured to be implemented by a combined system according to the invention.

[0044] Advantageously, the combined system according to the invention is configured to implement the combined method according to the invention.

[0045] The invention also relates to a combined computer program product for selecting training data of a machine learning model and annotating training data of the model, said computer program product comprising program code instructions for executing, when said computer program product is executed on a computer, the steps of the combined method according to the invention.

[0046] The invention also relates to a combined system, a combined method and a combined computer program product characterized in combination by all or part of the features mentioned above or below.

[0047] List of figures

[0048] Other aims, characteristics and advantages of the invention will appear on reading the following description given solely for non-limiting purposes and which refers to the appended figures in which:

[0049] [Fig. 1] is a schematic view of a combined system for selecting training data of a machine learning model and annotating training data of the model according to one embodiment of the invention.

[0050] [Fig. 2] is a schematic view of a combined method of selecting training data of a machine learning model and annotating training data of the model according to one embodiment of the invention.

[0051] Detailed description of an embodiment of the invention

[0052] In the figures, scales and proportions are not strictly respected, for the purposes of illustration and clarity.

[0053] Furthermore, identical, similar or analogous elements are designated by the same references in all figures.

[0054] In the figures, the machine learning model is represented with a neural network symbol for illustrative purposes but other types of machine learning models can be used.

[0055] Figure 1 schematically represents a combined system 10 for selecting training data of a machine learning model 12 and for annotating training data of the machine learning model 12 according to an embodiment of the invention. The system 10 comprises the machine learning model 12 and two databases for managing unannotated data and annotated data, in particular a first database 14 comprising a set of unannotated data and a second database 16 comprising a set of annotated data. The machine learning model 12 is pre-trained with initial annotated training data. Initial test data is also provided to test the performance of the system after pre-training and each future training.

[0056] In particular, the system was tested on four datasets: MRI segmentation of the female pelvis on a dataset created by the applicant comprising ninety-seven MRI series representing three thousand sixty-six slices in total; CT (computed tomography) segmentation of the liver and pancreas on the “Medical Segmentation Decathlon” datasets, described in the publication “Simpson, AL, Antonelli, M., Bakas, S., Bilello, M., Farahani, K., Ginneken, BV, Kopp-Schneider, A., Landman, BA, Litjens, GJ, Menze, BH, Ronneberger, O., Summers, RM, Bilic, P., Christ, PF, Do, RK, Gollub, MJ, Golia-Pernicka, J., Heckers, S., Jarnagin, WR, McHugo, M., Napel, S., Vorontsov, E., Maier-Hein, L., & Cardoso, MJ (2019). A large annotated medical image dataset for the development and evaluation of segmentation algorithms. ArXiv, abs / 1902.09063.» : « Liver tumors » and « Pancreas tumors » ; MRI segmentation of the kidney on the AMOS dataset, described in the publication: Yuanfeng, Ji & Bai, Haotian & Yang, Jie & Ge, Chongjian & Zhu, Ye & Zhang, Ruimao & Li, Zhen & Zhang, Lingyan & Ma, Wanling & Wan, Xiang & Luo, Ping. (2022). AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation . 10.48550 / arXi v .2206.08023.

[0057] The model can be pre-trained on 12% (fifteen out of ninety-seven data volumes), 11% (fifteen out of one hundred and thirty-one data volumes), 12% (thirty-three out of two hundred and eighty-two volumes) and 3.3% (forty out of one thousand and three hundred volumes) of all data respectively to be able to efficiently produce quality annotations for the rest of the data using the system.

[0058] Machine learning model 12 is for example a neural network, for example an existing encoder-decoder neural network architecture, for example a ResNet34 encoder described in the publication “He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770-778.” and a generic decoder.

[0059] The system 10, when deployed, is able to automatically predict an annotation of the unannotated data, allowing correction or verification by an oracle 18, which is a human user sufficiently competent for this correction or verification, in other words a human expert. The correction or verification by the oracle is particularly important in the medical context where the annotation of the unannotated data presents an issue for a future therapeutic or surgical intervention, for example for segmentation of medical images intended for the generation of 3D models of organs for assistance in surgery by augmented reality.

[0060] Following pre-training, the machine learning model 12 requires re-training to improve its performance. This training requires data that must be annotated by the human oracle 18.

[0061] The first database 14 may comprise a large amount of unannotated data. The unannotated data is sorted to obtain a set 20 of the most useful unannotated data, thereby reducing the time required for the human oracle 18 to generate annotated training data.

[0062] The learning model 12 thus includes as inputs all of the most useful unannotated data and interactions from the human oracle 18 for annotating the unannotated data. In particular, the human oracle 18 has access to a user interaction loop 22 connected by the machine learning model 12. The interaction loop 22 receives data automatically annotated by the machine learning model 12. The user interaction loop 22 makes it possible to take into account a correction and / or validation by the human oracle 18 of this annotated data, via a user interface of the interaction loop 22. The machine learning model 12 takes into account any corrections and proposes a new automatic annotation accordingly.

[0063] Using machine learning model 12 as an annotation assistant is a method that can be called interactive annotation.

[0064] A machine learning model system comprising such an interaction loop for taking annotations into account is for example described in the publication of the inventors of the present application: Mikhailov, L, Chauveau, B., Bourdel, N., Bartoli, A. (2022). A Deep Learning-Based Interactive Medical Image Segmentation Framework. In: Wu, S., Shabestari, B., Xing, L. (eds) Applications of Medical Artificial Intelligence. AMAI 2022. Lecture Notes in Computer Science, vol 13540. Springer, Cham.

[0065] Once the annotation has been validated by the human oracle 18, the machine learning model 12 provides validated data 24 as output, which is stored in the second database 16. The new annotated data added to the second database 16 thus allows retraining of the machine learning model 12.

[0066] In addition to its automatic annotation functions, the machine learning model 12 is itself configured to provide the 20 most useful data to be annotated as output, which will be used as input for annotation. This method is called active learning.

[0067] An active learning method that can be used is, for example, a method based on uncertainty-based data selection through entropy calculation. This method is based in particular on the principle that a prediction with high entropy linked to an uncertain prediction is representative of informative data and useful for training the model.

[0068] Figure 2 schematically represents a combined method 100 of selecting training data of a machine learning model and annotating training data of the model according to an embodiment of the invention.

[0069] The method 100 represents an iteration of data selection and data annotation that can be implemented by a system as described with reference to FIG. 1. In practice, the method can be implemented iteratively for the annotation of the unannotated data, until it is stopped or when all of the unannotated data are annotated. Each iteration ends with a retraining of the machine learning model from a training set comprising the new annotated data. The stopping of the iterations can in particular be carried out when the tests carried out with the test data give an error rate below a certain threshold, or when a human expert, for example the oracle, considers that the performance of the annotation by the machine learning model is sufficient.

[0070] The method 100 comprises a step 104 of selection by the machine learning model and from the set of unannotated data 102, of a set 20 of the most useful unannotated data via an active learning method. The data 200 considered to be less useful are ignored and are not annotated at this iteration.

[0071] The determination of the most useful data 20 is carried out by inference from the machine learning model and is based on usefulness criteria which can be: related to the internal properties of the data, for example the type of data or the quality of the data or an intrinsic value of the data (value of a field, value of a pixel or a pixel area, etc.); related to the external properties of the data, of the metadata type, for example the origin of the data, etc.; related to the prediction uncertainty, because the data for which the predictor gives results with the greatest uncertainty generally indicates that this data is the most informative; related to the representativeness, the selection of single-feature data being more informative; or personalized criteria, for example a combination of these criteria or others.

[0072] The method 100 then comprises a step 106 of interactive annotation, comprising in particular a step of automatic annotation by the machine learning model of the most useful non-annotated data 20 to provide automatically annotated data, and a step of receiving at least one validation and / or one correction of annotation of the automatically annotated data by an oracle to provide validated data 24, which are annotated data and the annotation of which has been validated by the human oracle.

[0073] The method finally comprises a step 108 of training the machine learning model by the validated data 24 for the selection of the most useful unannotated data and for interactive annotation. The machine learning model is thus improved for the following iteration, and the model is thus retrained at each iteration. The number of annotated data used for training is thus increased at each iteration, ensuring that the most useful data are annotated to optimize the annotation time by the oracle and increase the performance of the machine learning model. In addition, the data selection maximizes the impact of each annotation on the performance of the machine learning model, which allows for rapid improvement in the performance of the machine learning model.

[0074] The system and method can in particular be used for the segmentation of medical images. In this context, the criteria for selecting the most useful data can for example be: related to the internal properties of the data, for example the type of image or the quality of the image; related to the external properties of the data, of the metadata type, for example a selection according to the age or sex of the patient from whom the data comes, the source of the data, etc.; related to the uncertainty of prediction by an inference of the machine learning model on all the unannotated data; related to the representativeness, for example the images of which a characteristic is at a position far from the average; or personalized criteria, for example a pre-selection of data considered as the most difficult to annotate by a human oracle.

[0075] The invention can also be used in contexts other than image segmentation, in particular object detection and recognition, and / or image classification, more generally any type of task related to image processing, medical or not, or to computer vision, including, but not limited to classification, categorization, segmentation, semantic segmentation, instance segmentation, object detection, localization.

Claims

CLAIMS 1. Combined system for selecting training data for a machine learning model and for annotating training data for the model, comprising: the machine learning model (12) pre-trained with initial annotated training data, a first database (14) comprising a set of unannotated data, a second database (16) configured for storing annotated data, characterized in that the machine learning model is configured to: select, from the set of unannotated data in the first database, a set of the most useful unannotated data (20) to be annotated via an active learning method, automatically annotate the most useful unannotated data (20) to provide automatically annotated data, receive, via an interactive annotation interface (22) of the system associated with the machine learning model,at least one correction and / or one validation of annotation of the data automatically annotated by an oracle (18) to provide validated data (24), said validated data (24) being stored in the second database (16), being retrained by the validated data of the second database (16) for the selection of the most useful non-annotated data and for interactive annotation., 2. Combined system according to claim 1, characterized in that the selection of the most useful unannotated data set (20) to be annotated is carried out via the machine learning model using one or more of the following selection criteria: one or more internal properties of the data, one or more external properties of the data, the uncertainty of prediction by the machine learning model, the representativeness of the annotated data and the unannotated data, a criterion of difficulty of annotation of the data.

3. Combined system according to one of claims 1 or 2, characterized in that it is configured for the annotation of new non-annotated data for at least one of the following applications: segmentation of new non-segmented or partially segmented images, detection and recognition of objects in new images, classification of new images, categorization of new images.

4. Method for selecting training data of a machine learning model and annotating training data of the machine learning model, said machine learning model being pre-trained with initial annotated training data, comprising: a step (104) of selection by the machine learning model and from the set of unannotated data, of a set of the most useful unannotated data to be annotated via an active learning method, a step of automatic annotation by the machine learning model of the most useful unannotated data (20) to provide automatically annotated data, a step of receiving at least one validation and / or an annotation correction of the automatically annotated data by an oracle to provide validated data, a step (108) of training the machine learning model with the validated data for the selection of the most useful unannotated data more useful and for interactive annotation.

5. Method according to claim 4, characterized in that the selection of the most useful set of unannotated data to be annotated is carried out via the machine learning model using one or more of the following selection criteria: one or more internal properties of the data, one or more external properties of the data, the uncertainty of prediction by the machine learning model, the representativeness of the annotated data and the unannotated data, a criterion of difficulty of annotation of the data.

6. Combined computer program product for selecting training data of a machine learning model and annotating training data of the machine learning model, said computer program product comprising program code instructions for executing, when said computer program product is executed on a computer, the steps of the combined method according to one of claims 4 or 5.

Images