METHOD FOR QUALIFYING A TRAINING DATASET

Abstract

METHOD FOR QUALIFYING A TRAINING DATASET Method for accessing a computing resource in a two-step sequence, said method comprising a first step for qualifying a training dataset for training a machine learning model and a second step for unlocking access to a computing resource, the first step comprising: generation of a first set (ENS1) of images; generation of a target image (IMC); selection of at least one first image; generation of a first image label associated with a first selected image; assignment of the first label, so as to obtain a first set of labeled images intended for training a machine learning model, said selection of the first image being associated with a first step of an access sequence, and engagement of a second step of said access sequence, in which a code check is performed. Figure for the abstract: Fig.4.

Description

Title of the invention: METHOD FOR QUALIFYING A TRAINING DATASET Scope of the invention

[0001] The invention relates to a method for qualifying a training dataset for learning a machine learning model. The invention also relates to a method for generating a crowdsourced training database for learning a machine learning model. State of the art

[0002] Annotated data plays a crucial role in training machine learning models in various domains. By providing labels or annotations that describe the characteristics and categories of the data, annotated data enables machine learning algorithms to understand and generalize models from the provided examples. Whether in image recognition, natural language understanding, anomaly detection, or other tasks, annotated data serves as a reference for teaching models to recognize patterns and make relevant decisions. A high-quality annotated dataset is essential to ensure the accuracy and robustness of machine learning models, and often, its creation requires human intervention to guarantee the correctness of the annotations.

[0003] This is the case in the medical field, where new analytical techniques using artificial intelligence are being developed. In particular, machine learning algorithms are increasingly used to solve problems related to structure detection or the characterization of medical images. These algorithms require a considerable volume of annotation and training data.

[0004] One problem with image annotation, for example in the medical or astronomical fields, is that the images to be annotated do not correspond to any structure commonly known to an individual. Indeed, classifying headlights, dogs, or cars relies on a known ontological framework because an individual can associate an image with an object whose ontology is known to them. However, when one wants to classify an image presenting patterns that are difficult to isolate because one is looking for an overall similarity or a similarity based on several criteria, only a domain expert is able to properly label the image.

[0005] An object of the invention is to propose a solution improving the disadvantages of prior art solutions. Summary of the invention

[0006] According to a first aspect, the invention relates to a method for qualifying a training dataset for learning a machine learning model, said method comprising: • generation of a first set of images, corresponding to a segmentation of at least one given input image by a graphics component, • display of the first set of images within a graphic element inscribed within a predefined geometric shape; • generation of a target image representing the target of interest and associated with a label named "target label"; • selection of at least one first image from the first set using a selector in a graphical interface accessible via said graphical component; • generation of a first image label associated with at least one first selected image; • qualification of at least one first selected image by assigning the first label, so as to obtain a first set of labeled images intended to train a machine learning model.

[0007] All embodiments relating to image labeling described according to the second aspect of the invention also relate to the first aspect.

[0008] According to a second aspect, the invention relates to a method for accessing a computing resource in a two-step sequence, said method comprising a first step for qualifying a training dataset for learning a machine learning model and a second step for unlocking access to a computing resource, the first step comprising: • generation of a first set of images, corresponding to a segmentation of at least one given input image by a graphic component and comprising a first panel of images including at least one image likely to correspond to a target of interest and a second panel of images not corresponding to the target of interest; • display of the first set of images within a graphic element inscribed within a predefined geometric shape; • generation of a target image representing the target of interest and associated with a label named "target label"; • selection of at least one first image from the first set using a selector in a graphical interface accessible via said graphical component; • generation of a first image label associated with at least one first selected image; • qualification of at least one initial selected image by assigning the first label, so as to obtain a first set of labeled images intended to train a machine learning model,

[0009] said graphical component generating an element enabling a second step of said unlocking sequence to be engaged in said computer resource, in which a code check is performed.

[0010] According to one embodiment, the selection of said at least one first image is associated with a first step in a sequence of access to a computer resource.

[0011] Selecting at least one image allows you to automatically proceed to the second step of the sequence.

[0012] According to one embodiment, the generation of a first set of images comprises a first panel including at least one image representing a target of interest and a second panel of images not representing the target of interest.

[0013] According to one embodiment, the generation of the first set of images results in a random display by a graphic component of the set of images arranged within a graphic element that fits within a predefined geometric shape.

[0014] According to one embodiment, the segmentation of a given input image by a graphics component is performed by randomly fragmenting the input image into fragments of identical resolution. When different input images are used, the selection and generation of the random arrangement can be performed so as to display images from different sources in the same window.

[0015] According to one embodiment, each image of the first set is extracted from a given input image having an identifier.

[0016] According to one embodiment, the qualification step includes the qualification of a set of images by associating said set of images with the first label, so as to obtain a first set of qualified images.

[0017] According to one embodiment, the selection of at least one first image is associated with a first step in a sequence of unlocking access to a computer resource.

[0018] According to one embodiment, the graphic component displaying a graphic element within a predefined geometric shape and containing the images allows a second step of the unlocking sequence to be initiated. of access to said computer resource, in which a code check is performed.

[0019] Thus, advantageously, the method according to the invention promotes image labeling actions and encourages a user to perform such actions by associating them with actions of entering identifiers or codes. Indeed, when performing an action of entering identifiers or codes, a user pays more sustained attention, which makes the annotation action performed in combination with the input action equally sustained and focused. The probability that the resulting annotation will be correct, or more precise, is then increased.

[0020] In one embodiment, each image in the first set is extracted from a given input image containing an identifier. Thus, the set of images in the first set can originate from a plurality of source images. Advantageously, each image extracted from a source image is associated with an identifier of the source image so as to allow subsequent labeling of the source image by reassociating it with the label of the extracted image fragment. Reassociation is possible if the image fragment included in the first set of images is associated with its source image, for example, through metadata associated with the fragment, such as an identifier.

[0021] In some embodiments, the method includes sending the first set of qualified images to a remote server.

[0022] In some embodiments, the images in the first set of images represent biological organisms or fragments of biological organisms, objects or fragments of objects.

[0023] According to one embodiment, the images of the first set of images represent fragments of elements whose image capture is defined at the microscopic scale.

[0024] According to one embodiment, the images of the first set of images represent fragments of elements whose image capture is defined at the macroscopic scale.

[0025] According to one embodiment, the images of the first set of images are satellite images of portions or areas of space defining contours of shapes of terrestrial elements, such as trees, forests, watercourses, urban areas, etc.

[0026] In some embodiments, the images in the first set of images represent stellar bodies, stars, galaxies and any other body present in space.

[0027] Thus, medical images can, for example, be annotated during the process according to the invention. Advantageously, large medical databases can be annotated and constructed in this way.

[0028] In certain embodiments, the images in a panel of displayed images, which fit within a geometric shape, have identical or substantially equivalent resolution. Equivalent resolution is defined as a difference in resolution between two images of less than 10%.

[0029] Generally, CAPTCHA codes display images of varying resolutions to deceive a robot that could more easily analyze images of the same resolution. In other words, changes in resolution are among the means used to deceive a robot in automated analysis. In the present intention, displaying images of the same or identical resolution provides a means of improving the quality of training for a machine learning model configured to discriminate between image fragments corresponding to a given target.

[0030] In some embodiments, the second step of the unlocking sequence is successive to the first step of the unlocking sequence.

[0031] Thus, access to and completion of the second step of the unlocking sequence is conditional upon completion of the first step of the unlocking sequence.

[0032] In some embodiments, the process further includes a step of verifying the first label.

[0033] In certain embodiments, the process further comprises: - selection of a second image from the first panel associated with the target label, the selection allowing verification of the value of the first label.

[0034] Such a check makes it possible, for example, to validate the first unlocking sequence of the process. Such a check can also make it possible to validate that the user enters information that is a priori correct on the other selected images.

[0035] Thus, the verification step allows the labeling carried out to be validated or invalidated.

[0036] In certain embodiments: • at least one first selected image has been previously associated with a predefined label; • The first label verification step includes a comparison of the first label with the predefined label.

[0037] Thus, advantageously, when information on at least one first selected image is known a priori, the labeling verification can be carried out in real time.

[0038] According to one embodiment, the second step of the sequence involves authenticating a user with a data server. The second step may correspond to any authentication step such as authentication involving the entry of a username or email address and a password, or two-factor authentication, or authentication involving the entry of a code received on a terminal, etc.

[0039] In some embodiments, the step of verifying the first label is carried out by a human being.

[0040] Thus, advantageously, the labeling can be verified on a human scale, for example by experts.

[0041] According to various embodiments, the verification of said first label is carried out by computer equipment in an automated, supervised or unsupervised manner.

[0042] In some embodiments, the first label verification step includes a comparison with a prediction from a second machine learning model trained to generate a label prediction for the image. According to this embodiment, a model is trained to generate a statistic indicating whether an image belongs to a class within the image. One advantage of this solution is that it improves the training of a model whose statistics are not yet consolidated.

[0043] For example, if the second model assigns a class of images with a 60% confidence level in class assignment, the classification of the class generated by a user's image selection can be validated. In other words, beyond a given prediction threshold, a target class assigned to a user-selected image can be validated because the confidence in a predicted class is sufficient. One advantage is that it can strengthen the training of a pre-trained machine learning model.

[0044] Conversely, if the second model assigns a 25% confidence statistic to the classification of a trusted image, and this image has been previously selected by a user, then the validation of the target class is not performed. In this latter case, a notification can be issued so that an expert can assign the target class to the selected image or not. This process makes it possible to qualify images for which doubt remains regarding the application of the target label. Such a process improves the training quality of a machine learning model.

[0045] In this case, the first learning model can result as a combination of the qualified returns of the image labels assigned to the images when the A second model is implemented. According to one embodiment, the second model can be the first machine learning model.

[0046] In some embodiments, the selection, first label generation and qualification steps are implemented by a first user terminal, the process comprising following the qualification step of the first set of images: • display of the first set of images (ENS1) within a window defining or fitting within a predefined geometric shape on a display of a second user terminal; the window is for example a graphic element superimposed on a page or a digital document; • generation of a second target image representing the first target of interest and associated with the label named "target label"; • selection of at least one image from said first set (ENS1) using a selector from a graphical interface accessible via a graphical component of the second terminal; • generation of the first image label (LB1) associated with at least one selected image; • qualification of at least one selected image by assigning the first label (LB1) in order to obtain a second set of labeled images, • the first label verification step including a comparison of the first labeled images with the second labeled images.

[0047] Advantageously, the verification step can be performed automatically by pooling the results of the process according to the invention when implemented through the interaction of several user terminals with a remote server. When a minimum number of identical images generated within different terminals and selected by users correspond to the target and therefore have a target label, a verification step validates the qualification of the target label for each selected image. Thus, the target label assigned by a single user or a single terminal is a provisional or temporary target label. When different users have qualified a selected image with the same target label, this label can be definitively validated.

[0048] In some other embodiments of the invention, several checks are performed on the same image before validating the target label of the selected image.

[0049] In certain other embodiments of the invention, no human verification is performed before validating the target label of the selected image. Learning in this latter case can be more easily carried out and requires fewer validation steps.

[0050] In some embodiments, the step of selecting at least one first image includes a multiple selection of several images.

[0051] In some embodiments, the process includes, after selecting at least one first image, a processing step using said graphical interface, so as to obtain at least one first processed image.

[0052] Thus, advantageously, simultaneously with an annotation action, processing of the set of images is carried out, so that the annotation action is used to perform operations on the set of images.

[0053] In some embodiments, the processing step includes a modification of a parameter of at least a first image, said parameter being chosen from a contrast, a sharpness, a saturation, an intensity, a color, etc.

[0054] A third aspect of the invention relates to a method for generating a training database by supplying it with images from a population of users labeling images from a plurality of initial image sets for training a machine learning model, comprising: - implementation, by a plurality of remote graphical components, of the process to qualify a training dataset for learning a previously described machine learning model, so as to obtain a plurality of first sets of qualified images;

[0055] - generation of the training database by union of the first sets of qualified images from said plurality of first sets of qualified images.

[0056] Advantageously, when the method for qualifying a training dataset includes a processing step, so as to obtain a plurality of at least one first processed image, the method for generating a training database by crowdsourcing further includes: - determination of a treatment called global treatment from the plurality of at least one first processed image.

[0057] Thus, advantageously, by pooling the implementations of the process to qualify a set of training data, the process for generating a training database by crowd provision makes it possible to determine and define a global treatment from all the implementations of the process to qualify a set of training data.

[0058] In some embodiments, the treatment includes averaging.

[0059] Thus, advantageously, it is possible to define a processing representative of a plurality of processing carried out by remote users, which can serve as a standard for input data of the machine learning model.

[0060] In some embodiments, the process comprises: • saving the global processing, so that when the machine learning model is run, the execution of the machine learning model includes a preliminary step of applying the processing to an input image of the machine learning model.

[0061] Thus, the method for generating a training database by crowdsourcing makes it possible to generate a process from the set of implementations of the method to qualify a training data set which can subsequently be applied to an input image of the machine learning model when it is executed. Brief description of the figures

[0062] Other features and advantages of the invention will become apparent from the following detailed description, with reference to the accompanying figures, which illustrate:

[0063] [Fig.1]: an example of a system configured to implement a method for qualifying a set of training data according to the invention;

[0064] [Fig.2]: an example of a device configured to implement the method for qualify a training data set according to the invention;

[0065] [Fig.3]: an example of a set of steps that can be carried out to put in implements the method to qualify a set of training data according to the invention;

[0066] [Fig.4]: a representation of an example of a generated and displayed image set by a graphical component of the device of [Fig.2] and used in the process to qualify a set of training data according to the invention;

[0067] [Fig. 5]: a second representation of the example image set illustrated in the [Fig.4];

[0068] [Fig. 6]: a representation of an example set of images from different sources, generated and displayed randomly by a graphical component within the window;

[0069] [Fig.7]: an example of a system configured to implement a method for generate a crowd-supply training database according to the invention;

[0070] [Fig.8]: an example of a set of steps that can be carried out to implement the method for generating a crowd-supplied training database according to the invention;

[0071] [Fig.9]: a schematic representation of a training database generated by the process to generate a crowd-supply training database according to the invention, at different times. Description of the invention

[0072] In many fields, new analytical techniques using artificial intelligence are being developed. Thus, for training machine learning models, it is common to have to produce a large volume of annotation and training data. This is the case in certain medical disciplines, such as histology, where the analysis of histological sections is increasingly automated. In this field, however, it is difficult to collect training data because it is currently provided by trained users, such as students or qualified pathologists, who are few and far between. These individuals are responsible for making precise annotations, representing a large-scale, time-consuming, and costly task.

[0073] In the problem of obtaining a larger volume of annotated data for the purpose of training machine learning models, one of the objects of the invention aims to promote data annotation actions, by associating such actions with frequent and daily actions, by conditioning the latter on one or more annotation actions.

[0074] One aspect of the invention relates to a method 100 for qualifying a training dataset for learning a machine learning model, illustrated in [Fig. 3]. Another aspect of the invention relates to a method 200 for generating a training database by crowdsourcing for learning a machine learning model, illustrated in [Fig. 8].

[0075] The method 100 can be implemented using a system 10. [Fig. 1] shows an example of a set of elements of the system 10. [Fig. 1] more specifically represents a data network NET1, which can be the internet. A user terminal T1 provides access to a remote server SERVI.

[0076] The user terminal Tl can send and receive information to and from the remote server SERV1 by means of electronic communication systems.

[0077] The remote server SERVI stores a set of images ENS0. The images in the set of images ENS0 are intended to train a machine learning model and require annotation, i.e. the assignment of a label.

[0078] Images are understood to mean any element defined by a set of pixels and contained within a closed area, capable of being displayed on a medium or equipment such as a display. Images can be 2D, 3D, animated, or non-animated. Images can be extracted from other images. Images can be ultrasound, MRI, CT scan, telescope, or microscope images, and more generally, images from any optical, electromagnetic, or acoustic sensor. images can be derived from a reconstruction step of a dataset acquired from one or more sensor(s).

[0079] According to one example, the images in the ENSO image set are medical images and the machine learning model is configured to classify these medical images and predict the presence of specific structures.

[0080] According to another example, the images in the ENSO image set are astronomical images representing astronomical objects and the machine learning model is configured to classify these astronomical images and predict the presence of specific structures.

[0081] Specific structures can correspond to shapes, contrasts, colors or intensities, thicknesses of structures, patterns, dimensions of structures or a combination of these criteria.

[0082] Figure 2 represents a schematic diagram showing the components of a Example of the user terminal TL: The user terminal Tl can be implemented as a single hardware device, for example, as a desktop personal computer (PC), laptop computer, personal digital assistant (PDA), smartphone, smartwatch, server, or console, or it can be implemented on separate, interconnected hardware devices linked by one or more communication links, with wired and / or wireless segments. The user terminal Tl can, for example, communicate with one or more cloud computing systems, one or more servers, or remote devices to implement the functions described herein for the device in question. The user terminal Tl can also be implemented itself as a cloud computing system.

[0083] As shown in [Fig. 2], the user terminal T1 comprises a computer, this computer including a memory 11 for storing program instructions that can be loaded into a circuit 12 and adapted to cause a circuit to execute steps of the process 100 illustrated in [Fig. 3], described below, when the program information is executed by the circuit. The memory can also store data and information useful for executing the steps of the present invention as described below.

[0084] Circuit 12 can be, for example: - a processor or processing unit adapted to interpret instructions in a computer language, the processor or processing unit being able to understand, be associated with, or be attached to a memory containing the instructions, or - the combination of a processor / processing unit and a memory, the processor or processing unit being adapted to interpret instructions in a computer language, the memory containing said instructions, or - an electronic circuit board in which the steps of the invention are described in silicon, or - a programmable electronic chip such as an FPGA chip (for "Field-Programmable Gate Array").

[0085] The memory 11 may include random access memory (RAM), cache memory, non-volatile memory, backup memory (e.g., programmable or flash memory), read-only memory (ROM), a hard disk drive (HDD), a solid-state drive (SSD), or any combination thereof. The ROM of the memory 11 may be configured to store, among other things, an operating system and / or one or more computer program codes for one or more software applications. The RAM of the memory 11 may be used by the circuit 12 for the temporary storage of data.

[0086] The computer may also include an input interface 13 for receiving input data and an output interface 14 for providing output data. Examples of input and output data will be provided later.

[0087] To facilitate interaction with the computer, a screen 15 and a keyboard 16 can be provided and connected to the computer circuit.

[0088] Fig. 3 is a flowchart representing an example of a set of steps that can be carried out to implement the method 100 for qualifying a training dataset for learning a machine learning model.

[0089] Suppose a user of user terminal Tl wishes to access a computing resource via user terminal TL. In one example, the computing resource is user terminal Tl itself when access to it is locked, for example, due to prolonged inactivity. In another example, the computing resource is a secure application installed on user terminal Tl that can be launched from user terminal Tl by entering credentials. In yet another example, the computing resource is a secure application installed on a server and accessible from a data network and user terminal TL. The secure application can then be launched from user terminal Tl by entering credentials.

[0090] Advantageously, the method 100 according to the invention is part of a sequence for unlocking access to the computer resource via the user terminal TL. The unlocking sequence comprises a first step and a second step, which can be carried out successively in a predefined order or in reverse order, or simultaneously, in order to unlock access to the user terminal TL.

[0091] In a first generation step GEN1 of a set of images ENS1 from at least one input image.

[0092] According to one embodiment, the input image is cut or segmented into a plurality of images defining images corresponding to thumbnails representing only a part of the input image.

[0093] According to another embodiment, the set of images ENS corresponds to as many input images

[0094] According to another embodiment, the image set ENS1 corresponds to subsets of images from different input images.

[0095] According to one example, a first set ENS1 of images generated in a display window comprises a first panel PA1 which may include at least one image representing a target of interest CIB1 and a second panel PA2 of images not representing the target of interest.

[0096] Fig. 4 represents an example of the first set of images ENS1.

[0097] In [Fig. 4], the first panel PA1 is shown with a white fill and the second panel PA2 is shown with a fill of hatched lines, here horizontal. It is assumed in this case that the images in the first panel PA1 are those that the user is to select and therefore annotate. The user selects the images because they are representative of the target IMC image.

[0098] However, the first PA1 image panel is not known in advance except when certain images whose label is known are used to validate that the user is taking steps to identify the actual representative images of the target BMI image. The second PA1 panel is therefore defined a posteriori after the user's selection.

[0099] According to one example, input images likely to include images corresponding to the target image and therefore belonging to the first PA1 panel can be pre-identified with a confidence statistic. The aim is then to validate whether this pre-identification is correct or not.

[0100] The first set ENS1 corresponds, for example, to the result of dividing an image IM0, configured to be annotated, into sub-images from the set of images ENS0 stored by the remote server SERVi. The first set of images ENS1 can result from segmenting a given input image into thumbnails, each representing a part of the given input image.

[0101] According to one example, the resolution and / or size of the input image is determined so that an image corresponding to the target image is contained within one thumbnail. According to another example, the resolution and / or size of the input image is determined so that an image from the first set corresponding to the target image is contained within a maximum of 4 thumbnails.

[0102] Other configurations allow the input image to be adapted so that the elements to be selected, i.e., a target image, are contained within a maximum number of adjacent thumbnails. This ensures that the display window dimensions are optimized for displaying a target image.

[0103] According to one embodiment, the dimension of an IMC target image is estimated so as to ensure that the display of an element corresponding to a target is contained within a thumbnail.

[0104] Figure 6 illustrates a scenario in which the target image IMC represents a cell, tissue, or microorganism with a unique characteristic of structure, shape, color, regularity, etc. This target image IMC is displayed at the top of the image in a dedicated area that can be supplemented with a description in the form of an instruction to help an individual identify images in the ENS1 set containing an element resembling the target IMC. In this example, the elements being sought are smaller than the dimensions of an image when generated within the ENS1 set in a window.

[0105] According to one embodiment, a target has dimensions less than 50% of the size of a thumbnail. One advantage is to minimize the cases where such a target would be arranged on a boundary of the thumbnail, that is to say on a boundary of an image of the ENS1 set.

[0106] When the images of the set ENS1 form a coherent image, that is to say that it is a single image decomposed into 9 pieces for example, or into 16 pieces, the targets contained in the images and positioned at the limit of a thumbnail can spill over into an adjacent thumbnail.

[0107] When the images of the set ENS1 form an incoherent image, that is to say that it is a plurality of images assembled randomly and composing 9 pieces for example, or 16 pieces a priori independent, possibly coming from different source images, the overflow of an element resembling a target at the limit has no reason to appear on an adjacent image of the set ENS1.

[0108] In the latter case, it is understood that the dimensions of a target are a priori smaller than those of an image of the set ENS1 so that the user can recognize a target in the images provided.

[0109] The invention relates to an embodiment in which the images are randomly generated in a window and whose dimensions allow visibility of target-like elements on several images of the set ENS1.

[0110] According to one embodiment, each image generated in the ENS1 set comes from a different source image. More generally, this case relates to an embodiment in which the images of each thumbnail of an image displayed in the graphic element come from different source images.

[0111] According to one embodiment, the images of the ENS1 set are extracted from a larger original image which has been segmented.

[0112] The target of interest CIB1 is, for example, a biological structure or a fragment of a biological organism of a biological tissue. Non-limiting examples of biological structures are: a gland, cross-section, a lymphocyte, a surface epithelium, apoptosis, a mitosis, a lymphoid mass, a villus, a Helicobacter-type germ, a red blood cell, a mucus cell, an enterocyte nucleus, a plasma cell, a neutrophil (PMN), an eosinophil (EPN), a lymphocytic cryptitis, an abscess, an ulceration, a cryptic abscess, a granuloma.

[0113] According to another example, the target of interest CIB1 may be an astronomical object. Non-limiting examples of astronomical objects are: a star, a planet, a nebula, a galaxy, a quasar, a black hole.

[0114] According to another example, the target of interest CIB 1 can be a microorganism-type object, an object, a landscape, etc.

[0115] The GENi generation step is advantageously implemented by components of the remote server SERVi. For example, the GENi generation step is advantageously implemented by a processing unit of the remote server SERVi. By processing unit is meant an electronic component or a plurality of electronic components comprising, for example, a computer, comprising a set of at least one processor, and optionally a memory operationally coupled to the computer.

[0116] According to a first example, the first set ENS1 is transmitted to the user terminal Tl, in the form of a random display of thumbnails.

[0117] According to a second example, the first set ENS1 is transmitted to the user terminal Tl, in the form of an ordered display of thumbnails arranged so as to maintain an overall consistency of the input image which has been segmented.

[0118] The display of all the generated images is inscribed within a geometric shape on the screen 15 of the user terminal TL

[0119] In a second generation step GEN2, a target image IMC representing the target of interest CIB1 is generated and displayed on screen 15. The target image IMC is associated with a label called the "target label". The target label can be: "presence of the target of interest CIB1" within an image or a group of images.

[0120] According to an exemplary embodiment, during the second generation step GEN2, an INSTR instruction can also be generated and displayed near the target image IMC. For example, the INSTR instruction can prompt the selection of images showing the target of interest CIB1.

[0121] Non-limiting examples of instructions are: - when the target of interest CIB1 is a cross-section gland: "Can you find one or more daisies?", - when the target of interest CIB1 is a lymphocyte: "Can you find this little bead?", - when the target of interest CIB1 is a surface epithelium: "Can you identify the boundary of the great white?", - when the target of interest CIB1 is apoptosis: "Can you find those very dense little grains?", - when the target of interest CIB1 is a mitosis: "will you be able to find this badly wound ball of yarn?", - when the target of interest CIB1 is a lymphoid cluster: "can you identify this pile of marbles?", - when the target of interest CIB1 is a villus: "can you spot one or more glove fingers?", - when the target of interest CIB1 is a Helicobacter type germ: "can you spot this nasty little bug?", - when the target of interest CIB1 is a red blood cell: "Look, it's a red blood cell! Can you identify them here?" - when the target of interest CIB1 is a mucus-secreting cell: "can you identify this cell with the large belly?", - when the target of interest CIB1 is an enterocyte nucleus: "a bit like a lychee pit? Can you spot them?", - when the target of interest CIB1 is a plasma cell: "It too has a big belly but it is not transparent, find it!" - when the target of interest CIB1 is a PNN: "can you spot this king of contortion?", - when the target of interest CIB1 is a PNE: "he always has two plump cheekbones on his pinkish-orange cheeks, will you be able to spot him?", - when the target of interest CIB1 is a lymphocytic cryptitis: "These little beads don't belong in this daisy! Find the image that looks like it," - when the target of interest CIB1 is an abscess, an ulceration, a cryptic abscess: "These contortionists shouldn't be playing together! Not in the yard, not on a daisy! Find these rascals!" - when the target of interest CIB1 is a granuloma: "can you find this cluster of pink cells?".

[0122] According to this latter embodiment, the instruction may correspond to a non-limiting popularization of structure to be identified in order to facilitate annotations and the choice of thumbnails.

[0123] According to one embodiment, the choice may be limited to selecting an image, or it may involve drawing, outlining, delimiting, masking, coloring, or covering the structure to be visualized, in order to refine the annotation. In the latter case, an automatic analysis of the pixels of the selected image makes it possible to record metadata relating to the position, size, or shape of a structure within an image.

[0124] According to one embodiment, instructions are not generated. For example, when images are randomly generated within the window containing all the ENS1 images, an instruction is not necessarily generated. It should be noted that, within the scope of the present invention, the instructions that may be implemented in the solution of the invention are generated optionally depending on the use case.

[0125] According to one embodiment, the images are 3D images, that is to say images containing depth information.

[0126] According to another example, the images are animated, for example, images comprising a short animated sequence such as a video. This can be useful for characterizing a movement, a flow, a process, for example, mutations of proteins, organoids or other biological structures, or the aging of cells or other biological elements. For this purpose, animated images comprising a chronological sequence of images can be used.

[0127] According to one example, ultrasound images of the echographic type can be used to form the images of the ENS1 set displayed in the window.

[0128] According to an embodiment depending on the specific case, the resolution and size of the images of the first set ENS1 can be adapted so that a target likely to be present in the input image is included in one or more images of the first set ENS1.

[0129] In a selection step SEL, the user selects at least one first image IM1 from the first set ENS1 using a selector on the user terminal T1. The first selected image IM1 is represented by a dotted fill in [Fig. 5]. The selector is accessible via the screen 15. For example, the selector is a pointer that can be moved on the screen 15 using a mouse. The user may be prompted to select a plurality of images. This may occur when the target is present in several images of the set ENS1 or when the target spans several images.

[0130] The target(s) sought are not necessarily identical to the target represented in the IMC target image. Indeed, variations in shape or color may be tolerated since the aim is precisely to label images of the same class having common characteristics. One advantage of the invention is precisely to help An individual, through the display of the target image, can recognize similar images with common characteristics.

[0131] The result of the SEL selection step is sent to the remote server SERVI.

[0132] In a third generation step GEN3, a label called the first label LB1 is generated and associated with at least one first image IM1. In some embodiments, the third generation step GEN3 is advantageously implemented by components of the remote server SERVi. For example, the third generation step GEN3 is advantageously implemented by a processing unit of the remote server SERVi. In other embodiments, the third generation step GEN3 is advantageously implemented by components of the user terminal Tl, such as circuit 12.

[0133] In a QU AL qualification step, the first set of images ENS1 is associated with the first label LB1, so as to form a first set of images qualified ENSLq.

[0134] In certain embodiments, in which the QU AL qualification step is implemented by one or more components of the user terminal Tl, the first set of qualified images ENSlq is sent to a remote server such as the remote server SERVI.

[0135] In other embodiments, the first label LB1 is sent to the remote server SERVI, so that the qualification step is performed by one or more components of the remote server SERVI. In other words, the first label LB1 is assigned to the first set ENS1. For example, the qualification step QU AL is advantageously implemented by a processing unit of the remote server SERVI. The first qualified set ENSlq is stored in memory of the remote server SERVI. Alternatively, data encoding the association of the first label LB1 with the first set ENS1, and called annotation data DI, is stored in memory of the remote server SERVI.

[0136] Once the QU AL qualification step has been completed, the first step of the sequence for unlocking access to the user terminal Tl is validated and the second step of the sequence for unlocking access to the computer resource is accessible.

[0137] Different embodiments allow the first step of the sequence to be validated. As stated previously, according to a first embodiment, the selection and validation of the selected images allows access to the second step of the sequence. Access to the second step can be achieved without any selection check.

[0138] According to a second embodiment, a check of at least one selected image is performed to validate the first step. Thus, this first step can include generating an image of the ENS1 set whose label is known, here the label The check can be performed automatically; for example, if the result is contained in metadata associated with the image, no exchange with a server is necessary to validate the step. Alternatively, an exchange with a remote server can be used to verify that the image selection is correct.

[0139] If the set of images in the set ENS1 is coherent, that is to say that each image in the set ENS1 forms a portion of a larger image formed by different pieces representing the thumbnails, a priori, no image in the set ENS1 will be able to have a known label because by definition we seek to label unknown images.

[0140] It is understood that the embodiment in which a check of an image of the set can be carried out because its label is known is more suited to the mode in which the images entered in the window within the set ENS1 are generated randomly and come from at least one given source.

[0141] One advantage is to allow the introduction of an image whose label is known in order to perform a check.

[0142] Thus, this validation allows verification that the user has selected a coherent set of images. In this latter case, at least one image from the set is generated at a known position so that the user's selection can validate that the image has indeed been selected. It follows that the other images selected by the user were chosen with the intention of providing an element of truth from the user's perspective. Conversely, when the image whose label is known has not been selected, this generates an indicator that the selection is not qualified. Validation for proceeding to the second step of the sequence can be performed only if the image whose label is known has been selected, or conversely, validation can be initiated regardless of the selection result.According to a third example, an input image containing images previously labeled by another user can be used by the terminal user to verify the selection results. One advantage is that the selection can be directly qualified in the first step, validating this first step before proceeding to the second. Similarly, proceeding to the second step can be done independently of checking the labels of the selected images, or conversely, proceeding can only occur if the check results in a correct verification.

[0143] The user must perform the second step of the computer resource access unlock sequence to unlock access to the computer resource. In some embodiments, this second step includes the user entering a code and verifying that code.

[0144] In certain embodiments, the method 100 comprises, successively to the selection of the first image of the first PA1 panel associated with the target label, a step of selection by the user of a second image. The second image is associated with the target label.

[0145] In some embodiments, process 100 includes a step of verifying the first LB1 label.

[0146] Advantageously, information associated with at least one first selected image IM1 is known a priori. In other words, at least one first selected image IM1 can be associated with a predefined label. Thus, the step of verifying the first label LB1 includes a comparison of the first label LB1 with the predefined label, which can be performed automatically and in real time.

[0147] According to a first variant, the verification step of the first label LB1 is performed by a human. Thus, when the set ENS0 is a set of medical images, such as histological sections, the verification step can be performed by a medical expert such as a pathologist.

[0148] According to a second embodiment, the verification step of the first label LB1 includes a comparison of the first label LB1 with a prediction from a second machine learning model. For example, the second machine learning model may be partially trained. The second model may correspond to the first model whose prediction quality is to be improved. In this case, a prediction threshold may be used to verify that an image selected by the user in the first step is consistent with a minimum prediction threshold of the second machine learning model.

[0149] According to a third variant, the step of verifying the first label LB1 may be followed by the reception of a plurality of sets of qualified images from several different user terminals communicating with the remote server SERVI. More specifically, the method 100 may include, following the step of qualifying the first set of images ENS1, the implementation of the following steps: • generation of another set of ENS2 images, at least one of which is likely to represent a target. • said generation of said other set of images resulting in a random or ordered display by a graphic component of another user terminal of the set of images arranged within a geometric shape and defining a graphic element; • generation of a second target image representing the second target of interest which may be a different image from the first image or the same image but which includes the same target as the first image; • selection of at least one image from said other set by means of a selector in a graphical interface accessible via said graphical component; • generation of an image label associated with at least one other selected image corresponding to the target label; • qualification of at least one selected image by assigning the first label LB1 or another label in order to obtain a second set of labeled images: • verification of the first LB1 label including a comparison of the first labeled images with the second labeled images or a comparison of the labels with each other.

[0150] The above steps can be repeated for a plurality of other user terminals, so as to obtain a plurality of other sets of qualified images. The step of verifying the first label LB1 then includes a comparison of the first label LB1 with each of the second labels or a comparison of the images with each other. According to one example, the first label LB1 is validated when it corresponds to a minimum number of second labels.

[0151] In some embodiments, the method 100 includes, after the step of selecting at least one first image IM1, a TRAIT processing step using a graphical user terminal interface Tl, so as to obtain at least one first treated image IM1 trait.

[0152] During the TRAIT processing step, an instruction is advantageously presented to the user to perform an operation on at least one first image IM1 or on the first set ENS1. The operation is, for example, a change in a parameter of at least one first image IM1 or of the first set ENS1. The parameter is, for example, the contrast, sharpness, saturation, intensity, or a color of at least one first image IM1 or of the first set ENS1. Advantageously, the user can modify the parameter with a movable cursor displayed on the screen 15.

[0153] According to one embodiment, the manipulation may correspond to a selection of a group of pixels, a selection of at least one image, an association of an image with a keyword, called a "tag".

[0154] Another aspect of the invention relates to a method 200 for generating a crowdsourcing training database for training a machine learning model, as shown in [Fig. 8]. The output of the method 200 is a training database B(Tf), where Tf is the completion date of the implementation of the method 200. In some embodiments, Tf is a fixed date in time. In other embodiments, Tf represents a sliding instant, corresponding to an evaluation instant of the training database generated with process 200.

[0155] Advantageously, process 200 takes place over a time window [T, Tf] with T a fixed date in time corresponding to a start date of implementation of process 200. The training database B(Tf) is built progressively over the time window [T, Tf], so that B(t) denotes the training database at a time t within the time window [T, Tf].

[0156] The method 200 can be implemented using a system 20. Figure 7 shows an example of the components of the system 20. Figure 7 more specifically represents a data network NET1, which may be the internet. Several user terminals, including user terminals Ti, T2, T3, ..., TN, are connected to the data network NET1 so as to be able to communicate with the remote server SERVI.

[0157] Information can be exchanged between the remote server SERV1 and each of the user terminals Tb T2, T3.. TN by means of electronic communication systems. Each user terminal Ti, T2, T3.. TN includes components similar to the components of the user terminal Ti described and illustrated in [Fig. 2]. In particular, each user terminal T2, T3.. TN includes a graphics component such as a screen.

[0158] As before, the remote server SERVI stores a set of ENS0 images. The images in the ENS0 image set are intended to train a machine learning model and require annotation, i.e., the assignment of a label.

[0159] The method 200 comprises, during the time window [T1, T2], in a first step E1, one or more implementations of the method 100 described above by a plurality of users, each user being associated with one of the set of user terminals T2, T3, ..., T1, such that at a current instant t of the time window [T1, T2], to a user associated with the user terminal T1, corresponds a plurality of qualified sets Pqi(t) stored in a memory of the remote server SERVI. In a second step E2, the training database B(t) at time t comprises the union of the plurality of qualified sets received and stored in a memory of the remote server SERVI.

[0160] For example, for a user associated with the user terminal Ti, with i an integer between 1 and N, at a current time t in the time window [T, Tf], a plurality of qualified sets Pqi(t) has been stored in a memory of the remote server SERVI. Thus, at time t, the training database B(t) comprises the union of the pluralities of qualified sets Pql(t), Pq2(t)... PqN(t). Figure [9] schematically illustrates the training database generated at time t, B(t) and the final input database obtained at time Tf, B(Tf).

[0161] In certain embodiments, when a processing step has been implemented by one or more user terminals from among the set of user terminals Ti, T2, T3...TN, the process 200 includes a step for determining a process called global processing. For example, global processing is an average of the processing steps implemented by the user terminal(s) from among the set of user terminals Tb, T2, T3...TN. Thus, when the processing steps are image colorings by different users, each associated with a user terminal, global processing can consist of a normalization of all the colorings.

[0162] Advantageously, the global processing is saved, for example in a memory of the remote server SERVI. Thus, when the machine learning model is executed, receiving an input image, its execution includes a preliminary step of applying the global processing to the input image.

Claims

Demands

1. A method for accessing a computing resource in a two-step sequence, said method comprising a first step for qualifying a training dataset for learning a machine learning model and a second step for unlocking access to a computing resource, the first step comprising: • generation (GEN 1) of a first set (ENS 1) of images, corresponding to a segmentation of at least one given input image by a graphics component, • display of the first set of images within a graphics element inscribed within a predefined geometric shape; • generation (GEN2) of a target image (IMC) representing the target of interest (CIB1) and associated with a label named "target label"; • selection (SEL) of at least one first image (iM1) of the first set (ENS1) by means of a selector of a graphical interface accessible via said graphics component;• generation (GEN3) of a first image label (LB1) associated with at least one first selected image (IM1); • qualification (QUAL) of at least one first selected image (IM1) by assigning the first label (LB1), so as to obtain a first set of labeled images (ENSlq) intended to train a machine learning model, said selection (SEL) of said at least one first image (IM1) being associated with a first step of a sequence of access to a computer resource, and said graphic component generating an element enabling the engagement of a second step of said sequence of access to said computer resource, in which a code check is performed.

2. A method according to the preceding claim, characterized in that the first assembly (ENS1) comprises a first panel (PA1) of images including at least one image likely to correspond to a target of interest and a second panel (PA2) of images not corresponding to the target of interest.

3. A method according to any one of the preceding claims, characterized in that the images of the first set have identical or substantially identical resolution to within 10%.

4. A method according to any one of the preceding claims, characterized in that the segmentation of a given input image by a graphics component is carried out by random fragmentation of the input image into fragments of identical resolution.

5. A method according to any one of the preceding claims, characterized in that each image of the first set (ENS1) is extracted from a given input image having an identifier.

6. A method according to any one of the preceding claims, characterized in that the images in the first set of images (ENS1) represent fragments of biological organisms.

7. A method according to any one of the preceding claims, characterized in that the images of the first set of images (ENS1) represent fragments of elements whose image capture is defined at the microscopic scale.

8. A method according to any one of the preceding claims, wherein the second step of the computer resource access sequence is successive to the first step for qualifying a training dataset of the computer resource access sequence.

9. A method according to any one of the preceding claims, comprising sending the first set of labeled images (ENSlq) to a remote server and a step of verifying said first label (LB1).

10. A method according to any one of the preceding claims, wherein: • at least one first selected image (IM1) has been previously associated with a predefined label; • a first label verification step (LB1) includes a comparison of the first label (LB1) with the predefined label.

11. A method according to any one of the preceding claims, further comprising: • selection of a second image of the first panel (PA1) associated with said target label, said selection enabling verification of the value of the first label (LB1).

12. A method according to claim 11, characterized in that when the verification is correct the method comprises: • validation of the first step of the two-step sequence and generation of a graphic window to acquire a code to validate a second step and / or; • validation of the second step concomitantly with the validation of the first step, said validation of the second step corresponding to the expected code.

13. A method according to claim 11, characterized in that when the verification is incorrect the method comprises: • generation of a new window within a new geometric shape comprising a new set of images and / or; • validation of the first step of the two-step sequence and generation of a graphics window to acquire a code to validate a second step.

14. A method according to any one of the preceding claims, characterized in that the second step of the sequence involves authentication of a user with a data server or access to the individual computer workstation.

15. A method according to claim 11, wherein the first label verification step (LB1) includes a comparison of the first label (LB1) with a prediction from a second machine learning model trained to generate a prediction of the image label.

16. A method according to any one of the preceding claims, wherein the selection (SEL), generation (GEN2) of said first label (LB1) and qualification (QUAL) steps are implemented by a first user terminal, the method comprising following the qualification (QUAL) step of said first set (ENS1) of images: • Displaying the first set of images (ENS1) within a window that fits within a predefined geometric shape on a display of a second user terminal; • Generating a second target image representing the first target of interest and associated with the label named "target label"; • Selecting at least one image from said first set (ENS1) using a selector in a graphical interface accessible via a graphical component of the second terminal; • Generating the first image label (LB1) associated with the at least one selected image; • Qualifying the at least one selected image by assigning the first label (LB1) so as to obtain a second set of labeled images; • The step of verifying the first label (LB1) including a comparison of the first labeled images with the second labeled images.

17. A method according to any one of the preceding claims, characterized in that the step of selecting at least one first image (IM1) includes a multiple selection of several images.

18. A method according to any one of the preceding claims, further comprising, after the selection of at least one first image (IM1), a processing step using said graphical interface, so as to obtain at least one first processed image (IM1 trait).

19. A method according to the preceding claim, wherein the processing step includes a modification of a parameter of at least a first image (IM1), said parameter being chosen from a contrast, a sharpness, a saturation, an intensity, a color.

20. A method for generating a training database by supplying a population of users labeling images from a plurality of first sets of images for training a machine learning model, comprising: • implementation (E1) of method 100 according to any one of claims 1 to 19 by a plurality of components remote graphs, so as to obtain a plurality of first sets of qualified images; • generation (E2) of said training database by union of the first sets of qualified images from said plurality of first sets of qualified images.

21. A method for generating a training database according to claim 20, characterized in that step (E1) implements the method according to one of claims 13 or 14, so as to obtain a plurality of at least one first processed image (IM 1 trait), said method further comprising: • Determination (E3) of a processing referred to as global processing from said plurality of at least one first processed image (IM 1 trait).

22. A method according to any one of claims 20 or 21, further comprising: • saving said global processing, such that, when the machine learning model is executed, the execution of said machine learning model includes a preliminary step of applying said global processing to an input image of said machine learning model.

Images