Method for improving the classification of a digital document from a plurality of learning models

Abstract

The invention relates to a method for improving the classification of a document from a plurality of machine learning models (ML1), characterised in that it comprises:  Receiving a digital document (D1);  Executing a first learning function (FA1) generated from a first machine learning model (ML1) trained from a first training domain (DOM1) processing the first data sequence (S1) as input and making it possible to classify a document type (TYP1) and generate a prediction of a first automatic action (A1) to be performed on the digital document (D1);  Acquiring a first corrective action (AC1) of a user relating to the modification of the movement of the first digital document (D1) to a second directory (REP2);  Generating a first annotation (ANN1);  Modifying the first training domain (DOM1) by adding the first annotation (ANN1); and  Generating a re-training of the first machine learning model (ML1).

Description

[0001] METHOD FOR IMPROVING THE CLASSIFICATION OF A DIGITAL DOCUMENT USING A PLURALITY OF LEARNING MODELS

[0002] Field of invention

[0003] The field of the invention is that of methods and systems for improving the classification of a digital document, in particular for its recording in a given memory space and for its exploitation in a memory resource.

[0004] State of the art

[0005] There are solutions for analyzing digital documents based on machine learning models. State-of-the-art solutions generally allow the extraction of either data of interest relating to the type of document in question to classify it, or data of interest to exploit the document in a certain way by performing operations such as anonymization, verifications on named entities, etc.

[0006] Prior art solutions rely on learning that enriches a model in order to improve the classification efficiency of a learning function. A problem with state-of-the-art solutions is to involve users of a classification solution by requiring actions from them dedicated to improving the learning of the learning function. These actions may include label corrections, modifications of extracted data, annotations, etc.

[0007] However, these dedicated actions are not always carried out correctly due to the heterogeneity of human behaviors to contribute to the retraining of a machine learning model and the fact that corrective actions are not always carried out.

[0008] There is a need to define a solution that allows the retraining of a machine learning model that is simple and almost transparent to users who exploit the classification results.

[0009] Summary of the invention According to a first aspect, the invention relates to a method for improving the classification of at least one document from at least one machine learning model characterized in that it comprises:

[0010] ■ Receipt of a first digital document;

[0011] ■ Execution of a first learning function generated from a first machine learning model trained from a first training domain making it possible to generate a prediction of a classification of the first digital document within a first class to carry out a first automatic action on the digital document,

[0012] ■ Generation of a representation of this classification and / or of the action carried out on the first digital document from a user interface;

[0013] ■ Acquisition of a first corrective action from a user;

[0014] ■ Generation of a first annotation comprising on the one hand the modified value of the prediction relating to the first corrective action and on the other hand a set of values ​​of interest extracted or generated in particular from the first digital document;

[0015] ■ Modification of the first training domain by adding the first annotation;

[0016] ■ Generation of retraining of the first machine learning model.

[0017] According to one embodiment, the actions may correspond for example to:

[0018] - moving a digital document from one directory to another;

[0019] - a copy of a document in a folder / directory;

[0020] - renaming a digital document,

[0021] - creation of a new document and / or a new directory of a data space.

[0022] The invention is implemented by computer.

[0023] According to one aspect, the invention relates to a method for improving the classification of at least one document or data from a document and / or the prediction of data qualifying at least one document or data from a document from at least one machine learning model characterized in that it comprises:

[0024] ■ Receipt of a first digital document;

[0025] ■ Execution of a first function trained from a first training domain allowing the generation of a prediction of a first automatic action to be carried out on the digital document,

[0026] ■ Generation of a representation of this prediction and / or of the action carried out on the first digital document from a user interface;

[0027] ■ Acquisition of a first corrective action from a user;

[0028] ■ Generation of a first annotation comprising on the one hand the modified value of the prediction relating to the first corrective action and on the other hand a set of values ​​of interest extracted or generated in particular from the first digital document;

[0029] ■ Modifying the first training domain by adding the first annotation after collecting at least one annotation;

[0030] ■ Generation of retraining of the first machine learning model.

[0031] According to one embodiment, the learning function is a function generated from a first machine learning model trained from a first training domain. According to another case, the learning function is a predefined function making it possible to extract data from a data set to generate a prediction, said data set being able to evolve over time by an addition or selection of data or an enrichment of data.

[0032] According to a second aspect, the invention relates to a method for improving the classification of at least one document from at least one machine learning model characterized in that it comprises:

[0033] ■ Receipt of a first digital document;

[0034] ■ Execution of a first learning function generated from a first machine learning model trained from a first training domain making it possible to generate a prediction of a classification of the first digital document within a first class to carry out a first automatic action on the digital document, said first action comprising at least one movement of said first digital document to a first directory,

[0035] ■ Generation of a representation of this classification and the action carried out on the first digital document from a user interface;

[0036] ■ Acquisition of a first corrective action from a user relating to the modification of the movement of the first digital document to a second directory;

[0037] ■ Generation of a first annotation comprising on the one hand the modified value of the prediction relating to the first corrective action and on the other hand a set of values ​​of interest extracted or generated in particular from the first digital document;

[0038] ■ Modification of the first training domain by adding the first annotation;

[0039] ■ Generation of retraining of the first machine learning model.

[0040] Advantageously, the directory into which the first document is moved depends on the classification. Thus, depending on the classification predicted by the first trained machine learning model, a given directory is targeted for moving said first document. The association between a class and a directory can be preconfigured. Generally, all classes can be associated with one or more directories.

[0041] According to one aspect, the invention relates to a method for improving the classification and / or prediction of at least one document or data from a document from at least one machine learning model characterized in that it comprises:

[0042] ■ Receipt of a first digital document;

[0043] ■ Execution of a first function trained from a first training domain making it possible to generate a prediction of a first automatic action to be carried out on the digital document, ■ Generation of a representation of this prediction and / or of the action carried out on the first digital document from a user interface;

[0044] ■ Acquisition of a first corrective action from a user;

[0045] ■ Generation of a first annotation comprising on the one hand the modified value of the prediction relating to the first corrective action and on the other hand a set of values ​​of interest extracted or generated in particular from the first digital document;

[0046] ■ Modifying the first training domain by adding the first annotation after collecting at least one annotation;

[0047] ■ Generation of retraining of the first machine learning model.

[0048] According to one embodiment, the learning function is a function generated from a first machine learning model trained from a first training domain. According to another case, the learning function is a predefined function making it possible to extract data from a data set to generate a prediction, said data set being able to evolve over time by an addition or selection of data or an enrichment of data.

[0049] According to one aspect, the invention relates to a method for improving the classification and / or prediction of at least one document or data from a document from at least one machine learning model characterized in that it comprises:

[0050] ■ Receipt of a first digital document;

[0051] ■ Execution of a first learning function generated from a first machine learning model trained from a first training domain making it possible to generate a prediction of a first automatic action to be carried out on the digital document,

[0052] ■ Generation of a representation of this prediction and / or of the action carried out on the first digital document from a user interface;

[0053] ■ Acquisition of a first corrective action from a user; ■ Generation of a first annotation comprising on the one hand the modified value of the prediction relating to the first corrective action and on the other hand a set of values ​​of interest extracted or generated in particular from the first digital document;

[0054] ■ Modifying the first training domain by adding the first annotation after collecting at least one annotation;

[0055] ■ Generation of retraining of the first machine learning model.

[0056] According to one embodiment, the first corrective action relating to the modification of the movement of the first digital document is carried out by moving the first digital document from the first directory to a second directory by a user action from a document explorer, an algorithm for detecting a change in memory resources allocated to a digital document being implemented.

[0057] According to one embodiment, the first corrective action relating to the modification of the movement of the first digital document is carried out by a modification of the class of the first digital document by a user action from a user interface presenting a set of data characteristic of the first digital document.

[0058] According to one embodiment, the method comprises:

[0059] ■ Receipt of a second digital document;

[0060] ■ Execution of a second learning function generated from a second machine learning model trained from a second training domain making it possible to generate a prediction of a classification of the second digital document within a second class to carry out a second automatic action on the second digital document, said second action comprising at least one automatic renaming of said second digital document,

[0061] ■ Generating a representation of this classification and the action performed on the first digital document from a user interface; ■ Acquiring a second corrective action from a user relating to the modification of the name of the second digital document;

[0062] ■ Generation of a second annotation comprising on the one hand the modified value of the prediction relating to the second corrective action and on the other hand a set of values ​​of a second sequence of data extracted or generated in particular from the second digital document;

[0063] ■ Modification of the second training domain by adding the second annotation;

[0064] ■ Generation of retraining of the second machine learning model.

[0065] According to one embodiment, the second corrective action relating to the modification of the name of the second digital document is carried out by a modification of the name of the second digital document directly on the file corresponding to the second digital document.

[0066] According to one embodiment, the second corrective action relating to the modification of the name of the second digital document is carried out by a modification of the class of the second digital document by a user action from a user interface presenting a set of data characteristic of the second digital document.

[0067] According to one embodiment, the method comprises:

[0068] ■ Receipt of a digital document;

[0069] ■ Execution of a third learning function generated from a third machine learning model trained from a third training domain to generate a prediction relating to the detection of a class of a characteristic date present in the digital document and extract said characteristic date from the first document;

[0070] ■ Generation of a representation of the classification of at least the characteristic date extracted from a user interface;

[0071] ■ Acquisition of a third corrective action from a user relating to the modification of the class of the characteristic date of the first digital document; ■ Generation of a third annotation comprising on the one hand the modified value of the prediction relating to the first corrective modification and on the other hand a set of values ​​of interest extracted or generated in particular from the digital document;

[0072] ■ Modification of the third training domain by adding the third annotation;

[0073] ■ Generation of retraining of the third machine learning model.

[0074] According to one embodiment, the third learning function is executed prior to the first learning function, the corrected prediction being data from the set of values ​​of interest of an annotation, said corrected prediction modification resulting in the modification of the first training domain and / or the modification of the second training domain.

[0075] According to one embodiment, the method comprises receiving a dependency graph, said dependencies being defined between at least the first learning function and a second learning function, said dependency graph comprising a description of the inputs of at least one training domain, said inputs comprising at least one value corresponding to an output of at least one learning function.

[0076] According to one embodiment, the method comprises:

[0077] ■ Acquisition of a dependency graph between a plurality of learning functions chosen from a set of learning functions comprising the first learning function, the second learning function and the third learning function;

[0078] ■ Addition to at least one set of values ​​of interest produced when creating an annotation of a modified prediction of a given learning function in such a way as to:

[0079] ■ Modification of a training domain of a learning function other than the given learning function;

[0080] ■ Generating a retraining of the machine learning model.

[0081] According to one embodiment, following an automatic action carried out by a learning function, a first notification is sent via a data network, said first notification comprising access to a user interface making it possible to modify the annotations associated with this action.

[0082] According to one embodiment, the first machine learning model is generated from the definition of a first digital document model comprising the identification of a plurality of characteristic data and areas of interest of said first document model and from a user interface.

[0083] According to one embodiment, following a corrective action generated by a user, the first digital document model is used to extract context data from the modified prediction to enrich the annotation that is created.

[0084] According to one embodiment, the definition of a model of a first digital document comprises the generation of a form comprising a set of choices defining annotations of said model, said annotations making it possible to update the first training model.

[0085] According to one embodiment, a plurality of first documents are received from the same class, and a plurality of machine learning models are applied, the method comprising generating a plurality of predictions, the method comprising selecting a prediction, said selection of said prediction making it possible to update the first training domain.

[0086] According to another aspect, the invention relates to a system comprising an electronic terminal of a user comprising at least one computer, a memory, a display and a communication interface for transmitting messages over a data network to at least one first server comprising hardware resources for executing the first learning function, the second learning function, the third learning function and a memory for recording the learned models in order to execute the steps of the method of the invention, the terminal comprising a user interface for acquiring an annotation comprising a modification of a prediction, said modification resulting in the retraining of at least one machine learning model. According to another aspect of the invention relates to a method for improving the classification of a digital document from a plurality of machine learning models, characterized in that it comprises:

[0087] ■ Receipt of a digital document;

[0088] ■ Execution of a first learning function generated from a first machine learning model trained from a first training domain and making it possible to classify a first type of document and generate a first prediction of a first automatic action to be carried out on the digital document,

[0089] ■ Execution of a second learning function generated from a second machine learning model trained from a second training domain to generate a second prediction making it possible to extract and classify data characteristic of the first document;

[0090] ■ Generation of a representation from a user interface of the first prediction associated with the classification and the action performed, of the second prediction comprising the date extracted from the first digital document and of the class of the date extracted from the first digital document;

[0091] ■ Acquisition of a first corrective action from a user relating to the modification of at least one prediction;

[0092] ■ Generation of a first annotation comprising on the one hand the modified value of the at least one prediction relating to the first corrective action and on the other hand a set of values ​​of interest extracted or generated from the first digital document;

[0093] ■ Acquisition of a dependency graph between at least the first learning function and the second learning function;

[0094] ■ Modifying the first training domain and / or the second training domain based on the annotation and the dependency graph by adding the first annotation; ■ Generating a retraining of the first machine learning model and / or the second learning model.

[0095] According to one embodiment, the retraining of the machine learning models is generated based on the dependency graph representing the links between learning functions.

[0096] According to one embodiment, the method comprises applying the first machine learning model and / or the second machine learning model to the first digital document or to another document received subsequent to the first digital document.

[0097] In one embodiment, the values ​​of interest include a position of a data item in the document or in a page of the document. In another example, a value of interest includes a sequence of discrete natural language symbols that precedes and / or follows the corrected prediction in the document. In one example, a value of interest is an intermediate prediction of a learning function that is in a same sequence as a learning function executed to produce the digital document that was the subject of a corrective action.

[0098] Brief description of the figures

[0099] Other characteristics and advantages of the invention will emerge on reading the detailed description which follows, with reference to the appended figures, which illustrate:

[0100] Fig. 1: a learning function architecture illustrating the retraining of a machine learning model of a function from a correction made to the result produced by another learning function according to the method of the invention;

[0101] Fig. 2: areas of interest of a page of a digital document from which a set of data is extracted by one or more learning function(s) according to the method of the invention;

[0102] Fig. 3: a system of the invention comprising different resources for implementing the method of the invention,

[0103] Fig. 4: an example of generation of a new document model following an annotation produced by the method of the invention. Figure 1 represents a software architecture implementing different learning functions FAi, FA2 and FA3 performing automatic operations on a digital document D1. Each of the learning functions FA1, FA2, FA3 calculates from the input document D1 to be processed an intermediate prediction respectively noted Pi, P2 and P3. According to one embodiment, a prediction graph GRAPHP makes it possible to calculate from a calculator a final prediction Pf to automatically perform an action Ai.on the digital document. The action Ai can be to move the document, to rename it or to use part of the data of the document D1 to generate another document.

[0104] Figure 1 also represents the operation Ci(Di) which consists of generating a representation of this action on a display. The representation can be generated in real time on a user interface or be represented after the prediction, classification or action. This representation can be made from a user interface. A file explorer can be used to represent the movement of a folder.

[0105] Figure 1 represents an operation corresponding to a corrective action AC1 by a user. The corrective action may correspond to a movement carried out by a user of a document which has been deemed incorrectly classified. Other corrective actions are possible according to the method of the invention, such as renaming or correcting a value in the document or in a generated document.

[0106] The corrective action results in the automatic creation of an ANN1 annotation. This ANN1 annotation includes a corrected value of the final prediction Pf resulting from the corrective action. For example, the corrected prediction can be a directory name of a file system. In addition, the ANN1 annotation includes a number of data that are related to the corrected prediction.

[0107] Figure 1 represents a GRAPHD dependency graph making it possible to establish all the intermediate predictions Pi, P2, P3 and the machine learning models ML2, ML1, ML3 and the learning functions to be corrected to take into account the change in the value of the final prediction Pf. The method of the invention makes it possible, thanks to this dependency graph, to collect the modified values ​​of the intermediate predictions, here P2', and to pass this new value into the training domain of the learning functions. An advantage is to allow relearning of the learning functions having only contributed to calculating an erroneous value of the intermediate prediction.

[0108] In the case of Figure 1, we understand that the intermediate prediction P2 calculated by the second learning function FA2 is used as input to the first learning function FA1. As a result, the training model ML2 is retrained, just like the training model ML1. The GRAPHR dependency graph allows this retraining sequence to be established.

[0109] Figure 4 represents a scenario in which a document model MOD(Di) is used to process a document type recognized by a learning function FA1. In this scenario, only one learning function FA1 is used, however other scenarios could implement different learning functions to classify the document D1 received as input. A prediction Pi which can be in this case an intermediate prediction and a final prediction is generated by the first learning function FA1. This prediction Pi in this case allows for example to classify the document D1 and to apply an action A1 automatically on the document. A corrective action AC1 is initiated by a user. This corrective action AC1 generates a new value of the prediction PT.

[0110] The invention not only allows to generate a new annotation automatically to retrain the model but also to modify the MOD document model which will be used for the next actions. The new MOD document model can be generated by a calculation rule, a predefined algorithm or even by a new learning function allowing to be trained by user data and / or data from the annotations produced by the corrective actions.

[0111] Document type

[0112] Document D1 may be a technical document such as a technical specification, a user manual, an assembly manual, an accounting document such as a balance sheet, a quote, an invoice, a letter, a plan, a certificate, an official document, a medical certificate, a form or any other document produced by an organization, an automated process or an individual. The documents processed within the framework of the invention are preferably documents that relate to a type of document or a family of documents. The type of document may be defined by a generic name or a label allowing it to be associated with a type of document. It is understood that an invoice may be a type of document in an organization as well as a technical specification of a product.

[0113] The document may include different formats such as a .txt format, a .pdf format, a .png format, a pg format, a Json format, an .xml format, a .doc or docx format or any other document format that allows encoding a plurality of discrete symbols in a natural language.

[0114] In some embodiments, the document may be an electronic mail message. In one embodiment, the digital document is an image file, a video file, or an audio file. In one embodiment, the document is a set of data received in real time, so it may be a video stream or an audio stream.

[0115] Receipt of the document

[0116] The document Di may be received from a communication interface such as a network card for acquiring data from a remote entity such as a data server. Alternatively, the document Di may be stored in a memory within which a function is used to retrieve the document stored therein in order to perform an operation on the latter document.

[0117] An objective of the invention is to extract data from said document to classify it with the best probability in order to perform actions on the document, in particular to automatically save it in a given memory.

[0118] Learning function

[0119] The method of the invention comprises executing at least one learning function generated by a machine learning model trained from a training data set called a training domain.

[0120] According to an alternative, the method of the invention comprises the execution of at least one learning function which is configured to query and / or extract data within a history. The function is said to be “learning” in this case because the history can include new data which is aggregated over time. The history can take the form of a database or a file. The updating of the history with new data can be carried out automatically or from a user action.

[0121] In the latter case, the learning function is not obtained from a machine learning model but can result from a predefined parameter setting. The parameter setting can, for example, correspond to the type of function, the coefficients of the latter and / or the directory(ies) which is / are used to extract and / or record information.

[0122] According to one embodiment of the invention, the method comprises the execution of a plurality of learning functions trained from different models. The execution of the plurality of learning functions makes it possible in particular to perform different functions on the document Di.

[0123] According to one embodiment, when a plurality of learning functions are executed, the method of the invention comprises applying a prediction graph GRAPHp and / or a dependency graph GRAPHD.

[0124] The GRAPHD dependency graph includes, in particular, for each action to be performed, a first sequence Si for executing the various learning functions in order to perform said given action. The first sequence Si may result from a predefined configuration. It corresponds to the sequence of a series of algorithms. According to one example, each algorithm is executed from a learning function. It is recalled that the learning function is a machine learning model trained to calculate a prediction from input data. According to one embodiment, not all the algorithms in a sequence are solely learning functions; other algorithms such as algorithms comprising calculation rules may be sequenced among the functions implemented in a first sequence Si.

[0125] The first sequence Si of the GRAPHD dependency graph includes a set of learning functions and / or algorithms that are configured to be chained together in an automation. In addition, the first sequence Si includes the scheduling conditions and / or execution conditions of each function or algorithm. This means that the first sequence Si includes parameters for scheduling functions / algorithms with each other and parameters for validating, timing, or invalidating the execution of a function / algorithm.

[0126] The GRAPHD dependency graph also includes a second sequence S2 for each user-corrected action of retraining machine learning models that were executed during the first sequence Si. The second sequence S2 allows to take into account all the models affected by a prediction error when the model depends directly or indirectly on the erroneous prediction.

[0127] The second sequence S2 of the GRAPHD dependency graph includes a set of retrainings for each model of the learning functions or modifications of the algorithm parameters. The second sequence S2 allows these model trainings or these algorithm modifications to be reconfigured within an automation forming a feedback loop. In addition, the second sequence S2 includes the scheduling conditions and / or execution conditions of each model retraining or each algorithm modification. This means that the second sequence S2 includes parameters allowing the retrainings or modifications to be scheduled between them and parameters allowing the execution of a retraining or an algorithm modification to be validated, delayed or invalidated.

[0128] A GRAPHp prediction graph allows a final prediction to be calculated from a plurality of intermediate predictions generated by a plurality of learning functions.

[0129] Learning functions can be generated by machine learning models with their own architectures. For example, a network architecture can be a machine learning model of the RNN type, meaning "Recurrent Neural Network" in the English literature, or an LSTM type model, meaning "Long short-term memory" in the English literature, can be used. For another example, the machine learning model can be a Transformer, such as GPT-3, meaning "generative Pre-Training Transformer", which is a model based on the Transformer architecture, i.e., some layers of the model have the structure of a Transformer. An advantage of using a pre-trained network, for example a Transformer type, is to use their good capabilities to process data defining documents containing discrete symbols encoded in a natural language.According to one embodiment, the invention is compatible with the use of already pre-trained networks already existing, for example on platforms accessible from the Internet.

[0130] A pre-trained BERT-type network, designated in the literature as “Bidirectional Encoder representations from Transformers”, whose model also includes certain layers having the structure of a Transformer, can also be implemented within the framework of the present invention.

[0131] For example, when few classes are addressed by a classifier, for example between 5 and 10 classes, a machine learning model architecture may include the implementation of binary trees such as "Random Forest" or "XGBoost" or a convolutional neural network called CNN.

[0132] In another example, when a large number of classes are addressed by a classifier, a machine learning model architecture may include the implementation of a Transformer model, for example of the BERT type, or a convolutional neural network called CNN.

[0133] Machine learning models may be configured within the scope of the invention to extract data from digital documents such as dates, addresses, proper names, tables, signatures, structured information, sentiments and / or target phrases. A “regex” type model architecture or a library such as “spacy” comprising different types of models, or a CNN or a “transition-based models” type model may be used.

[0134] The models used to extract objects, named entities can implement a "transformer" type model or a BERT type model.

[0135] According to one embodiment, an architecture for recognizing patterns or motifs, such as a signature, in a digital document may implement a "YOLO" type or "autoencoders" type model.

[0136] Classification function According to one embodiment, at least one learning function is executed in order to classify a received digital document Di. The learning function FAi can implement, for example, an analysis aimed at extracting a document type, a date, a named entity such as an organization name or an individual name, an address or any other data making it possible to execute an action based on knowledge of this data once extracted. In the latter case, certain data extractions can be used to perform a classification. However, according to another example, the classification can be performed without data extraction. For example, a classification of a received document does not necessarily implement an extraction of data from said document. According to one embodiment, one or more learning function(s) are executed to generate a prediction of the class of the extracted data.Each learning function can generate an intermediate prediction in order to calculate a final prediction.

[0137] According to one example, extracting a date from a received Di document can allow this document to be classified in a "to be processed" directory if it is an extraction of issue date(s) positioned in the header of the document or in a "to be signed" directory if a signature date is noted at the end of the document, on the last page.

[0138] This example shows that the method of the invention can be applied from a single extraction to classify the document. However, according to different embodiments, the classification can result from a plurality of data extractions and an analysis of all the extracted data.

[0139] As an example, Figure 1 represents three learning functions FAi, FA2, FA3 which allow data to be extracted from a received D1 document.

[0140] In this example, a first function FA1 allows you to automatically extract a characteristic date DATE1, such as a due date, the second function FA2 allows you to extract an entity named NAME1 such as an organization name, and the third function FA3 allows you to extract a document type TYPE1, such as an invoice or a quote.

[0141] The extraction of the three characteristic data {DATE1, NAME1, TYPE1} makes it possible to generate a prediction based on all three predictions generated by the three learning functions FA1, FA2, FA3. This prediction allows a classification of the document which can lead to the execution of an automatic action. Each extraction is carried out from each learning function which has been defined from a machine learning model trained from a training domain.

[0142] The final prediction calculated from the three intermediate predictions of each of the learning functions makes it possible to automatically perform an action on the document Di.

[0143] The extracted data can implement learning functions that exploit variables related to the font, style, or font size. The position of certain elements can also be used, or context data for the data to be extracted.

[0144] According to an exemplary embodiment, a prediction graph GRAPHp can be implemented to calculate the final prediction Pf. This prediction graph allows, according to the embodiments:

[0145] ■ weight intermediate predictions Pi to calculate a final prediction and / or;

[0146] ■ exclude actions or action scenarios based on the values ​​or ranges of values ​​of certain intermediate predictions and / or;

[0147] ■ prioritize actions or sequences of actions between them.

[0148] ■ Wait for the receipt of an intermediate prediction from a given learning function after calculating a set of intermediate predictions from a plurality of learning functions to initiate an action. “Wait” means creating an alert upon receipt of data produced by a given function.

[0149] According to an exemplary embodiment, the learning functions can be executed on different documents to generate an action on another document already created or which will be created. The documents can be messages from a messaging system or files.

[0150] According to an exemplary embodiment, a part of a digital document Di is extracted, such as a table.

[0151] Actions

[0152] Different actions can be generated depending on the prediction made and on a given configuration. In other words, a configuration makes it possible to define a link between the reception unit of a document Di, the choice of learning functions to be executed and the action to be carried out under the conditions of a final prediction generated.

[0153] According to one embodiment, different actions can be generated by the method of the invention.

[0154] A first action Ai includes automatically saving the document Di in a predefined directory. According to an example, this action can be accompanied by another action aimed at deleting the first document which was saved prior to processing in a temporary directory.

[0155] A second action A2 includes the automatic renaming of a document D2. According to one example, this action can be accompanied consecutively or before an action Ai of saving the document D2. In this case, the document D1 and D2 can be the same document, when several actions are generated on the same document.

[0156] A third action A3 involves extracting a portion of interest from document D3 and integrating it into another given document, called an edited document From. This document may be newly created. Here again, it is understood that the third action A3 may be combined or consecutive or prior to an action Ai or A2 or Ai and A2.

[0157] The edited document De may be saved in a predefined directory. According to one example, this third action A3 may be accompanied by another action A3' aimed at extracting another portion of another document D3' of another type or of the same type to produce a single edited document De from two portions extracted from two documents D3, D3'. For example, a document of the "Purchase Order" type may be generated from content extracted from a quote and from a document of the acceptance email type or from the quote signed by a party. This third action A3 may be accompanied by a plurality of actions aimed at producing an edited document and / or actions aimed at validating the editing of such an edited document. According to one example, at least one extracted portion is used to edit a new document De. According to another example, several portions extracted from the same document D3 are used to produce an edited document De.

[0158] According to an embodiment in which a table is extracted with values ​​from a first document D3, an action may be to regenerate another document comprising values ​​from said table combined for example with other values ​​from another table of another document.

[0159] In another example, a fourth action may correspond to the creation of a category of a parameter or an extracted value. This may be, for example, a new named entity, i.e., a new organization that is not yet referenced in a database. In this case, the final prediction calculated from a plurality of learning functions having, for example, determined a document type, a date, an amount, and a new organization name, is the creation of a new directory. The new directory may correspond to the directory in which all the invoices of this new named entity will be recorded.

[0160] In another example, a fifth action corresponds to the segmentation of a document Di into at least two documents. The segmentation may correspond to a set of pages of a document. A segmented document can then be classified by another learning function and be processed by another learning function to extract other data to generate another action. Thus, this example illustrates that an action can include several sub-actions.

[0161] Representation of the classification

[0162] According to one embodiment, a representation of the decision, i.e. the classification of the document Di or the action is generated. The representation is preferably produced from a user interface from a display.

[0163] The representation can be, for example, an iconography of a file saved in a directory, i.e. an icon representing the document D1. This representation is particularly suitable when the action aims to automatically move a document following the calculation of the final prediction.

[0164] In one example, the representation can be generated from an interface representing the classification of the document Di. In this case, a keyword or a class name can be associated with the document Di. This representation is useful when one wishes to verify the classification of a document and the score associated with its prediction. Another interest is to identify the variables used to calculate the final prediction. In another example, the representation of an action is the display of the name of the document Di that has been renamed automatically. In the latter case, the action can correspond to the renaming of the document Di.

[0165] In another example, the representation of an action is the opening of an edited or created document to display at least one area of ​​interest of the digital document.

[0166] According to one embodiment, the final prediction or the classification made from the final prediction generates the emission of a notification sent automatically to a user whose email address is predetermined or whose email address is deduced from metadata of an action performed by said user. The notification includes a link to a resource of a remote entity and displays either the prediction, the action, or the classification by means of a user interface. The user is then invited to validate the prediction, the classification or the action or to correct it.

[0167] Automatic prediction verification

[0168] According to one embodiment, the method of the invention comprises a step of verifying the final prediction with a consistency test of the prediction. The consistency test can be executed from a document model, also called a "template" in English literature, or a scenario model aimed at verifying the consistency of a final prediction, for example from rules.

[0169] A document model is used to validate that the final prediction results in generating, modifying, or creating a document that has a given predefined model. This verification allows, for example, testing the consistency of expected data in a given area of ​​a document with the area actually checked in a document model.

[0170] The document model can be associated with a score that is automatically calculated based on the annotations that have been previously issued and therefore the domain training of each learning function acting on a digital document Di comprising a document model. An advantage is to quickly identify whether a document model is reliable or whether it is necessary to train the machine learning model to improve the fidelity of the document model. A scenario model makes it possible to validate that the final prediction results in an action that fits into a list of expected tasks or results in a classification without affecting other parameter values ​​or at least that these values ​​remain within a given range.

[0171] Figure 2 represents an example of a document model that can be used to check the consistency of a prediction and possibly calculate a score for the calculated prediction.

[0172] Figure 2 represents a model in which a number of objects OB1, OB2, OB3 representing headings, illustrations or paragraphs can be labeled to indicate characteristic data fields. In the model of Figure 2, a first date field noted DAT1 and a second date field DAT2 representing for example respectively a mail date and a signature date.

[0173] In this example, the NM1 field is that of a named entity, for example, the name and address of an organization receiving the document. Another NM2 field is used to label another named entity, such as another organization name, for example, an organization publishing the document. In this example, a signature zone is indicated as SIG1 in Figure 2 and represents an area where a signature is expected.

[0174] According to an exemplary embodiment, a document model such as that of Figure 2 can be used initially to generate a first training of a machine learning model making it possible to execute one or more learning functions. According to another example, the model is constructed from real documents and makes it possible to carry out a consistency check with predictions calculated by learning functions.

[0175] In another example, a document template can be used to generate another document template.

[0176] Corrective action

[0177] The method of the invention makes it possible, from a user interface, to carry out a corrective action carried out by a user on the document Di.

[0178] According to one embodiment, the corrective action AC1 corresponds to a movement of the document D1 which was previously automatically recorded by the execution of at least one learning function. A movement resulting from a corrective action ACi makes it possible to modify the memory resource storing the document or to modify the link allowing access to it.

[0179] According to another embodiment, the corrective action AC2 corresponds to a renaming of a document D1 or D2 which was previously renamed automatically by the execution of at least one learning function. A renaming of a document D1 or D2 resulting from a corrective action AC1 makes it possible to modify the name of the digital document D1 or D2. Here the name "D1" is used when the document was previously automatically moved into a directory before being renamed. The name "D2" is used when the document was automatically renamed without having been previously moved into a directory.

[0180] According to another embodiment, the corrective action AC3 corresponds to a correction made to a document produced by the first action such as action A3 following the execution of at least one learning function.

[0181] The acquisition of the corrective action can be carried out according to different embodiments of the invention.

[0182] According to a first embodiment, the corrective action is acquired by means of a user interface specially designed to provide a corrective action. This user interface includes a field for designating the document to which corrective action is to be taken and a field for performing the corrective action. This corrective action is then carried out in two stages, a first stage for defining which action must be taken and a second stage for executing this corrective action from the data acquired by the interface. An advantage is that it allows for better acquisition of the corrected data which can be enriched by the user interface.

[0183] According to a second embodiment, the corrective action is acquired directly from a file explorer allowing to move a file from one directory to another directory or to access the name of a file to edit it for example by selecting an iconography representing the digital document. An advantage of this option is to be carried out in only one time directly thanks to the user action. According to a third embodiment, the corrective action is acquired directly within the document Di, D2, or D3 from a user interface allowing to view the content of the document to edit at least one data of this document. The editing consists of a correction of a document data, it can be a numeric or alphabetic data such as a date, a named entity. According to an example, it can be a portion of a page such as a paragraph, an image or a portion of the page.As another example, it may be an area of ​​one page or a plurality of pages.

[0184] According to a fourth embodiment, which is an improved mode of the third embodiment, a document model is used to interpret the corrective action of a user within a document which was generated by the first action. The advantage of using a document model is to automatically recognize modified information in a file to extract additional data associated with this modification or to enrich the modified information with additional data from another source. An advantage is to identify on the one hand the erroneous prediction made by the learning function and to determine the set of parameter values ​​which contributed to producing this erroneous prediction.

[0185] For example, if a list of values ​​in a document is corrected by a corrective action, either by deleting a value, adding a value, or modifying a value in the list, then the model can recover the set of values ​​in the list that has not been corrected and recover all the data used to produce the erroneous prediction and the correct predictions. One interest is to improve the training of machine learning models.

[0186] Generating a modified value of the prediction

[0187] According to an exemplary embodiment, the corrective action results in automatically generating a corrected value of the prediction and an annotation.

[0188] The method of the invention makes it possible to record the modified value representative of the corrective action AC1, AC2, that is to say the value of the final prediction. The value can be a directory name or a file name comprising a given nomenclature. The modified value of the corrective action ACi, AC2 can be recorded with the resulting value of the first action Ai, respectively A2. If the document has been recorded in a folder "A" and a corrective action has led to moving this document to a new directory "B", the method of the invention makes it possible to record this new value "B" and to associate it with the old value "A".

[0189] Generating an annotation

[0190] At the end of the corrective action, an annotation is automatically generated following the corrective action. The annotation may include the new value resulting from the corrective action, i.e. the corrected value of the prediction and possibly the old value of the first action that was carried out on the first digital document, i.e. the final prediction.

[0191] In addition, the generated annotation advantageously includes a set of values ​​of interest extracted or generated in particular from the first digital document D1 or any other document analyzed. These values ​​of interest are associated with variables of a system such as an organization name, a due date or a delivery date or even an address, etc. These variables of interest are themselves associated with metadata that can be used in an annotation in order to improve the training of a domain.

[0192] The annotation may include, in particular, all the predictions of each learning function and all the data of the first document D1 used to calculate each prediction. Among these data, it may be a text field, a date field or a number. These data may include indicators of the presence of iconography, a signature, a header or even initials, etc. Among these data, characteristic values ​​of a geometric zone of a page of the document, of one or more characteristic positions, or even a page number of a document, etc.

[0193] A large amount of data, denoted as interest values ​​or interest variables, used to calculate predictions can be collected when a corrective action has been carried out in order to enrich a training domain. For example, the interest values ​​preceding a corrected prediction and the following interest values ​​in a reading order of the document can be extracted in order to be inserted into the annotation. One interest is to obtain the context data directly associated in the document with the corrected prediction. This data makes it possible, for example, to improve retraining by consolidating the contextual controls of the corrected prediction.

[0194] According to one embodiment, the annotation is recorded in a memory.

[0195] In one example, the annotation is generated according to a predefined sequence that requires saving data temporarily while waiting for other data to be calculated later so as to generate the annotation completely.

[0196] According to an exemplary embodiment, the produced annotation includes a document identifier and / or a user identifier, for example an email and / or a date on which the first action Ai was carried out and / or the date on which the corrective action ACi was carried out. These data also constitute variables of interest.

[0197] According to an exemplary embodiment, the generated annotation is displayed in a user interface. According to one example, this annotation is editable by a user.

[0198] In one embodiment, generating a modified value of a prediction or a new annotation allows for automatically issuing a notification to a given recipient user. The recipient user identifier or address may be preconfigured based on the document type or directory associated with the prediction. In another example, the recipient user is identified using metadata associated with the processed digital document.

[0199] According to one example, retraining of at least one model may be initiated after generating a given number of annotations.

[0200] Training area and training

[0201] According to a first embodiment, the annotation makes it possible to feed a training domain of at least one learning function. According to one example, a plurality of training domains of a plurality of learning functions is modified following the corrective action. According to a second embodiment, a modification of at least one predefined rule is carried out following the generation of a corrective action.

[0202] According to a third embodiment, validation of an operation is required from a given user following the generation of a corrective action. Validation of the operation may result in the retraining of a learning function from the modified training domain and / or the modification of a rule from the annotation.

[0203] For example, a modification to a rule may include adding a value to a set of possible values ​​of a variable, such as a document type, a named entity category, or a date nomenclature.

[0204] The method of the invention makes it possible to take into account the second sequence S2 of the GRAPHD dependency graph in order to modify the models and the algorithms. The second sequence S2 of the GRAPHD dependency graph makes it possible in particular to link the predictions produced by an algorithm according to other predictions produced by other algorithms when this is the case.

[0205] According to one embodiment, the second sequence S2 takes into account the occurrences of similar annotations or their frequency over time or their singularity to apply the retraining or algorithm modifications in a second sequence S2 aimed at retraining or modifying a plurality of functions or algorithms. For example, an annotation that has many occurrences may be in a configuration of an embodiment less frequently used to update a machine learning model of a given learning function. Conversely, an annotation having few occurrences may be taken into account more quickly by the retraining process.

[0206] According to one embodiment, the corrective action AC1 or AC2 or the modification of the prediction results in the modification of a parameterization of the second sequence S2 in order to generate a retraining sequence adapted to the corrective action or the modified prediction. One advantage is to retrain the models having only contributed to generating an erroneous prediction without affecting the retraining of the machine learning models of the functions having produced non-erroneous intermediate predictions.

[0207] The GRAPHD dependency graph therefore makes it possible to optimize useful retraining operations without affecting the entire processing chain - organized by the first Si or the second S2 sequence. Furthermore, the dissociation in the GRAPHD dependency graph of the first sequence and the second sequence makes it possible to execute functions while organizing their retraining according to the implementation of the process.

[0208] History management

[0209] According to one embodiment, the method comprises a step of validation by a user from a user interface of a retraining of a set of documents already classified in a directory. An interest of this characteristic during a corrective action resulting in a change of prediction likely to affect a modification of the behavior of a learning function is to pass this behavior on to a history in order to update a document classification set.

[0210] This implementation works with different actions such as automatically moving a received document or renaming a document or segmenting or creating a document.

[0211] According to another embodiment, the validation step, on the contrary, makes it possible to avoid a repercussion of a change in the behavior of a function on a history of documents already processed. Thus, the user can dissociate, depending on the actions carried out, a different processing of the history depending on the use cases.

[0212] Integration into a system

[0213] According to one embodiment, the method of the invention comprises the integration of a data resource which may be a database of named entities such as an address book, a database of documents such as a messaging system or any other database comprising data likely to be queried by a query generated by the method. One advantage is to integrate a layer allowing an existing system to communicate with the machine learning models of the method of the invention. This layer makes it possible to standardize a set of data from an existing system with data that can be vectorized and taken into account as input to the learning functions or algorithms implemented by the method of the invention.

[0214] According to one embodiment, the method of the invention is implemented in a collaborative workspace accessible from at least one data server. In this case, when a file is moved from one directory to another in a collaborative workspace, an event is automatically generated. This operation may relate to each or certain user action(s). This event is generally generated in order to synchronize the different resources for a set of individuals having access to the workspace. Such an event may be an electronic notification issued by a communication interface. According to one embodiment, this event is used to activate the detection of an action of moving a document from one directory to another as part of a corrective action carried out by a user.

[0215] According to another embodiment, the method of the invention is implemented in a workspace in which an algorithm is configured to browse the changes made within a directory such as adding a document, deleting a document, renaming a document, etc.

[0216] This algorithm allows to automatically detect a change and to generate a modification of the prediction and therefore to generate a new annotation.

[0217] Figure 3 represents an example of the elements of a system of the invention making it possible to implement the method of the invention. Figure 3 represents a data network NETi which can be the internet network. A first user terminal Ti allows access to a remote server SERVi allowing in particular to host a collaborative workspace. This space includes in particular memory resources for storing received digital documents and processing them automatically. A second server SERV2 comprises the means making it possible to generate the models and to execute certain functions and algorithms of the method of the invention. According to the embodiments, the models and the configurations of the learning functions can be stored directly on the server hosting the collaborative workspace on which the method of the invention is executed. However, in a preferred mode, a server dedicated to their storage and execution is implemented.

[0218] According to one embodiment, one or more configuration files may include additional data for training or applying a model that may include parameters, intermediate results, or annotations generated following a corrective action. The configuration or the document using the configuration may be downloaded at any time and put on hold until all the data and documents necessary for applying or retraining the models are associated. According to one case, the configurations are associated with the documents if one or more conditions are validated. A condition may be the execution of a function or a command.

[0219] One benefit is to allow the collection of documents and configuration data from multiple systems that do not communicate with each other, and to only apply models to documents when enough documents and data are available.

[0220] One advantage is that it allows new documents to be processed to be considered independently of any retraining that would be reapplied. The configuration for retraining one or more learning function(s) can be activated or programmed following the processing of a plurality of documents. Thus, retraining can take into consideration a plurality of input documents.

[0221] Depending on the scenario, a corrective action on a single document can be taken into account to generate a new prediction for a plurality of documents.

Claims

CLAIMS 1. Computer-implemented method for improving the classification of at least one document from at least one machine learning model (MLi) characterized in that it comprises: ■ Receipt of a first digital document (Di); ■ Execution of a first learning function (FAi) generated from a first machine learning model (MLi) trained from a first training domain (DOM1) making it possible to generate a prediction (Pi) of a classification of the first digital document (Di) within a first class (Ci) to carry out a first automatic action (Ai) on the digital document (Di), said first action (Ai) comprising at least one movement of said first digital document (Di) to a first directory (REPi), ■ Generation of a representation of this classification (Ci) and of the action carried out on the first digital document (Di) from a user interface (INT1); ■ Acquisition of a first corrective action (ACi) from a user relating to the modification of the movement of the first digital document (Di) to a second directory (REP2); ■ Generation of a first annotation (ANNi) comprising on the one hand the modified value (Pi ') of the prediction (Pi) corrected from the first corrective action (ACi) and on the other hand a set of values ​​of interest extracted or generated in particular from the first digital document (Di); ■ Modification of the first training domain (DOM1) by adding the first annotation (ANNi) after collecting at least one annotation (ANNi); ■ Generation of a retraining of the first machine learning model (MLi) from the modified training domain.

2. Method according to claim 1 characterized in that the first corrective action (ACi) relating to the modification of the movement of the first digital document (Di) is carried out by moving the first digital document (Di) from the first directory (REPi) to a second directory (REP2) by a user action from a document explorer, an algorithm for detecting a change in memory resource allocated to a digital document being implemented.

3. Method according to claim 1 characterized in that the first corrective action (AC1) relating to the modification of the movement of the first digital document (D1) is carried out by a modification of the class of the first digital document (D1) by a user action from a user interface presenting a set of data characteristic of the first digital document (D1).

4. Method according to any one of claims 1 to 3, characterized in that it comprises: ■ Receipt of a second digital document (D2); ■ Execution of a second learning function (FA2) generated from a second machine learning model (ML2) trained from a second training domain (DOM2) making it possible to generate a prediction (P2) of a classification of the second digital document (D2) within a second class (C2) and to carry out a second automatic action (A2) on the second digital document (D2), said second action (A2) comprising at least one automatic renaming of said second digital document (D2), ■ Generation of a representation of this classification (C2) and of the action carried out on the first digital document (D2) from a user interface (INT1); ■ Acquisition of a second corrective action (AC2) from a user relating to the modification of the name of the second digital document (D2); ■ Generation of a second annotation (ANN2) comprising on the one hand the modified value (P2') of the prediction (P2) relating to the second corrective action (AC2) and on the other hand a set of values ​​of a second data sequence (S2) extracted or generated in particular from the second digital document (D2); ■ Modification of the second training domain (DOM2) by adding the second annotation (ANN2); ■ Generation of retraining of the second machine learning model (ML2).

5. Method according to claim 4 characterized in that the second corrective action (AC2) relating to the modification of the name of the second digital document (D2) is carried out by a modification of the name of the second digital document (D2) directly on the file corresponding to the second digital document (D2).

6. Method according to claim 4 characterized in that the second corrective action (AC2) relating to the modification of the name of the second digital document (D2) is carried out by a modification of the class of the second digital document (D2) by a user action from a user interface presenting a set of data characteristic of the second digital document (D2).

7. Method according to any one of claims 1 to 6, characterized in that it comprises: ■ Receipt of a digital document (D1, D2, D3); ■ Execution of a third learning function (FA3) generated from a third machine learning model (ML3) trained from a third training domain (DOM3) to generate a prediction (P3) relating to the detection of a class of a characteristic date (DAT1) present in the digital document (Di, D2, D3) and extract said characteristic date (DAT1) from the first document (D1); ■ Generation of a representation of the classification of at least the characteristic date (DATi) extracted from a user interface; ■ Acquisition of a third corrective action (AC3) from a user relating to the modification of the class (C3) of the characteristic date (DAT1) of the first digital document (D1); ■ Generation of a third annotation (ANN3) comprising on the one hand the modified value (P3') of the prediction (P3) relating to the first corrective modification (AC1) and on the other hand a set of values ​​of interest extracted or generated in particular from the digital document (D1, D2, D3); ■ Modification of the third training domain (DOM3) by adding the third annotation (ANN3); ■ Generation of a retraining of the third machine learning model (ML3).

8. Method according to claim 7, characterized in that the third learning function (FA3) is executed prior to the first learning function, the corrected prediction (P2') being a piece of data from the set of values ​​of interest of an annotation, said modification of the corrected prediction (P2') resulting in the modification of the first training domain (DOM1) and / or the modification of the second training domain (DOM2).

9. Method according to any one of claims 1 to 8, characterized in that it comprises the reception of a dependency graph (GRAPHD), said dependencies being defined between at least the first learning function and a second learning function, said dependency graph (GRAPHD) comprising a description of the inputs of at least one training domain (D0M1, D0M2), said inputs comprising at least one value corresponding to an output of at least one learning function (FA1, FA2).

10. Method according to claim 9, characterized in that it comprises: ■ Acquisition of a dependency graph (GRAPHD) between a plurality of learning functions chosen from a set of learning functions comprising the first learning function (FAi), the second learning function (FA2) and the third learning function (FA3); ■ Addition to at least one set of values ​​of interest produced during the creation of an annotation (ANN1, ANN2, ANN3) of a modified prediction of a given learning function (FA1, FA2, FA3) in order to: ■ Modification of a training domain (DOM1, DOM2, DOM3) of a learning function other than the given learning function; ■ Generation of retraining of the machine learning model (MU, ML2, ML3).

11. Method according to any one of claims 1 to 10, characterized in that following an automatic action carried out by a learning function (FA1, FA2, FA3), a first notification is sent via a data network, said first notification comprising access to a user interface making it possible to modify the annotations associated with this action.

12. Method according to any one of claims 1 to 11, characterized in that the first machine learning model is generated from the definition of a model of a first digital document (MDI) comprising the identification of a plurality of characteristic data and areas of interest of said model (MDI) of the first document and from a user interface.

13. Method according to claim 12, characterized in that following a corrective action generated by a user, the model (MDI) of the first digital document is used to extract context data from the modified prediction to enrich the annotation which is created.

14. Method according to any one of claims 12 to 13, characterized in that the definition of a first digital document model (MDI) comprises the generation of a form comprising a set of choices defining annotations of said model, said annotations making it possible to update the first training model.

15. Method according to any one of claims 1 to 14, characterized in that a plurality of first documents are received from the same class, and that a plurality of machine learning models are applied, the method comprising the generation of a plurality of predictions, the method comprising the selection of a prediction, said selection of said prediction making it possible to update the first training domain (DOMi).

16. System comprising an electronic terminal of a user (Ti) comprising at least one computer, a memory, a display and a communication interface for transmitting messages over a data network to at least one first server (SERVi) comprising hardware resources for executing the first learning function (FAi), the second learning function (FA2), the third learning function (FA3) and a memory for recording the learned models (ML1, ML2, ML3) in order to execute the steps of the method of any one of claims 1 to 15, the terminal comprising a user interface for acquiring an annotation comprising a modification of a prediction (Pi, P2, P3), said modification resulting in the retraining of at least one machine learning model.

Images