Annotating textual data

Abstract

The invention relates to a method for iteratively annotating textual data implemented in a computer device (CD), said method comprising the following at a current iteration (i): - receiving (S2) an unannotated text (Ti) as input from at least three different text annotation models, the at least three models (MA1, MA2, MA3) having been trained from a first corpus of training data comprising annotated textual data, - generating (S4) three annotated texts respectively (TA1i, TA2i, TA3i), - selecting (S5) one of the three annotated texts, the selected annotated text (TAsi) being the one that contains the fewest semantic and / or syntactic differences, compared with the other two annotated texts, - constructing (S7) a new corpus of annotated textual data (N_CO) by adding to it the selected annotated text (TAsi) and the unannotated text (Ti),said new corpus constituting a second corpus of training data for the text annotation model. Figure 3,

Description

Title of the invention: Annotation of textual data Scope of the invention

[0001] The field of the invention is that of semantic analysis. More specifically, the present invention relates to a method for annotating textual data, the annotated textual data obtained being intended for use in constructing an annotated textual data corpus. Such a textual data corpus is intended for training and improving text annotation models, which may be, for example, statistical, machine learning, or other models, as well as for validating these models. The present invention also relates to a corresponding computer system (computer, server, platform, etc.), computer program, and storage medium. Previous art

[0002] In the current state of knowledge, text annotation models of the type mentioned above are trained and validated with manually constructed corpora of annotated text data. The drawback of such corpora is that they are generally too small, which significantly reduces the performance of model training. As a result, the models trained in this way lack robustness. There are some larger corpora of manually annotated text data, the size of which varies depending on the type of annotations implemented, such as syntactic, semantic, named entity, etc. However, such corpora may be subject to a fee, making the training of text annotation models expensive.Finally, because the textual data in these corpora has been manually annotated, such corpora may contain errors and lack scalability, which could also negatively impact the performance and smooth running of the training and validation of text annotation models. Object and summary of the invention

[0003] One of the aims of the invention is to remedy at least one of the drawbacks of the aforementioned prior art by proposing a new text annotation technique that is more efficient than current annotation techniques, by making it possible to automatically obtain a corpus of annotated textual data, such a corpus being able to reach a very large size and comprising high-quality annotated texts, particularly from a semantic and / or syntactic point of view.

[0004] To this end, an object of the present invention relates to a method for iteratively annotating textual data, such a method comprising the following at a current iteration:

[0005] - receive an unannotated text as input from at least three different models text annotation, with at least three models having been trained from an initial training data corpus comprising annotated textual data,

[0006] - generate three annotated texts respectively,

[0007] - select one of the three annotated texts, the selected annotated text being the one which contains the fewest semantic and / or syntactic differences, compared to the other two annotated texts,

[0008] - construct a new corpus of annotated textual data by adding the text annotated selected and unannotated text, the new corpus constituting a second corpus of text annotation model training data.

[0009] Such a process leads to an improvement in the quality of annotations through the use of several annotation models. The process allows for the comparison of at least three annotated texts, each produced by at least three text annotation models, and the selection of the most accurate annotated text, thereby reducing semantic and / or syntactic errors generated during the annotation stage. Thanks to this process, the second corpus is automatically generated and built up through the various iterations of the aforementioned process steps. As a result, at the end of these iterations, the second corpus proves to be of significantly higher quality than the first corpus, which was used to train the three annotation models. Thus, it is ensured that this second corpus will be a high-quality corpus, particularly relevant for training and improving different text annotation models, promoting a continuous learning cycle.These training programs and improvements will have the advantage of being implemented at a reduced cost.

[0010] Furthermore, the fact that the construction of the second corpus is automated reduces the need for human intervention, thereby accelerating the process of building the second corpus and lowering costs. Thanks to the invention, it is thus possible to build large annotated text data corpora in a short time.

[0011] The use of different annotation models also makes it possible to adapt the process to various types of texts and contexts, thus increasing its flexibility, to the benefit of mitigating the potential biases of a single model, thus offering a more balanced representation of the data.

[0012] These advantages thus contribute to greater efficiency and superior quality in the construction of textual data corpora.

[0013] According to a particular embodiment, the selected annotated text is the one that obtains a maximum semantic and / or syntactic similarity score obtained at the end of a comparison of the semantic and / or syntactic similarity between the at least three annotated texts.

[0014] Such a semantic / syntactic similarity score can be between 0 and 1, where 0 indicates that an annotated text has a maximum of possible differences with another annotated text and 1 indicates that an annotated text is identical to another annotated text.

[0015] Alternatively, such a semantic / syntactic similarity score can be between 0% and 100%, where 0% indicates that an annotated text has a maximum of possible differences with another annotated text and 100% indicates that an annotated text is identical to another annotated text.

[0016] By choosing the annotated text with the maximum semantic / syntactic similarity score, the process ensures that the annotation is the most relevant in relation to the unannotated text, thus improving the quality of the annotated textual data of the second corpus.

[0017] The use of semantic / syntactic similarity makes it possible to capture nuances and contexts, which is essential for applications such as natural language processing, the use of chatbots, plagiarism detection, etc. The use of a semantic / syntactic similarity score provides a robust metric for efficiently evaluating and comparing the performance of different annotation models.

[0018] By relying on a quantifiable criterion, the process thus minimizes the risks of annotation errors that could occur with more subjective methods.

[0019] Selection based on semantic and / or syntactic similarity makes it possible to optimize the use of annotation models, focusing on those that produce the most reliable results.

[0020] These advantages contribute to more precise and efficient annotation, thereby reinforcing the value of the second annotated textual data corpus.

[0021] According to another particular embodiment, a filtering criterion is applied to the selection of the annotated text, according to which the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if the maximum semantic and / or syntactic similarity score is less than a threshold value.

[0022] Such filtering makes it possible to avoid the inclusion, in the second corpus, of annotated texts which could be irrelevant or erroneous, thus reducing the risks of semantic and / or syntactic errors in the second corpus.

[0023] Furthermore, by not adding low-quality annotated texts to the second corpus, for a common iteration of the process, computer resources and time are saved, making the annotation process more efficient.

[0024] The reliability and quality of the second corpus are thus strengthened, to the benefit of a more precise selection of annotated texts.

[0025] According to another particular embodiment, a filtering criterion is applied to the selection of the annotated text, according to which the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if the difference between the maximum semantic and / or syntactic similarity score and the minimum semantic and / or syntactic similarity score obtained at the end of the comparison of the semantic / syntactic similarity between the at least three annotated texts is greater than a threshold value.

[0026] By excluding annotated texts when the difference between maximum semantic and / or syntactic similarity and minimum semantic and / or syntactic similarity is greater than a threshold, such an embodiment makes it possible to favor the selection of annotated texts that are more consistent with each other, which improves the quality of the second corpus.

[0027] Thanks to such a filtering criterion, undesirable variations between the at least three annotated texts are not taken into account, which makes it possible to avoid the inclusion, in the second corpus, of annotated texts presenting too many variations, which could indicate divergent interpretations or annotation errors.

[0028] Applying such a criterion also makes it possible to optimize computer resources by filtering out annotated texts that do not meet this criterion.

[0029] All these advantages enhance the quality and relevance of the second corpus of textual data, while optimizing the process of selecting annotated texts.

[0030] According to another particular embodiment, the selected annotated text is the one that obtains the maximum average between two semantic and / or syntactic similarity values ​​generated as a result of a pairwise comparison between the at least three annotated texts.

[0031] Thanks to the consideration of the average of the semantic and / or syntactic similarities between several annotated texts, such an embodiment makes it possible to obtain a more balanced evaluation and less sensitive to anomalies or extreme values.

[0032] Since the two-by-two comparison between the annotated texts is exhaustive, it allows for capturing finer relationships and better understanding the nuances between the different annotated texts.

[0033] Moreover, by using several semantic and / or syntactic similarity values, such an embodiment minimizes the impact of an annotation model that could produce biased or unrepresentative results.

[0034] The fact that the selection of the annotated text is based on an average favors the choice of an annotated text which is overall the most coherent and relevant, thus increasing the quality of the second corpus which is intended to contain this selected annotated text.

[0035] These advantages thus contribute to a more precise and reliable selection of the annotated text, thereby strengthening the quality of the second corpus of textual data.

[0036] According to another particular embodiment, a filtering criterion is applied to the selection of the annotated text, according to which the annotated text which has been selected and the unannotated text are not used to construct the second corpus of textual data if the maximum average between two values ​​of semantic and / or syntactic similarity is less than a threshold value.

[0037] Such a criterion makes it possible to favor the selection of annotated texts which present semantic and / or syntactic coherence, which aims to maintain the integrity of the second corpus of textual data.

[0038] Such an embodiment thus makes it possible to minimize the risk of including annotations that are not very relevant or are erroneous from a semantic and / or syntactic point of view.

[0039] These advantages thus contribute to a more precise and reliable selection of the annotated text, thereby strengthening the quality of the second corpus of textual data.

[0040] According to another particular embodiment, a filtering criterion is applied to the selection of the annotated text, according to which the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if, at the end of the pairwise comparison between the at least three annotated texts, the minimum average obtained for an annotated text is less than a threshold value.

[0041] Such a criterion makes it possible to identify annotated texts which have sufficient coherence and relevance, thus reducing the impact of extreme variations.

[0042] More specifically, this criterion favors the selection of annotated texts which are not only the best on average, but which also meet a minimum standard of quality, thus ensuring better semantic and / or syntactic consistency in the second corpus.

[0043] Such an embodiment thus makes it possible to minimize the risk of including, in the second corpus, annotations that are not very relevant or are erroneous from a semantic and / or syntactic point of view.

[0044] These advantages thus contribute to a more precise and reliable selection of the annotated text, thereby strengthening the quality of the second corpus of textual data.

[0045] According to another particular embodiment, a filtering criterion is applied to the selection of the annotated text, according to which the selected annotated text and the unannotated text are not used to generate the second corpus of textual data if is greater than a threshold value, the difference between the semantic similarity values ​​and / or maximum and minimum syntactic values ​​obtained from the pairwise comparison between the at least three annotated texts.

[0046] Taking into account the difference between the maximum and minimum semantic and / or syntactic similarity values ​​allows for a more balanced evaluation of the at least three annotation models, reducing the impact of outliers.

[0047] By filtering out annotated texts that do not meet this criterion, the process saves resources by focusing on more meaningful annotations.

[0048] Such a filtering criterion provides a solid basis for comparing the performance of different annotation models, making it possible to identify those that produce more homogeneous and reliable results.

[0049] These advantages thus contribute to a more precise and efficient selection of annotated texts, thereby strengthening the quality of the second corpus of textual data.

[0050] According to another particular embodiment, the unannotated text provided as input to the at least three different text annotation models is taken from at least two different online content resources.

[0051] Such an embodiment makes it possible to obtain, after several iterations of the annotation process, a second corpus of annotated text data containing a much larger number of annotated text data items than that of state-of-the-art annotated text data corpora, since the activation of at least three annotation models is implemented using content from at least two online resources, whose data is very numerous and varied. In the state of the art, the annotated text data corpora used for training annotation models are much more limited because they are based on manually annotated texts.

[0052] Furthermore, the diversity of content sources helps minimize potential biases associated with a single type of content source, which can lead to more balanced and representative annotations. Texts from different resources can contain varied styles and contexts, which helps models to better adapt to diverse situations and produce higher-quality annotations.

[0053] These advantages contribute to the creation of a second, richer and more relevant corpus of textual data, thereby strengthening the effectiveness of the annotation models that will be trained by this second corpus.

[0054] The various modes or embodiments mentioned above can be added independently or in combination with each other to the iterative annotation process of textual data as defined above.

[0055] The invention also relates to an iterative text data annotation device, characterized in that it is configured to implement the following:

[0056] - receive an unannotated text as input from at least three different models text annotation, with at least three models having been trained from an initial training data corpus comprising annotated textual data,

[0057] - generate three annotated texts respectively,

[0058] - select one of the three annotated texts, the selected annotated text being the one which contains the fewest semantic and / or syntactic differences, compared to the other two annotated texts,

[0059] - construct a new corpus of annotated textual data by adding the text annotated selected and unannotated text, the new corpus constituting a second corpus of text annotation model training data.

[0060] Such a device is notably configured to implement the aforementioned annotation process, according to one or the other of its embodiments.

[0061] The invention also relates to a computer program comprising instructions for implementing the annotation method according to the invention, according to any one of the particular embodiments described above, when said program is executed by a processor.

[0062] Such instructions can be stored permanently in a non-transient memory medium of the computer device implementing the annotation method according to the invention.

[0063] This program may use any programming language, and be in the form of source code, object code, or intermediate code between source code and object code, such as in a partially compiled form, or in any other desirable form.

[0064] The invention also relates to a recording medium or information medium readable by a computer, and comprising instructions for a computer program as mentioned above.

[0065] The recording medium can be any entity or device capable of storing the program. For example, the medium can include a storage means, such as a ROM, for example a CD-ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a mobile medium, a hard drive or an SSD.

[0066] On the other hand, the recording medium can be a transmissible medium such as an electrical or optical signal, which can be transmitted via an electrical or optical cable, by radio or by other means, so that the computer program The program it contains is executable remotely. In particular, the program according to the invention can be downloaded onto a network, for example, an Internet-type network.

[0067] Alternatively, the recording medium may be an integrated circuit in which the program is incorporated, the circuit being adapted to execute or to be used in the execution of the aforementioned annotation process.

[0068] According to one embodiment, the present technique is implemented using software and / or hardware components. In this context, the term "device" or "module" may refer in this document to a software component, a hardware component, or a set of hardware and software components. Brief description of the drawings

[0069] Other features and advantages will become apparent upon reading particular embodiments of the invention, given by way of illustrative and non-limiting examples, and the accompanying drawings, among which:

[0070] [Fig-1] Fig. 1 represents an architecture in which the a method for iteratively annotating textual data, according to a particular embodiment of the invention,

[0071] [Fig.2] Figure [Fig.2] represents a device for annotating textual data according to a particular embodiment of the invention, as implemented in the architecture of [Fig. 1],

[0072] [Fig.3] Fig.3 represents the main actions implemented in the process iterative annotation of textual data, according to a particular embodiment of the invention, as implemented in the architecture of [Fig.1],

[0073] [Fig. 4A] [Fig. 4A] represents one embodiment of the annotated text selection step implemented in the annotation process illustrated in [Fig. 3],

[0074] [Fig. 4B] [Fig. 4B] represents another embodiment of the annotated text selection step implemented in the annotation process illustrated in [Fig. 3],

[0075] [Fig. 5] [Fig. 5] represents, for different trained annotation models either using a corpus constructed with annotated textual data obtained in accordance with the annotation method of the invention, or with a corpus of annotated texts of the prior art, the evolutions of their respective similarity scores, as a function of the number of annotated sentences contained in these corpora.

[0076] Detailed description of an embodiment of the invention

[0077] Fig. 1 represents an architecture in which an iterative annotation process for textual data is implemented, according to one embodiment of the invention.

[0078] Such an architecture includes a DA device for annotating textual data which includes:

[0079] - at least three MAI, MA2, MA3 models for textual data annotation of different architecture, each having been trained from the same CO corpus of training data comprising annotated textual data, the models MAI, MA2, MA3 being configured to, at a current iteration i, i being an integer greater than 1, generate respectively three annotated texts TA1;, TA2;, TA3;, from an unannotated text T;,

[0080] - a selection M_SEL module that is configured to select, at iteration current i, one of the three annotated texts TA1;, TA2;, TA3;, the selected annotated text being designated by TAs; on [Fig.1] and being the one which contains on average fewer semantic and / or syntactic differences, compared with the other two annotated texts,

[0081] - an M_CNT module for constructing annotated textual data corpora which is configured to add, at the current iteration i, the selected annotated text TAs; and the unannotated text T; to a new corpus N_CO of annotated text data which may already include at least one annotated text previously selected at a previous iteration.

[0082] A semantic difference includes, for example, different concepts or different relation labels, etc.

[0083] A syntactic difference is for example an article not attached to a word (e.g. the article "the" is not attached to the noun "mouse" in the sentence "the mouse"), a subject which is annotated as a direct object complement instead of a subject, etc.

[0084] The aforementioned architecture also includes a user interface (UI) configured to be activated by a user (UT) to communicate with the annotation device (AD). In particular, the UI is configured to provide, at the current iteration i, the unannotated text T; as input to each of the annotation models MAI, MA2, and MA3. The unannotated text T; may comprise one or more sentences, an extract from a text, a paragraph extracted from an article available on an online content resource, etc. The text T; may be entered by the user (UT) using the UI, extracted from an online content resource, or from a text database stored on a terminal (computer, tablet, smartphone, etc.) connected to or including the UI.

[0085] Compared to the text T;, the annotated text TA; can be, depending on the type of annotation models used, a semantic graph, a syntax tree (e.g., dependency tree), a plurality of named entities, etc.

[0086] The DA annotation device is a computer device which may include, for example, a computer, a server, a platform, etc.

[0087] In [Fig. 1], the at least three annotation models MAI to MA3 are integrated into the annotation device DA. Of course, the at least three annotation models MAI to MA3 can be separated from the annotation device DA, a dedicated communication interface being provided to allow the annotation device DA to communicate with each of the at least three annotation models MAI to MA3.

[0088] The UI interface may include, for example, a text-based graphical interface or a sound sensor coupled with a speech recognition interface. Such an interface may belong to the DA annotation device or be separate from it.

[0089] The architecture of [Fig. 1] is of course not limited to the three annotation models MAI to MA3 and may include more than three annotation models with different architectures or functions, provided that these different annotation models are trained on the same CO corpus of training data mentioned above. By way of non-limiting examples, the CO corpus may be the AMR 3.0 corpus, LDC2020T02, proposed by the LDC consortium (Linguistic Data Consortium), as described in the document: Kevin Knight, Bianca Badarau, Laura Baranescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O'Gorman, and Nathan Schneider. 2020. Abstract meaning representation (AMR) annotation release 3.0, https: / / catalog.ldc.upenn.edu / LDC2020T02. This is the largest corpus of annotated text data available for training text data annotation models.Such a corpus comprises approximately 60,000 sentences and their corresponding 60,000 semantic graphs. Alternatively, the following may be used:

[0090] - the QALD-9 corpus as described in the document: Ricardo Usbeck, Ria Hari Gusmita, Muhamad Saleem, and Axel-Cyrille Ngonga Ngomo. 2018. 9th challenge

[0091] on question answering over linked data (QALD-9). In Joint proceedings of the 4th Workshop on Semantic Deep Learning (SemDeep-4) and NLIWoD4 and the document: Young-Suk Lee. 2022. Human annotations of AMR graphs for QALD-9-AMR treebank, https: / / github.com / IBM / AMR-annotations,

[0092] - the Bio AMR corpus as mentioned at the following address: https: / / github.com / flipz357 / AMR-World / blob / main / data / reference_amrs / amr-release-bio-v3.0.txt.

[0093] In one embodiment, the annotation models MAI to MA3 can be of the AMR (Abstract Meaning Representation) type. These models are configured, as is known per se, to transform at least one sentence into a semantic graph called an AMR graph. By way of non-limiting examples:

[0094] - The MAI model can be the AMRlib model accessible from a library Python available online at the following address: https: / / github.com / bjascob / amrlib,

[0095] - The MA2 model can be the X-AMR model as described in the document: “Yitao Cai, Zhe Lin, and Xiaojun Wan. 2021. Making Better Use of Bilingual Information for Cross-Lingual AMR Parsing. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, page 1537-1547, Online. Association for Computational Linguistics » and whose software implementation is available at the following address: https: / / github.com / jcyk / XAMR,

[0096] - The MA3 model can be the AMR-transition-parser model available online at the following address: https: / / github.com / IBM / transition-amr-parser.

[0097] A fourth MA4 model could be a differently parameterized model with AMRlib.

[0098] A fifth MA5 model could be a model parameterized differently from the AMR-transition parser model.

[0099] Alternatively, the MAI to MA3 annotation models can be of the UMR (Uniform Meaning Representation) type. In a manner known per se, these models are configured to transform at least one sentence into a semantic graph, the semantic graph being composed of a graph at the sentence level and a graph at the document level in which the sentence is located.

[0100] In another embodiment, the MAI to MA3 or higher models can be configured to transform at least one sentence into a syntax tree (e.g., a dependency tree). By way of non-limiting examples:

[0101] - the MAI model may be the UDParse model as described at the following address: https: / / github.com / Orange-OpenSource / UDParse,

[0102] - the MA2 model can be the UDAPI model as described at the following address: https: / / udapi.readthedocs.io / en / latest / ,

[0103] - the MA3 model can be the UDPipe model as described at the following address: http: / / ufal.mff.cuni.cz / udpipe.

[0104] In yet another embodiment, the MAI to MA3 or higher models can be configured to annotate at least one sentence with named entities, that is, to associate, for example, with "people" entities present in the sentence the proper names of these people or with "places" entities the names of these places, etc. By way of non-limiting examples:

[0105] - the MAI model can be the Stanford model as described at the following address: https: / / nlp.stanford.edu / ner / index.shtml,

[0106] - the MA2 model may be the spotlight model as described at the following address: http s : / / ww w. dbpedia-spotlight. org / ,

[0107] - the MA3 model may be one of the models offered at the following address:

[0108] https: / / www.clarin.eu / resource-families / tools-named-entity-recognition.

[0109] We will now describe, with reference to [Fig.2], the simplified structure of the DA annotation device.

[0110] According to the invention, the DA device comprises:

[0111] - a COM communication module configured to receive generated requests using the UI interface, said queries comprising, at each iteration, an unannotated text, here the unannotated text T; at the current iteration i,

[0112] - at least three text data annotation models MAI, MA2, MA3 of different architecture, configured to each receive as input the unannotated text T; at the current iteration i, and respectively deliver three annotated texts TA1, TA2, TA3, TA3i,

[0113] - an M_SEL module for selecting a text annotated TAs; among the three texts annotated TA1;, TA2;, TA3;.

[0114] - an M_CNT module for constructing annotated textual data corpora which is configured to add, at the current iteration i, the selected annotated text TAs; and the unannotated text T; to a new corpus N_CO of annotated text data which may already include at least one annotated text previously selected at a previous iteration, for example the annotated text TAsm selected at the previous iteration i-1.

[0115] As explained above, the at least three text data annotation models MAI, MA2, MA3 may not be included in the DA annotation device. In this implementation, the at least three text data annotation models MAI, MA2, MA3 are interfaced between the COM communication module and the M_SEL selection module.

[0116] The M_SEL selection module itself comprises:

[0117] - an M_SIM comparison submodule configured to perform a comparison from a semantic and / or syntactic point of view between the three annotated texts TA1, TA2i, TA3i, at the end of which three semantic and / or syntactic similarity scores are issued: SC1, SC2, SC3.

[0118] - a calculation submodule M_MAX configured to calculate a similarity value maximum VSmax; from the similarity scores SCI, SC2, SC3; in order to select the annotated text TAs; corresponding to this maximum similarity value VSmax;.

[0119] These three similarity scores are defined as follows:

[0120] - SCh measures the semantic and / or syntactic similarity between the annotated TA I texts, TA2„

[0121] - SC2i measures the semantic and / or syntactic similarity between the annotated TA1 texts;, TA3i,

[0122] - SC3; measures the semantic and / or syntactic similarity between the annotated TA2 texts;, TA3i.

[0123] Optionally, according to the invention, the DA annotation device comprises:

[0124] - an M_FLT filtering module that is configured to apply a filtering criterion to the selection of the annotated text TAs; such an application being based on the comparison of the maximum similarity value VSmax; at different thresholds TH1, TH2, TH3, ... according to the filtering criterion applied,

[0125] - a storage module M_ST in which different thresholds TH1 are stored, TH2, TH3, .... The M_FLT and M_ST modules being optional, they are represented by dotted lines on [Fig.2].

[0126] Depending on the result of comparing the maximum similarity value VSmax to a given threshold among TH1, TH2, TH3..., the selected annotated text TAs; is either added to the new corpus of annotated textual data N_CO in association with the unannotated text TAs;, or is not added.

[0127] At initialization, the code instructions of the computer program PG are, for example, loaded into RAM (not shown) before being executed by the PROC processor. The PROC processor of the UTR processing unit implements, in particular, the following actions at a current iteration i, within the framework of the iterative annotation process of textual data that will be described below, according to the instructions of the computer program PG:

[0128] - receive a request including unannotated text 1),

[0129] - provide the unannotated text T; as input to at least three annotation models of textual data MAI, MA2, MA3,

[0130] - select an annotated text TAs; from among the three annotated texts TA1, TA2, TA3i,

[0131] - possibly filter the annotated text TAs;,

[0132] - add the annotated text TAs; and the unannotated text T; to the new data corpus annotated textual N_CO, in the case where no filtering is applied or depending on the result of the filtering if the latter is applied.

[0133] We now describe, in relation to [Fig.3], together with figures 1 and 2, the process of iterative annotation of textual data, according to a particular embodiment of the invention.

[0134] In the example shown, the iterative annotation process is implemented using four AMR-type annotation models MAI, MA2, MA3, MA4, where, by way of non-limiting examples:

[0135] - the MAI model is an AMRlib model,

[0136] - the MA2 model is an X-AMR model,

[0137] - the MA3 model is an AMR-transition-parser model comprising the model of natural language AMR3-jointontowiki-seed44 model,

[0138] - the MA4 model is an AMR-transition-parser model comprising the model natural language AMR3-jointontowiki-seed44 available at the following address: https: / / arxiv.org / abs / 2112.07790 or the AMR3-structbart-Lmodel available at the following address: https: / / arxiv.org / abs / 2205.01464.

[0139] These four models were previously trained with annotated textual data from a CO corpus which is for example the aforementioned AMR 3.0 corpus.

[0140] At a current iteration i of annotation, the iterative annotation process proceeds as follows:

[0141] During a step SI, the user UT sends a request REQ to the annotation device DA via the IU interface. This request REQ comprises an unannotated text Tj. In various embodiments, the unannotated text Tj can be extracted from at least one online content resource, a text database, entered manually or orally by the user UT, etc. This request is received in S2 by the annotation device DA via its COM module. In the example shown, the unannotated text Tj is a sentence extracted from content available on an online content resource, such as, for example, Wikipedia®, Wikinews®, WikiVoyage®, WikiBooks® (all from MediaWiki), online newspapers, blogs, etc. Preferably, the annotation method according to the invention uses at least two online content resources, from one of which an unannotated text Tj can be extracted. from one iteration to the next.This implementation allows for the generation of a new N_CO corpus containing various annotated textual data. It should be noted that not all sentences in the available content are necessarily included. A filtering step can be implemented to remove sentences such as, for example:

[0142] - those that contain too many numbers, or codes, such as ISBN (International Standard Book Number), EAN (European Article Numbering), etc.

[0143] - sentences that contain parentheses, brackets, braces, etc.,

[0144] - the headers,

[0145] - bubble lists,

[0146] - sentences without capital letters at the beginning or without punctuation at the end,

[0147] - etc.

[0148] During a step S3, the unannotated text T; is provided as input to each of the four annotation models MAI, MA2, MA3, MA4.

[0149] During a step S4, from this unannotated text T;, the four annotation models MAI, MA2, MA3, MA4 respectively generate four annotated texts TAlb TA2i, TA3i, TA4;.

[0150] During an S5 step, a selection of one of the annotated texts TA1;, TA2;, TA3i, TA4; is implemented using the selection module M_SEL.

[0151] Selection S4 includes:

[0152] - a substep S51 of semantic and / or syntactic comparison between the four annotated texts TA1, TA2, TA3, TA4, at the end of which six similarity scores are calculated: SCL, SC2, SC3, SC4, SC5b, SC6.

[0153] - a substep S52 for calculating a maximum similarity value VSmax; from similarity scores SCL, SC2;, SC3;, SC4;, SC5;, SC6;.

[0154] These six similarity scores are defined as follows:

[0155] - SCI; measures the similarity between the annotated texts TA1;, TA2;,

[0156] - SC2i measures the similarity between the annotated texts TA1;, TA3;,

[0157] - SC3i measures the similarity between the annotated texts TAh, TA4;,

[0158] - SC4i measures the similarity between the annotated texts TA2;, TA3;,

[0159] - SC5i measures the similarity between the annotated texts TA2;, TA4;,

[0160] - SC6; measures the similarity between the annotated texts TA3;, TA4;.

[0161] At the end of step S5, the selected annotated text TAs; is delivered. This is the one among the annotated texts TAC, TA2;, TA3i, TA4; which obtained the maximum similarity value VSmaXi.

[0162] During an optional step S6, which for this reason is represented by dotted lines on [Fig.3], filtering is applied by the filtering module M_FLT to the selected text TAs; according to a filtering criterion CF, examples of which will be described later in the description.

[0163] Following such filtering, it is then decided whether to take into account (“O” on [Fig.3]) or not (“N” on [Fig.3]) the selected annotated text TAs; in the construction of the new corpus of annotated textual data N_CO.

[0164] During a step S7, the selected annotated text TAs; obtained at the end of step S5 or retained after the filtering step S6, if the latter is implemented, together with the unannotated text T;, are added to the new corpus of annotated textual data N_CO.

[0165] The text data annotation process just described is then iterated N times (N being an integer greater than or equal to 1, the number N of iterations being for example predefined or a function for example of the number of annotated sentences desired for the new corpus of annotated text data N_CO.

[0166] Thanks to such a method, it is possible to automatically construct a corpus of high-quality annotated text data that is significantly larger than a corpus of annotated text data in the prior art. Furthermore, the corpus generated according to the invention can be adapted to a user's needs to train a specific text data annotation model of their choice. Indeed, thanks to the invention, the user can choose which unannotated texts they wish to provide. input of at least three annotation models, the type of annotation models to use, the number of iterations, the size of the new corpus of annotated textual data to be built, etc.

[0167] We will now describe, with reference to [Fig.4A], an embodiment of the selection step S5 of the annotated text TAs;.

[0168] In this embodiment, the similarity scores SCL, SC2, SC3i, SC4, SC5i, SC6, which are calculated in S51, are, for example, semantic similarity evaluation scores called "smatch scores," as described in particular in the document: Shu Cai and Kevin Knight. 2013. Smatch: an Evaluation Metric for Semantic Feature Structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748-752, Sofia, Bulgaria. Association for Computational Linguistics. Of course, other metrics can be used, such as, for example, the metric described in the document: Shira Wein and Nathan Schneider. 2022. Accounting for Language Effect in the Evaluation of Cross-lingual AMR Parsers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3824-3834, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.As alternative, non-limiting examples, there is also the Smatch++ score available at the following address: https: / / github.com / flipz357 / smatchpp, the XSmatch score available at the following address: https: / / github.com / shirawein / Crossling-AMR-Eval, etc.

[0169] Other types of similarity scores can of course be used in addition or as an alternative, such as, for example, a syntactic similarity score described in particular in the document Joakim Nivre and Chiao-Ting Fang. 2017. Universal Dependency Evaluation. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 86-95, Gothenburg, Sweden. Association for Computational Linguistics. A well-known example of a syntactic similarity score is the LAS (Labeled-Attachment-Score).

[0170] At the end of sub-step S51, six respective values ​​Vh, V2i, V3i, V4j, V5i, V6i of these similarity scores are obtained respectively.

[0171] By way of non-limiting examples, V1;=90%, V2;=92%, V3;=94%, V4;=80%, V5i=82%, V6i=84%.

[0172] Substep S52 calculates the maximum value Vsmax; of these six score values, i.e. Vsmaxi=V3i=94% in the example shown.

[0173] It is therefore the annotated text TA1; which is selected as the selected annotated text TAs;.

[0174] In the case where the S6 filtering step is implemented, the CF filtering criterion may include the following:

[0175] - comparison of the value Vsmax; =V3; of the similarity score SC3; to a value predefined threshold TH1, stored in the M_ST memory of the DA annotation device, the threshold value TH1 can be equal to, for example, 90%,

[0176] - if Vsmaxi <TH1 ou en variante Vsmaxi <TH1, le texte annoté TA 1; et le texte non annotated T; are discarded,

[0177] - if Vsmaxi >TH1 or, alternatively, Vsmaxi >TH1, the annotated text TA1; and the unnoted text annotated T; are added to the new corpus N_CO in S7.

[0178] Alternatively, the CF filtering criterion may include the following:

[0179] - calculation of the difference A; between the maximum value Vsmaxi and the minimum value Vsmini of the six similarity scores V3i - V4; in the example shown, i.e. A;=94%-80%=14%,

[0180] - comparison of A; to a predefined threshold value TH2, stored in memory M_ST of the DA annotation device, the threshold value TH2 can be equal to, for example, 80%,

[0181] - if Ai <TH2 ou en variante A; <TH2, le texte annoté TA1; et le texte non annoté TiSont écartés,

[0182] - if A; >TH2 or, alternatively, A; >TH2, the annotated text TA1; and the unannotated text TiSont added to the new N_CO corpus in S7.

[0183] We will now describe, with reference to [Fig.4B], another embodiment of the selection step S5 of the annotated text TAs;.

[0184] In this embodiment, substep S51 of semantic and / or syntactic comparison between the four annotated texts TA1, TA2, TA3, TA4 comprises:

[0185] - a substep S510 which corresponds in every respect to substep S51 of the mode of realization of [Fig.4A], and which, therefore, will not be described again,

[0186] - a substep S511, in which is calculated, for each of the MAI models at MA4, respectively four means M1; at M4; of the three similarity score values ​​obtained, with respective values ​​noted VM1 to VM4;.

[0187] In the example shown:

[0188] - for the MAI model, VM1;= , 3

[0189] - for the MA2 model, VM2;= , 3

[0190] - for the MA3 model, VM^+WSj, 3

[0191] - for model MA4, VM4;= V3,+V5.+V6, 3

[0192] In the example shown, VM1;=92%, VM2i=84%, VM3;=85.3%, VM4;=86.7%.

[0193] Substep S52 calculates the maximum value Vsmax; of these four average values ​​VM1; to VM4;, i.e. Vsmax;=VM1;=92% in the example shown.

[0194] It is therefore the annotated text TA1; which is selected as the selected annotated text TAs;.

[0195] In the case where the S6 filtering step is implemented, the CF filtering criterion may include the following:

[0196] - comparison of the value Vsmax,=VM I to a predefined, stored threshold value TH3 in the M_ST memory of the DA annotation device, the threshold value TH3 can be equal to the aforementioned threshold value TH1, or different from the latter.

[0197] - if Vsmaxi <TH3 ou en variante Vsmax; <TH3, le texte annoté TA1; et le texte non annotated T; are discarded,

[0198] - if Vsmaxi >TH3 or, alternatively, Vsmax; >TH3, the annotated text TA1; and the non-text annotated T; are added to the new corpus N_CO in S7.

[0199] Alternatively, the CF filtering criterion may include the following:

[0200] - comparison to a predefined TH4 threshold value, stored in M_ST memory of the DA annotation device, of the minimum average value Vsmin; among the average values ​​VM1; to VM4; obtained at the end of substep S511, with Vsmini=VM2i in the example shown,

[0201] - if Vsmin; <TH4 ou en variante Vsmin; <TH4, le texte annoté TA1; et le texte non annotated T; are discarded,

[0202] - if Vsmin; >TH4 or alternatively Vsmin; >TH4, the annotated text TA1; and the non-text annotated T; are added to the new corpus N_CO in S7.

[0203] According to yet another variant, the CF filtering criterion can be the same as that implemented for the embodiment of [Fig.4A].

[0204] According to yet another variant, the CF filtering criterion may include the following:

[0205] - calculation of the difference A; between the maximum value Vsmax; and the minimum value Vsmin; of the four average values ​​VM1; to VM4ijVMli - VM2; in the example shown, i.e. A;=92%-84%=8%,

[0206] - comparison of A; to a predefined threshold value TH5, stored in memory M_ST of the DA annotation device, the threshold value TH5 can be equal to, for example, 90%,

[0207] - if A; <TH5 ou en variante A; <TH5, le texte annoté TA1; et le texte non annoté TiSont écartés,

[0208] - if A; >TH5 or, alternatively, A; >TH5, the annotated text TA1; and the unannotated text TiSont added to the new N_CO corpus in S7.

[0209] With reference now to [Fig.5], for different annotation models trained either with the new corpus N_CO or with a corpus of CO annotated texts from the prior art, the evolutions of their respective SC similarity scores expressed as a percentage are shown, as a function of the number of annotated sentences contained either in the new corpus N_CO if the training is implemented with the latter, or in the corpus of the prior art CO if the training is implemented with the latter.

[0210] In the example shown, the CO corpus of prior art is the AMR 3.0 corpus.

[0211] The star symbol “★” represents the SC similarity score obtained when the AMR 3.0 annotation model is trained using the prior art CO AMR 3.0 corpus, which comprises 60,000 annotated sentences, the maximum number of annotated sentences that this corpus can contain. The similarity score obtained is 82.5%.

[0212] Curve Cl represents the evolution of the SC similarity score when the AMR 3.0 annotation model is trained using the new N_CO corpus according to the invention, which can include a much larger number of annotated sentences, 200,000 annotated sentences in the example shown. Although the SC similarity score is less than 82.5% when the new N_CO corpus contains 60,000 annotated sentences, the similarity score reaches 82.8% when the new N_CO corpus contains 200,000 annotated sentences, representing an improvement of 0.3%.

[0213] The cross symbol “X” represents the SC similarity score obtained when the DRS2AMR annotation model is trained using the CO AMR 3.0 prior art corpus. The similarity score obtained is 83.2%. The DRS2AMR annotation model is described, for example, in the paper: Siyana Pavlova, Maxime Amblard, Bruno Guillaume. Bridging Semantic Frameworks: mapping DRS onto AMR. The 15th International Conference on Computational Semantics (IWCS 2023), Jun 2023, Nancy, France.

[0214] Curve C2 represents the evolution of the SC similarity score when the DRS2AMR annotation model is trained using the new N_CO corpus according to the invention. Although the SC similarity score is very slightly lower than 83.2% when the new N_CO corpus contains 60,000 annotated sentences, the similarity score becomes greater than 83.2% when the new N_CO corpus contains 80,000 annotated sentences and reaches a similarity score of 84.2% when the new N_CO corpus contains 200,000 annotated sentences. Thus, the improvement in annotation by the DRS2AMR annotation model, as obtained using the N_CO corpus according to the invention compared to the prior art CO corpus, is 1%.

[0215] The diamond symbol “♦” represents the SC similarity score obtained when the QALD9 annotation model is trained using the CO AMR 3.0 corpus of art Previous. The similarity score obtained is 87.3%. The QALD9 annotation model is described, for example, in the document: Young-Suk Lee, Ramôn Astudillo, Hoang Thanh Lam, Tahira Naseem, Radu Florian, and Salim Roukos. 2022. Maximum Bayes Smatch Ensemble Distillation for AMR Parsing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5379-5392, Seattle, United States. Association for Computational Linguistics (https: / / github.com / IBM / AMR-annotations).

[0216] Curve C3 represents the evolution of the SC similarity score when the QALD9 annotation model is trained using the new N_CO corpus according to the invention. Although the SC similarity score is very slightly lower than 87.3% when the new N_CO corpus contains 60,000 annotated sentences, the similarity score becomes greater than 87.3% for certain sizes of the new N_CO corpus, notably 70,000 and 110,000 annotated sentences, reaching a similarity score of 89.2% when the new N_CO corpus contains 200,000 annotated sentences. Thus, the improvement in annotation by the QALD9 annotation model, as obtained using the N_CO corpus according to the invention compared to the prior art CO corpus, is 1.9%.Such improvements are explained in particular by the fact that, thanks to the iterative annotation process according to the invention, a very large corpus can be obtained depending on the chosen iteration and possibly the filtering applied to the selection of the annotated text.

Claims

Demands

1. Iterative text data annotation method implemented in a computer device (CD), said method comprising the following at a current iteration (i): - receiving (S2) by a communication module an unannotated text (Tj) as input to at least three text annotation models of different architecture, the at least three models (MA1, MA2, MA3) having been trained from a first corpus of training data (CD) comprising annotated text data, - generating (S4) three annotated texts (TA1, TA2, TA3) by the three models respectively, - selecting (S5), by a selection module, one of the three annotated texts, the selected annotated text (TAs;) being the one which contains the fewest semantic and / or syntactic differences, in comparison with the two other annotated texts, - construct (S7), by a construction module, a new corpus of annotated textual data (N_CO) by adding to it the selected annotated text (TAs;) and the unannotated text (Tj), said new corpus constituting a second corpus of training data for the text annotation model.;

2. Method for iterative annotation of textual data according to claim 1, wherein the selected annotated text (TAs;) is the one which obtains a maximum semantic and / or syntactic similarity score (VSmax;) obtained after a comparison of the semantic and / or syntactic similarity between the at least three annotated texts.

3. Method for iterative annotation of textual data according to claim 2, wherein a filtering criterion (CF) is applied (S6) to the selection of said annotated text, according to which the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if the maximum semantic and / or syntactic similarity score is less than a threshold value (TH1).

4. A method for iteratively annotating textual data according to claim 2, wherein a filtering criterion (FC) is applied to the selection of said annotated text, whereby the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if is greater than a threshold value (TH2) the difference between the maximum semantic and / or syntactic similarity score (VSmax;) and the minimum semantic and / or syntactic similarity score (VSmin;) obtained at the end of the comparison of the semantic and / or syntactic similarity between the at least three annotated texts.

5. Method for iterative annotation of textual data according to claim 1, wherein the selected annotated text is the one which obtains the maximum average (VSmax;) between two semantic and / or syntactic similarity values ​​generated as a result of a pairwise comparison between said at least three annotated texts.

6. Method for iterative annotation of textual data according to claim 5, wherein a filtering criterion (CF) is applied to the selection of said annotated text, according to which the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if the maximum mean (VSmaxi) between two values ​​of semantic and / or syntactic similarity is less than a threshold value (TH3).

7. Method for iterative annotation of textual data according to claim 5, wherein a filtering criterion (CF) is applied to the selection of said annotated text, according to which the selected annotated text and the unannotated text are not used to construct the second corpus of textual data if, at the end of the pairwise comparison between said at least three annotated texts, the minimum mean (VSmin;) obtained for an annotated text is less than a threshold value (TH4).

8. Method for iterative annotation of textual data according to claim 5, wherein a filtering criterion is applied to the selection of said annotated text, according to which the selected annotated text and the unannotated text are not used to generate the second corpus of textual data if is greater than a threshold value (TH5), the difference between the maximum and minimum semantic and / or syntactic similarity values ​​obtained at the end of the pairwise comparison between said at least three annotated texts.

9. A method for iteratively annotating textual data according to any one of claims 1 to 8, wherein the unannotated text provided as input is subjected to at least three different annotation models The text is sourced from at least two different online content resources.

10. Iterative text data annotation device (AD), characterized in that it is configured to implement the following: - receive, via a communication module, an unannotated text as input from at least three text annotation models of different architectures, the at least three models having been trained from a first corpus of training data comprising annotated text data, - generate three texts annotated by the three models respectively, - select, via a selection module, one of the three annotated texts, the selected annotated text being the one that contains the fewest semantic and / or syntactic differences, compared with the other two annotated texts, - construct, via a construction module, a new corpus of annotated text data by adding to it the selected annotated text and the unannotated text,said new corpus constituting a second corpus of training data for a text annotation model.

11. A computer program comprising program code instructions for implementing the iterative text data annotation method according to any one of claims 1 to 9, when executed on a computer.

12. Computer-readable information carrier, and containing instructions for a computer program according to claim 11.