An evaluation method for ai systems

Abstract

Evaluation methods for generative Al systems for processing inputs from various sources in order to provide informative responses and formative feedback for users.

Description

[0001] An Evaluation Method for Al Systems

[0002] This application claims benefit from the following applications:

[0003] 63 / 502,924, 63 / 504,172, 63 / 504,726, 63 / 505,417, 18 / 668,125, 18 / 668,262, PCT / US24 / 30110

[0004] An Evaluation Method for Al Systems 1

[0005] 1 Evaluation of NLP systems 2

[0006] 1.1 Grammatical Error Correction (GEC) 3

[0007] 1.2 Recognizing Textual Entailment 4

[0008] 1.3 Probing Deep Learning NLP Models 6

[0009] 1.4 Semantic similarity algorithms "I

[0010] 2 Chatbot / Edubot 8

[0011] Figure 1 : Description 9

[0012] Figure 2: Description 10

[0013] Figure 4: Description 11

[0014] Figure 3: Description 11

[0015] Figure 5: Description 12

[0016] The invention is best described in the drawings and their explanations.

[0017] The first chapter of this document Evaluation of NLP systems describes known NLP evaluation technologies that the Evaluation Method (EM) utilizes as options. However, this application does not assert that the undersigned has invented the technologies described in chapter 1 sections 1-4.

[0018] Evaluation Method (EM) refers to a computer-implemented process for improving the ability for an Al system to evaluate similarities and differences between natural language expressions. EM comprises a combination of publicly available state-of-the-art NLP evaluation technologies (along with trade secrets) executed in an evaluation chain where each step improves the accuracy of the evaluation result, iteratively when required, as depicted in Figure 2. The EM-method can utilize publicly known NLP evaluation technologies introduced in the specification section Evaluation of NLP Systems, however, the method is not restricted nor limited to them and can also include fewer technologies than exemplified in the Figure 2. The order of the application of the different technologies in the evaluation chain is not fixed but can be customized to optimize the performance of the method. The EM method utilizes evaluation metrics, for example confidence score of results and relevance score of utterances for the topic.

[0019] Information on pages 12-30 (Cup and Comparative Study) is utilized in the Evaluation Method and the Feedback Module described in the next pages.

[0020] This work also proposes an extended IRF-based exchange structure for educational chatbots providing automated formative feedback educational context. IRFRE supports chatbot-directed, reflective individual learning but does not exclude the collaborative learning component before the student’s answer.

[0021] I initiation by the computer or teacher

[0022] (C) collaboration

[0023] R response by the student

[0024] F feedback by the computer

[0025] R = reassessment of the answer by the student based on the feedback (iterative)

[0026] E evaluation of the answer by the computer

[0027] The model enables multi-trial feedback strategies; students can have multiple trials to correct errors, they are also allowed to skip trials. The function of the feedback is to motivate the student to reassess and correct their initial answer in order to highlight potential gaps in their learning. The final assessment does not need to be formal, that is, an exam etc.

[0028] 1 Evaluation of NLP systems

[0029] Several NLP evaluation methods have been proposed over the years. Overview of approaches for evaluating and understanding the inference capabilities of NLP systems are reviewed in a research paper by Adam Poliak. The following paragraphs summarize his work. A distinction is typically made between intrinsic vs extrinsic evaluations and general purpose vs task specific evaluations. "Intrinsic evaluations test the system in of itself and extrinsic evaluations test the system in relation to some other task". Kovar et al. referred to this distinction using a concept of application-free versus application-driven evaluations. To differentiate between the concepts in relation to a document summarization system, an intrinsic evaluation might focus on the coverage of the key ideas in the summary while an extrinsic evaluation might focus on the usefulness of the summary in a search engine. Evaluations are further classified into General Purpose vs Task Specific evaluations . The general purpose evaluation uses test cases to assess NLP systems' ability to offer insights to different linguistic phenomena. The task specific evaluation typically tests the performance of models on a held out test corpus. The most commonly used metrics include accuracy, precision, F-measure and recall, or metrics like BLEU and ROUGE in generation tasks. Researchers typically test their systems with training and held-out test corpora using a wide range of metrics in order to prove that their system outperforms other researchers’ systems.

[0030] 1.1 Grammatical Error Correction (GEC)

[0031] Grammar checking refers to the detection and correction of grammatical errors in the text. Language learners especially need feedback on their outputs generally but correcting grammatical errors manually is rather tasking . The process of detecting and correcting grammar errors can be automated by using Grammatical Error Correction (GEC). Some researchers summarized GEC research approaches in their work referenced in the following paragraphs. Three main approaches can be identified in the field: Rule-based, Classification based and Machine Translation with Machine Translation further classified into Statistical Machine Translation and Neural Machine Translation.

[0032] In Rule based systems, a Part-of- Speech (POS) tagged text is checked against grammar rules that are used for error correction when needed. Rule based systems are not difficult to implement but they have their limitations; they do not generalize well because defining rules exhaustively is impossible; thus they do not detect more complex errors. Rule based systems generally provide detailed explanation of flagged errors thus making them useful for computer aided language learning.

[0033] Large error-coded corpora enabled more data-driven approaches for GEC that utilize Machine Learning Algorithms to build classifiers for correcting certain types of errors. In this approach, the words / replacements etc. of interest are treated as class labels, and the surrounding n-grams, PoS tags, and grammatical relations are used as features that are dependent on the error type. One classifier can detect a specific error type. To correct several error types, multiple classifiers can be cascaded into a pipeline. However, the classifier approach does not work well in case of dependent errors.

[0034] Combining this approach with the Statistical Machine Translation (SMT) approach enabled decoding the original sentence into several possible sentence corrections iteratively in order to find the best sentence correction. Grammatical correctness and fluency are used by the decoder to score sentences. The process completes after a max number of iterations or when all sentences have been processed.

[0035] Neural Machine Translation (NMT) systems utilize encoder-attention-decoder models. The encoder takes words in the source language and encodes them into vector representations. These word embeddings pass through an attention mechanism that determines their relevance. The decoder maps the vector representations into the target language. A bi-directional model includes the context of both past and future words to create a more context-aware encoder output vector. This approach was further improved by multilayer convolutional encoder-decoder neural networks (CNN) that use multiple layers of convolution to capture even wider contexts.

[0036] A hybrid approach combines the systems described in the previous paragraphs.

[0037] 1.2 Recognizing Textual Entailment

[0038] Recognizing Textual Entailment (RTE) is a unified evaluation framework proposed for the comparison of semantic understanding and inference capabilities of different NLP systems. Recognizing inferences plays a key role in understanding human language. "The task of recognizing textual entailment (RTE) takes a pair of text fragments as input and determines whether the meaning of one fragment (called hypothesis, H) can be derived from the meaning of the other fragment (called text, T), that is, whether T logically entails H". RTE has impacted many areas of natural language processing (NLP), such as information extraction, information retrieval, question answering, summarization, paraphrase acquisition, machine translation and reading comprehension. RTE is rooted in linguistics. The terms Natural Language Inference (NLI) and RTE are often used interchangeably, however, NLI is a broader term covering tasks like sentiment analysis, event factuality, or even question-answering without having to convert them into the sentence pair classification format used in RTE. Recognizing in RTE refers to the task of classifying whether the truth of one sentence likely follows the other. The second term textual refers to RTE's focus on textual data. E refers to Entailment. The strict definition of entailment in linguistics states that “sentence A entails sentence B if in all models in which the interpretation of A is true, also the interpretation of B is true”. RTE does not use a strict definition but allows cases in which inference is probable but not certain to be judged as True. According to some critics, due to this fuzzier definition, the E for entailment should be changed to a more appropriate and descriptive term I = Inference, RTI.

[0039] "Textual entailment (TE) is a directional relationship between T-H pairs: the entailment relation may hold from T to H but not from H to T". Typically, RTE can be a 2 -way classification task where T-H pairs are labeled as YES or NO entailment (2 -way classification task), mutually contradicting or not (3-way task) or with labels Entailment / Contradiction / Unknown like in the example below.

[0040] T A woman is talking on the phone while standing next to a dog

[0041] Hl A woman is on the phone (entailment)

[0042] H2 A woman is walking her dog (unknown)

[0043] H3 A woman is sleeping (contradiction)

[0044] Most RTE datasets use categorical RTE labels, however, several researchers have argued for scalar RTE labels indicating inference likelihood that can provide more insight into contemporary NLP models.

[0045] RTE datasets can be created automatically, semi-automatically or manually. The ability of the datasets to test the inferential capabilities of NLP models has been questioned by some researchers. Though an accuracy metric does capture the degree a model recognizes whether T logically entails H, general NEU would require NLP models to capture also different semantic phenomena. Recent research has taken interest in exploring the type of linguistic phenomena neural NLP models can capture. The popular auxiliary classifier-based diagnostic technique has been used in the studies to evaluate sentence representations. RTE datasets have been developed to help gain insight into linguistic phenomena captured by neural, deep learning models.

[0046] Recent RTE datasets with focus on specific linguistic phenomena can be used to evaluate sentence representations from neural models or to rank generated text from NLP systems. Furthermore, RTE has been used to evaluate the output of text generation systems. Falke et al. ranked generated summaries with a textual entailment system trained on SNLI with rather discouraging results; however, Barrantes et al. demonstrated that contemporary transformer models trained on the Adversarial NLI dataset were able to select a coherent summary. Basak et al. proposed an efficient rule-based method for assigning grades to students which was able to score 560 answers sufficiently close to human examiners.

[0047] Many methods have been used for RTE. One potential approach is to describe T and H as syntactic or dependency parse trees and assess the entailment based on the degree to which the H tree is included in the T tree. Some solutions use lexical matching, such as n-gram matching, percentage of word overlap, longest common subsequence, skip-gram matching, etc. Semantic techniques, such as semantic role labeling, atomic propositions, inference rules, universal networking language augmented with semantic similarity measures, etc. have been explored by some methods. Some methods use machine learning-based classification algorithms for RTE. Basak et al. used a hybrid method for RTE tasks combining a set of rule-based features with decision-making based on machine learning.

[0048] 1.3 Probing Deep Learning NLP Models

[0049] Deep learning NLP methods are harder to interpret than NLP approaches relying on feature engineering. Deep learning NLP systems utilize pre-trained encoders to transform the meaning of a sentence in a vector representation. Adi et al. suggested using auxiliary classifiers as explained in this paragraph for examining what language information is encoded in the sentence representations. Learned sentence-representations are treated as features to train a classifier. Auxiliary prediction tasks use pre-trained sentence encodings as input for other prediction tasks, whereas auxiliary diagnostic tasks focus on how word order, word content, and sentence length, phonetics, morphology, syntax or semantics are captured in pre-trained sentence representations. “The general purpose methodology of auxiliary diagnostic classifiers is also used to explore how well different pre-trained sentence representation methods perform on a broad range of NLP tasks” like paraphrase detection, semantic textual similarity, and other classification problems.

[0050] Belinkov et al. explored the vector representations learned at different layers of NMT encoders in order to understand what they actually learn about language. They trained NMT systems on parallel data and used the trained models to extract features for training a classifier on two tasks: PoS and semantic tagging. The measured performance of the classifier reflects the quality of the original NMT model for the given task. According to their research, quantitative analysis on representation learning in NMT models indicated that higher layers seem to learn more about semantics while lower layers are better for word-level linguistic properties like part-of-speech (POS) and morphological tags.

[0051] There are many techniques for analyzing neural network models:

[0052] 1) visualization of hidden units,

[0053] 2) a search for quantitative correlations between parts of the neural network and linguistic properties in language processing models,

[0054] 3) probing hidden vectors from neural MT models to predict linguistic

[0055] Properties.

[0056] 1.4 Semantic similarity algorithms

[0057] Question answering, document summarisation, information retrieval, information extraction all utilize Semantic Textual Similarity (STS) Calculation. Similarly chatbots need to use STS methods to assess the similarity between learners' inputs and actual answers. There are several methods to calculate the similarity between two sentences, and they can be classified into three groups:

[0058] 1) traditional approaches like edit distance / Levenshtein distance ("the minimum number of single-character edits (i.e. insertions, deletions or substitutions) required to change one word into the other” ), 2) word vector approaches like glove, word2vec and fasttext word embedding models ("a real-valued vector that encodes the meaning of the word such that the words that are closer in the vector space are expected to be similar in meaning") with distance measures like cosine similarity, word moving distance, smooth inverse frequency) and

[0059] 3) context vector approaches like ELMO, BERT and FLAIR embeddings. BERT and ELMo embeddings are context-sensitive, meaning that words with the same spelling but different meanings will have different representations like "He is running outside" vs "He is running a company".

[0060] All recent STS methods rely on word embeddings to an extent. The modern contextualized word embeddings tend to perform better than traditional word embeddings in many NLP tasks. This work will evaluate different STS methods for the purposes of implementing automated formative feedback in an edubot implemented.

[0061] 2 Chatbot / Edubot

[0062] The edubot provides learners automatic formative feedback. The most important part of the bot is a module that builds the chatbot-mediated automatic formative feedback from the evaluation provided by the integrated tools. The rest of the system will use state-of-the-art and / or established approaches along with trade secrets. The proposed evaluation / feedback module of the edubot utilizes rule based, statistical and neural network based grammar checking, especially focusing on semantics, the meaning level of language. The system uses semantic information provided by different evaluation tools (GEC, RTE, semantic / syntactic tags in NN networks, NMT, answers selections algorithms, STS tools) in edubot dialogic context to offer learners automatic formative feedback beyond correct / incorrect binary assessment. My system builds on With the motivation in the preceding paragraphs, the objective of the system is to provide an answer how to best evaluate learners’ answers to open-ended questions and to give effective automatic formative feedback based on them in an immediate task-level feedback setting.

[0063] Figure 1: Description

[0064] Quizzes, fill-in-the-blank and open-ended questions areimplemented as activity types in the chatbot. After an error is found, the system generates an automatic formative feedback that is immediate and task-level. These tutoring feedback messages use multi -trial feedback strategies; students can try to answer the questions many times. The conversation flow and the feedback type differ depending on the question type.

[0065] Prior to feedback and assessment, the learner’s performance needs to be evaluated semantically. For quizzes, the evaluation process is straightforward; there is only one correct answer and it is technologically easy to assess whether students chose the right answer among the optional ones. When the QA pairs are entered into the chatbot database, the QA designer can indicate the correct option numerically in the input form / sheet.

[0066] The keywords can also be automatically extracted from the sentence / paragraph / text.

[0067] Also, fill-in-the-blank questions typically have one correct answer that Semantic Textual Similarity tools can compare with the student answer.

[0068] Figure 2: Description

[0069] However, the evaluation process is more difficult regarding open-ended questions.

[0070] Traditional grammar checkers typically return textual feedback that can be displayed to the user in the chatbot interface, however, there are not yet tools available that would be capable of providing extensive feedback on semantic similarity between two sentences. Current tools typically return a numeric score. RTE tools assess the inference between sentences with labels like entailment, contradiction or neutral. The lack of extensive automatic semantic feedback is also due to the fact that it is still a challenge for chatbots to reliably interpret what the user is saying which presents a challenge not only to systems trying to provide accurate answers to users’ questions but also for automated feedback / assessment of students’ answers. For the evaluation of open-ended questions, the edubot / chatbot will integrate external, available tools from GEC, RTE, NMT, semantic / syntactic tagging and semantic textual similarity research fields and utilize them for semantic level feedback in edubot / chatbot context.

[0071] In figure 2 the hot asks the user a question and uses Grammatical Error Correction (GEC) to analyze the answer for grammatical errors. The user is offered a possibility to correct the errors. Evaluation Method (EM) utilizing technologies presented in the specification chapter 1 analyzes the user answer and makes a prediction of its correctness utilizing evaluation metrics. The user is offered hints to correct the answer.

[0072] The Evaluation Method (EM) refers to a computer-implemented process for improving the ability for an Al system to evaluate similarities and differences between natural language expressions. The EM-method comprises a combination of publicly available state-of-the-art NLP evaluation technologies (along with trade secrets) executed in an evaluation chain where each step improves the accuracy of the evaluation result, iteratively when required, as depicted in Figure 2. The EM-method can utilize publicly known NLP evaluation technologies introduced in the specification section Evaluation of NLP Systems, however, the method is not restricted nor limited to them and can also include fewer technologies than exemplified in the Figure 2. The order of the application of the different technologies in the evaluation chain is not fixed but can be customized to optimize the performance of the method. The EM method utilizes evaluation metrics, for example confidence score of results and relevance score of utterances for the topic.

[0073] Figure 4: Description

[0074] QA-pairs in the QA database (QA-DB) can be retrieved, extracted, transformed and evaluated from the knowledge base (QA-KB). Evaluation Method evaluates their relevance to the topic.

[0075] QA-(L)LMs are trained and / or fine-tuned using primarily the QA-KB and QA-DB.

[0076] QA-DB is continuously updated with new knowledge. Figure 3: Description

[0077] 1) QA-pairs can be manually inserted or uploaded by the user. QA-pair sources can be documents, material (knowledge base QA-KB) provided by the user or other available legit sources. QA-pairs can also be generated by a language model (QA-GM) or derived directly from the QA-DB. QA-database and QA-knowledge base can be used to train the QA-language model (Figure 4).

[0078] 2) QA-pairs from the QA-KB are retrieved, extracted, transformed and evaluated by the

[0079] EM-method before being inserted to the QA database (QA-DB). Similarly, QA-GM (generative module) generated QA-pairs are transformed and evaluated. QA-pairs can be generated all at once or 1-by-l. In the former case the execution loops between steps 3-6 until the user quits the chat or questions end.

[0080] 3) The question selected from the QA-DB is being processed by the Question Processing Module before being displayed to the user.

[0081] 4) The user’s answer to the question is processed by the Answer Processing Module before being evaluated by the EM evaluation method.

[0082] 5) Feedback Module provides formative feedback to the user based on the output of the

[0083] Evaluation Method. Feedback is given both in case of a correct and incorrect answer.

[0084] Both task-level and summative feedback is provided.

[0085] 6) Users are given an option to try answering the same question again, answer a new question or quit.

[0086] 7) Before ending the chat, users are given summative feedback on their overall performance during the learning session.

[0087] Figure 5: Description

[0088] 1) Conversation starts with a user question.

[0089] 2) Question Processing Module (QPM) is responsible for processing the question.

[0090] 3) Initial check whether the QA-DB already contains an answer to the question with an evaluation metric > threshold value. If Yes, execution jumps to step 7 displaying the answer to the user, otherwise execution continues to step 4.

[0091] 4) Answer Selection Module (ASM) includes three submodules:

[0092] • QA-KB retrieves, extracts, transforms optional answers from the knowledge-base and evaluates them with EM-method

[0093] • QA-DB selects optional answers from the database and evaluates them with

[0094] EM-method

[0095] • QA-GM generates answers using language models and evaluates them with EM-method ASM module output -> optional answers

[0096] 5) Answer processing module (APM) processes the optional answers to ensure the right format

[0097] 6) EM-method evaluates the answers.

[0098] 7) The best answer is displayed to the user.

[0099] 8) The best answer is updated / inserted to the QA-database along with the evaluation metric.

[0100] 9) Users are given a chance to ask a new question, see optional answers for the same question or to quit, (if user asks the same question, the earlier retrieved optional answers are shown before starting the process again with the same question)

[0101] Anomaly Cup

[0102] Johanna Virtanen

[0103] Unknown / illegal opcodes

[0104] It might have been the most professional approach to list all the possible known opcodes and to work with them. An opcode database does exist. Also the so called illegal opcodes would probably need to be taken into account when building a real application

[0105] However, in this assignment the unique opcodes were extracted from the train files and then this list was used to extract opcodes from the train / test files.

[0106] Some opcodes only occured in the test set. It is probable that these unknown opcodes are significant when separating files

[0107] -These unknowns were considered by adding extra column before the 809 opcodes found from the train set In the train matrix, this column contains only zeros. In the test set these unknown opcodes are scattered into nine files.

[0108] If an opcode database had been used, it would have been possible to determine whether these unknowns are illegal opcodes and thus assign them a high anomaly value.

[0109] However, when working with the opcodes extracted from the training set, the extra column is not veiy useful, or might even weaken the model since one cannot know whether they occur frequently in normal or virus files

[0110] Opcode anomaly score with 1-grams

[0111] Opcodes were assigned an anomaly score between 0-1 by dividing the opcode count in virus files with the total sum of the opcodes in all files.

[0112] 1 - opcode occurs only in virus files, 0 - occurs only in normal files

[0113] Then these scores were used to assign the files an overall anomaly score: Sum(opcode count per file.*opcode anomScore)

[0114] However, a thresold was used to make sure this sum reflects the file anomality. Using a thresold 1 (opcodes only in viruses) the file anomaly score is only assigned to the virus files. However, this thresold value is not the only option.

[0115] In terms of the anomaly score the unknown opcodes are problematic. One cannot know whether they occur in normal or virus files in the test set. One option would have been to give them an anomaly score of 0.5, however, here 0 was used to make sure they won’t falsely affect a file anomaly score by giving too much weight on innocent opcodes - Anomaly score is only used with 1-grams, it would be possible to use it with n-grams as well with some modification of the code

[0116] The Total Opcode Count

[0117] There were 809 unique opcodes found from the training set.

[0118] The unique opcodes / bytes were extracted the following way:

[0119] 1) training files were read in a for-loop using dataset-function

[0120] 2) using the union-function all the uniques / file were united to form a vector containing all the opcodes in the training set: allOps-variable

[0121] 3) this vector was converted to decimals: allOpsDec=(0:809); (0 here represents the unknown opcodes in the test set) opData = datasetfFile', str, 'ReadVarNames', false); % all data into a dataset allOps = union(allOps,unique(opData.Varl)); % uniting all the opcodes found in the files

[0122] Byte values vaiy between 0-255 so this phase was not necessaiy for them

[0123] Extracting the Ops / Bytes - 1-grams

[0124] (opcode files 283 and 318 are so big they are read in blocks-> readLong(file) and inserted separately to the final matrix)

[0125] After constructing the allOpsDec-vector, the vector is used to extract the opcode counts per file:

[0126] 1. Read files in a loop and convert file opcodes into decimals (ib return value represents file opcodes in a decimal type)

[0127] 2. ([~,ib]=ismember(opData. Vari, allOps);

[0128] 3. Check unique (unique(ib)) opcodes per file, their counts (levelcounts(nominal(ib))) and the total opcode count (length(opData))

[0129] 4. Check which opcodes out of allOpsDec are in the file using ismember-function and insert their counts in the vector using the index vector

[0130] 5. ([~,b]=ismember(nominal(fileUnique),nominal(allOpsDec), ’rows');

[0131] 6. rowOps(b)=fileCount; % per opcode count in the current file

[0132] 7. Divide the unique opcode counts in the file with all the opcodes in the file to make the counts relative to the length of the file (rowOps=rowOps / length(opData);) 8. Insert the vector to the matrix of opcode / byte counts per file (freqs(i,:)=rowOps’)

[0133] 9. Count the file opcode anomaly score in the last column and join the byte and opcode matrixes -> freqsTrainl.mat / freqsTestl.mat

[0134] Extracting Ngrams - Combination

[0135] After constructing the allOpsDec-vector, the vector is used to extract the n-gram opcode counts per file:

[0136] All the combinations of the unique opcodes in the data are counted with the help of combntns(n,k) function in Mapping Toolbox (I don’t have this toolbox on my computer but used university Matlab to verify correct results of the code and to form 2 -gram combinations for testing: files uu.mat (ops) and dd.mat (bytes))

[0137] K is the size of an n-gram. If one takes 2 -grams out of 809 opcodes, the possible combinations are 809*809=654481. The code below returns just that number of 2-grams. u = []; k=2; for i=l:k u=vertcat(u,allOpsDec); % to have all the combinations one must have k*A110psDec vector end combos = combntns(u,k); % there are repeated combinations in combos allGrams = unique(combos, ’rows'); % here are the unique combinations=654481

[0138] Extracting Ngrams - Overview use regexp to iterate the files in search of n-grams. Below finding all k-grams m2s=mat2str(grams); % change the combinations to strings

[0139] [mat] = regexp(m2s, etsi, 'match'); % searching for ngrams from the combinations string for n=l:k one needs to start from the first opcode and take ngram-steps (for example if 3-grams iterate l:end,2:end,3:end) regexp returns the matched strings

[0140] Once the per ngram counts are looped, sum all the counts to get the total ngram count of a file. Divide the row / total to standardize the rowGram counts

[0141] Now check which ngrams of the allGrams occur in the file ([~,ib]=ismember(rowGrams, allGrams))

[0142] Insert the found ones in the matrix of ngrams(i,ib’)=rowGrams;

[0143] My computer is not powerful enough to actually test the code totally but I this code should extract ngrams of any length and with small data it seems to work

[0144] For this assignment I only use 1-grams but readNGrams has the code for reading any length of ngrams

[0145] Extracting Ngrams - Code

[0146] - opData = dataset(File', st, 'ReadVarN antes', false); %after forming the combinations read the file if(strcmp(s, 'opcode')) % if extracting ngrams from ops load('allOps') % load all 809 opcodes in a vector load(’grams') % load all the combinations of ngrams

[0147] [~,ib]=ismember(opData. Vari, allOps); % change per file opcodes numeric using ismember (ib is a decimal representation of opcodes) str=mat2str(ib'); % change the decimals to strings m2s=mat2str(grams); change the combinations to strings

[0148] Else % if extracting the bytes they are already decimals ib=opData.Varl; str=mat2str(ib'); % change the decimals to strings end

[0149] Extracting Ngrams - Code etsi=blanks(O); % create an empty string etsil='[0-9]\w* '; % search string that starts with a number, is followed by any chars

[0150] [a-zA-Z_0-9] and ends with a whitespace etsi2='[0-9]\w*'; % when the last string of the match string, whitespaces not needed for h=l:k % looping the length of the searched ngram if(h==max(k)) % if the last [0-9]\w* inserting no whitespaces at the end etsi=[etsi,etsi2]; else etsi=[etsi,etsil]; % here the search string is not yet finished end

[0151] %disp(etsi) %for testing end

[0152] [idx] = regexp(str, ' ', 'start'); % finding the whitespaces in the file string for iteration

[0153] [mat2] = regexp(str, etsi, 'match'); % searching for ngrams from the file string

[0154] [mat] = regexp(m2s, etsi, 'match'); % searching for ngrams from the combinations string matAll=mat2; % variable for all ngrams in the file

[0155] Extracting Ngrams - Code for j=l:k-l % looking the file for ngrams k-1 times g=idx(j)+l; % finding the j-th whitespace in the file strK=str(g:end); % splitting the string to start from the whitespace

[0156] [matK] = regexp(strK, etsi, 'match'); % make a new ngram search from the split string matAll=horzcat(matAll,matK); % concatenate the new ngrams to all ngrams end

[0157] An example: if the file string is str=[l 10 2 9 3 4 5 6 4 5]; and one wants 3-grams, the first round matK = regexp(str / [0-9]\w *[0-9]\w* '[0-9]\w*’, ’match'); returns (1 10 2, 9 3 4, 5 64), the second round starting with the first whitespace returns (10 2 9, 3 4 5, 6 4 5) and the third roud (2 9 3, 4 5 6). Now all the file 3-grams are found? u=unique(matAll); % find the unique opcodes per file l=levelcounts(nominal(matAll)); % and their levelcounts [~ ,hh] =ismemb er(u, mat); % check their position in all combinations row(l,l:length(mat))=0; % create an empty row with zeros for non-existent items in the file row(hh)=l; % insert levelcounts in the right index row=row / sum(l); % divide the counts with the total number of n-grams in the file

[0158] Extracting Ngrams

[0159] As mentioned I wasn’t able to fully test the ngram code for two reasons:

[0160] 1) I don’t have Mapping Toolbox on my computer

[0161] 2) my computer is not computationally powerful enough for ngrams bigger than 1-grams

[0162] However, I wanted to return a code that should be able to extract ngrams of any length from files

[0163] One can test the ngram extraction with script test_something. If one wants to try with bigger than 2-grams some code must first be taken out of comments %

[0164] For the assignment I will only evaluate 1-gram solution

[0165] Preprocessing and model building

[0166] - freqsTrain and freqsTest are standardized since the row opcode counts are divided with the total file opcode count

[0167] - the anomScore and the opcode count columns are between 0-1 so no preprocessing was used

[0168] The model uses Knn method with 6-fold

[0169] First the training set is divided into 6 folds. The folds are iterated to train the ClassificationKnn-object

[0170] The model does a good job of predicting the label for the training items perfMetrQ function returns the performance metrics perfcurveQ function is used to produce the ROC and AUC values

[0171] 0 is considered the positive class; ClassificationKnn returns the scores for ROC curve [ROC , AUG ]=perfcurve(truth,scores(:, 1))

[0172] Results

[0173] Confusion Matrix

[0174] 602 0

[0175] 0 87

[0176] CUP Performance Metrics

[0177] FPR 0

[0178] TNR 1

[0179] Accuracy 1

[0180] Recall 1

[0181] Precision 1

[0182] F-measure 1

[0183] 0.88% of normal files in the test set

[0184] Results A Comparative Study of Google's T5 Model, LanguageTool, and Wordtune for Grammatical Error Correction

[0185] (accepted for pubheation)

[0186] Johanna Virtanen

[0187] Abstract

[0188] This paper compares the performances of Google's T5 model [1-4], LanguageTool [5], and Wordtune for grammatical error correction. The models are evaluated on two benchmarks: BEA[6] and Conll-14 [7]. The results show that the T5 model and Wordtune outperform LanguageTool on both benchmarks. The Errant tool is used to analyse the errors made by the models. The results show that the T5 model and Wordtune are more accurate than

[0189] LanguageTool at correcting a variety of grammatical errors.

[0190] Keywords: GEC, T5, Wordtune, LanguageTool, NLP

[0191] Introduction

[0192] Grammatical error correction (GEC) is the task of automatically correcting grammatical errors in text. GEC is a challenging task, as it requires the model to generate grammatically correct sentences.

[0193] There are two main approaches to GEC: rule-based and neural network based. Rule-based GEC systems use a set of predefined rules (error patterns) to identify and correct grammatical errors in text. Neural network based GEC systems use a neural network to learn the relationship between grammatical errors and their corrections.

[0194] In recent years, neural-based GEC systems have outperformed rule-based GEC systems. This is due to the fact that neural networks are able to learn more complex relationships between grammatical errors and their corrections than hand-crafted rules.

[0195] This paper compares the performances of three GEC systems: Google's T5 model, LanguageTool, and Wordtune. We evaluate these tools on two benchmarks: BEA and Conll-14. Materials and Methods

[0196] The T5 is a neural network based pretrained transformer-based language model that can be fine-tuned for specific NLP tasks.The model is able to perform a variety of tasks, including translation, summarization, and question answering. This study uses a T5 small model that was fine-tuned with the cLang-8 dataset.

[0197] LanguageTool is a rule-based GEC system that supports over 30 languages.

[0198] Wordtune is a neural-based GEC system that uses a neural network to learn the relationship between grammatical errors and their corrections.

[0199] Models are evaluated on two benchmarks: BEA and Conll-14. The ERRANT scorer was used for BEA benchmark to perform the error analysis and identify the types of errors made by each system; M2 scorer performed the evaluation on CoNLL-14.

[0200] Results and Discussion

[0201] The Errant tool was used to analyse the errors made by the models. The results show that the T5 model and Wordtune outperform LanguageTool on both benchmarks. The T5 model and Wordtune are more accurate than LanguageTool at correcting a variety of grammatical errors. The T5 model fine-tuned with cLang-8 dataset performs the best since out of the 24 error categories evaluated by the Errant tool, T5 model got a F0.5 score > 50% in 19 / 24. Wordtune scored over 50% in 15 error categories whereas LanguageTool succeeded to get a F0.5 score > 50% in four categories. All three GEC tools were especially good at detecting errors in Verb Inflection. None of the GEC tools scored over 50% in the following error categories: Adjective, Conjunction, Noun, Verb and Other. In the remaining 19 error categories, the T5 model performed the best in 9 categories and Wordtune in 10; LanguageTool did not manage to outperform neural network models in any error categoiy that Errant tool analyses. Taking into consideration also the categories with scores under 50%, the T5 model scored the best in 12 error categories and so did Wordtune. The differences in scores between the three models range from 0% to 70%. T5 fine-tuned with cLang8 achieved an overall Span-level correction F0.5 score of 66.77 on the BEA benchmark and an F0.5 score of 0.5832 on the Conll-14 benchmark.

[0202] Wordtune achieved a Span-level correction F0.5 score of 68.74 on the BEA benchmark and an

[0203] F0.5 score of 0.5952 on the ConH-14 benchmark.

[0204] LanguageTool achieved an F0.5 score of 38.16 on the BEA benchmark and an F0.5 score of

[0205] 0.2660 on the ConH-14 benchmark.

[0206] Table 1: The list of 24 main error categories in ERRANT tool [8] and F0.5 > 50% for evaluated GEC systems.

[0207] Conclusions

[0208] The results of this study show that neural network based GEC systems are more effective than rule-based GEC systems. The T5 model and Wordtune are two neural-based GEC systems that outperform LanguageTool on two benchmarks. The T5 model and Wordtune are more accurate at correcting a variety of grammatical errors and thus seem to outperform humans in this task.

[0209] Future work

[0210] In the future, the aim is to improve the performance of the T5 model by using a larger dataset, fine-tuning the models on a more diverse set of text and developing new methods for analysing the errors made by GEC systems.

[0211] Ethics approval and consent to participate

[0212] Not applicable.

[0213] List of abbreviations

[0214] GEC - Grammatical Error Correction

[0215] Data Availability

[0216] Google’s T5 models: https: / / github.com / google-research / text-to-text-transfer-transformer

[0217] T5 small: https: / / huggingface.co / Unbabel / gec-t5 small cLang-8: https: / / github.com / google-research-datasets / clang8

[0218] Wordtune: https: / / www.wordtune.com /

[0219] L anguageTool: https : / / pypi. org / proj ect / language-tool-python /

[0220] Bea benchmark: https: / / codalab.lisn.upsaclay.fr / competitions / 4057

[0221] Conll-14: https: / / www.comp.nus.edu.sg / ~nlp / conlll4sthtml Acknowledgments

[0222] The author would like to thank her supervisor and the university.

[0223] References

[0224] [1] Google Research (2019). T5: Text-To-Text Transfer Transformer [Source code] https: / / github.com / google-research / text-to-text-transfer-transformer

[0225] [2] Rothe, Sascha & Mallinson, Jonathan & Malmi, Eric & Krause, Sebastian & Seveiyn,

[0226] Aliaksei. (2021). A Simple Recipe for Multilingual Grammatical Error Correction.

[0227] [3] Raffel, Cohn & Shazeer, Noam & Roberts, Adam & Lee, Katherine & Narang, Sharan &

[0228] Matena, Michael & Zhou, Yanqi & Li, Wei & Liu, Peter. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.

[0229] [4] https: / / huggingface.co / Unbabel / gec-t5_small (accessed Feb. 19, 2023).

[0230] [5] https: / / languagetool.org / about (accessed Feb. 20, 2023).

[0231] [6] Biyant, Christopher & Felice, Mariano & Andersen, 0istein & Briscoe, Ted. (2019). The

[0232] BEA-2019 Shared Task on Grammatical Error Correction. 52-75.

[0233] 10.18653 / V1 / W19-4406.

[0234] [7] Ng, Hwee & siew mei, wu & Briscoe, Ted & Hadiwinoto, Christian & Susanto, Raymond

[0235] & Biyant, Christopher. (2014). The CoNLL-2014 Shared Task on Grammatical Error Correction. 1-14. 10.3115M / W14-1701.

[0236] [8] Biyant, Christopher & Felice, Mariano & Briscoe, Ted. (2017). Automatic Annotation and

[0237] Evaluation of Error Types for Grammatical Error Correction. 793-805. 10.18653 / V1 / P17-1074.

[0238]

[0239]

[0240]

Claims

CLAIMSClaim 1: 1 claimA computer-implemented Evaluation Method (EM), which refers to a process for improving the ability for an Al system to evaluate similarities and differences between natural language expressions, comprising of steps. a combination of publicly available state-of-the-art NLP evaluation technologies (along with trade secrets) executed in an evaluation chain where each step improves the accuracy of the evaluation result, iteratively when required, resulting in the most accurate option, as depicted in Figure 2,The EM method utilizes evaluation metrics, for example confidence score of results and relevance score of utterances for the topic, the EM-method can utilize publicly known NLP evaluation technologies introduced in the specification section Evaluation of NLP Systems, however, the EM-method is not restricted nor limited to them and can also include fewer technologies than exemplified in the Figure 2, the order of the application of the different technologies / steps in the evaluation chain is not fixed but can be customized to optimize the performance of the method.Claim 2: 1 claim an Al Chatbot system and its process, comprising of steps.Generating as accurate as possible answer to users’ questions by producing optional answers from various available sources (including user provided knowledge bases, documents, LLMs, databases, any publicly available accurate data), evaluating these optional answers using the evaluation method (EM) depicted in the claim 1 and Figures, first within-source evaluation in PHASE 1 that produces the most accurate answers per source, then within-system evaluation in PHASE 2 that produces the most accurate answer from the PHASE 1 answers as depicted in Figure 5 - Chatbot, updating the system database continuously to serve the system,Users are offered a possibility to quit, continue to a new question, or review more answers to the same question.Claim 3: 1 claim an Al Edubot svstem ai irocessProviding learners automated formative task-level and summative feedback by first producing question-answer-pairs from various available sources including user provided knowledge bases, documents, manual uploads etc., LLMs, databases, any publicly available accurate data, evaluating these QA-pairs using an evaluation method (EM) depicted in the claim 1 and figures both within-QA-pairs in PHASE 1 and in PHASE 2 after the user has answered the system-generated question as depicted in Figure 3, 4 - Edubot, QA, storing the best QA-pairs in the QA-database in order to offer learners a possibility to automatically review their learning material by answering questions about it, and providing them automated feedback on their performance utilizing a Feedback module (FBM),After answering a question, users are given an option to 1) correct their answers and see a correct answer, 2) answer a new question or 3) quit, before ending the chat, users are given summative feedback on their overall performance during the learning session.

Images