RAG-based electrical protocol information extraction method

Abstract

The invention discloses an RAG-based electrical protocol information extraction method, which comprises the steps of generating a training data set according to a known electrical text and a collected electrical concept, and adjusting a text embedding model based on the training data set; when the information in the electrical protocol is extracted, on the basis of the adjusted text embedding model, a preset rule answer refusing mechanism and a preset uncertainty detection mechanism, wrong answers are recognized and refused, and answers meeting requirements are obtained through screening and serve as the extracted electrical protocol information. According to the method, the text embedding model can be adjusted, so that the method has higher accuracy and effectiveness when the electrical related text is processed. Meanwhile, an answer quality detection mechanism is introduced in the embodiment of the invention to ensure that the finally output answer is accurate and credible.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and artificial intelligence technology, and in particular to a method for extracting electrical protocol information based on RAG. Background Art

[0002] In industrial and power systems, electrical cabinet protocols usually contain a lot of technical details and complex information. This information is crucial to ensure the performance, safety and reliability of electrical cabinets. However, since protocols are usually long and complex, manual extraction of key information is not only time-consuming and laborious, but also prone to errors. Therefore, it is necessary to use corresponding text key information extraction technology to extract information.

[0003] Existing text key information extraction technology plays a key role in the field of natural language processing, aiming to automatically identify and extract important information from large amounts of text. The main methods that can be used in the corresponding extraction technology include rule-based methods, machine learning-based methods, and deep learning-based methods; among them,

[0004] (1) The rule-based method extracts information through predefined rules and patterns, usually using regular expressions or specific text pattern matching to identify the required information. This information extraction method has the disadvantages of poor flexibility and difficult rule maintenance.

[0005] (2) The machine learning-based method is to identify and extract key information from text by training machine learning models; common models include classifiers, sequence labeling models, etc.; this information extraction method has the problem of poor extraction effect when facing complex texts and extraction requirements;

[0006] (3) The deep learning-based method uses deep learning models (such as RNN, LSTM, Transformer, etc.) to automatically extract information from text. In particular, large language models represented by GPT3 have performed particularly well in recent years. The current mainstream text information extraction method is to use RAG (retrieval-augmented generation) technology to combine information retrieval with large language models to enhance the accuracy and relevance of the results generated by large language models. However, when this information extraction method is applied in professional fields, it often requires a large amount of high-quality data for pre-training, which has high data preparation costs. In addition, it may generate meaningless responses, affecting user experience.

[0007] In view of the defects of the above-mentioned information extraction methods in the prior art, the present invention is proposed. Summary of the invention

[0008] The purpose of the present invention is to provide a RAG-based electrical protocol information extraction method to solve the above-mentioned technical problems existing in the prior art and ensure that the final output answer is accurate and credible.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] A method for extracting electrical protocol information based on RAG, comprising:

[0011] Generate a training data set based on known electrical texts and collected electrical concepts, and adjust the text embedding model based on the training data set;

[0012] When extracting information from an electrical agreement, based on the adjusted text embedding model, a predetermined rule rejection mechanism, and a predetermined uncertainty detection mechanism, incorrect answers are identified and rejected, and answers that meet the requirements are screened as the extracted electrical agreement information.

[0013] Preferably, the process of generating a training data set based on known electrical texts and electrical concepts includes:

[0014] Generate a training data set of positively related text and negatively related text including questions and corresponding question-answer pairs by constructing a prompt word Prompt and a language model ChatGPT according to the obtained electronic text, wherein the negatively related text is obtained by searching based on a text embedding model;

[0015] According to the obtained electrical concepts, a training data set including electrical concepts and their corresponding similar semantic texts and irrelevant texts is generated by constructing prompt words Prompt and language model ChatGPT.

[0016] Preferably, the predetermined rules include: a preset format of an answer output when extracting information from an electrical protocol.

[0017] Preferably, the predetermined uncertainty detection mechanism includes:

[0018] The probability of all words in the answer, the probability of the most uncertain word in the answer, and the entropy of the words in the answer are output when extracting information from the electrical protocol, as well as pre-set rules for filtering answers based on the probability of all words in the answer, the probability of the most uncertain word in the answer, and the entropy of the words in the answer.

[0019] Preferably, the method for determining the likelihood of all words in the answer includes:

[0020]

[0021] Among them, J is the number of word tokens contained in the answer, and p iis the generation probability of the i-th token in the answer.

[0022] Preferably, the method for determining the likelihood of the most uncertain word in the answer includes:

[0023]

[0024] Among them, p i is the generation probability of the i-th token in the answer.

[0025] Preferably, the entropy of the words in the answer includes: the average entropy of all words in the answer and the entropy of the most uncertain word in the answer, wherein:

[0026] The average entropy of all words in the answer is:

[0027]

[0028] The entropy of the most uncertain word in the answer includes:

[0029]

[0030] The entropy definition of the word includes:

[0031] H i =∑ ω∈W p i (ω)log p i (ω);

[0032] W is the set of all possible words in the vocabulary, p i (ω) is the probability of generating word ω at the i-th word token.

[0033] Compared with the prior art, the present invention provides a RAG-based electrical protocol information extraction method, which can use a large language model (such as GPT-3.5) to automatically annotate data sets, thereby efficiently generating a large number of high-quality text data sets in the electrical field; after that, the model can be fine-tuned through the data set to make it more accurate and effective when processing electrical-related texts. In addition, the above implementation scheme also introduces an answer quality detection mechanism, which evaluates the reliability of the output by combining manually defined rules and answer uncertainty indicators, thereby filtering out unreliable answers and ensuring that the final output answers are accurate and credible. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0035] Figure 1 A schematic diagram of the implementation flow of the extraction method provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a process for generating a training data set from a public knowledge base provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a process for generating a training data set from electrical text provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0039] First, the terms that may be used in this article are explained as follows:

[0040] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0041] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0042] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0043] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.

[0044] When concentration, temperature, pressure, size or other parameters are expressed in the form of a numerical range, the numerical range should be understood to specifically disclose all ranges formed by the pairing of any upper limit, lower limit, and preferred value in the numerical range, regardless of whether the range is explicitly stated; for example, if a numerical range of "2 to 8" is stated, the numerical range should be interpreted as including ranges such as "2 to 7", "2 to 6", "5 to 7", "3 to 4 and 6 to 7", "3 to 5 and 7", "2 and 5 to 7", etc. Unless otherwise specified, the numerical ranges stated herein include both their end values ​​and all integers and fractions within the numerical range.

[0045] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.

[0046] With the development of natural language processing (NLP) technology, it has become possible to use automated tools to extract key information from electrical cabinet protocols. The implementation process of the present invention is based on the electrical protocol information extraction method implemented based on RAG technology. RAG technology is a natural language processing technology that combines retrieval and generation. It can use a large-scale corpus for information retrieval and provide rich background knowledge and context information for the generation process, thereby improving the accuracy and diversity of the generated results. The corresponding RAG technology can be widely used in text generation, dialogue systems, question-and-answer systems and other fields. Specifically, the present invention is mainly an electrical protocol information extraction method implemented based on the RAG knowledge question-and-answer system.

[0047] In the specific implementation process, the present invention mainly targets the two shortcomings of the information extraction scheme based on the RAG method in the "deep learning-based method" in the prior art, and proposes an implementation scheme for automatic generation of training data sets and answer quality control, thereby solving the problems of difficulty in preparing the corresponding data sets and low-quality answers.

[0048] The present invention specifically provides a method for extracting electrical protocol information based on RAG, which may include but is not limited to the following processing steps during implementation:

[0049] First, a training data set containing electrical related information is generated according to known electrical texts and collected electrical concepts and other electrical related information, and a text embedding model is adjusted based on the training data set;

[0050] Afterwards, when extracting information from the electrical protocol, based on the adjusted text embedding model, the predetermined rule rejection mechanism and the predetermined uncertainty detection mechanism, the wrong answer (i.e., answer) can be identified and rejected, and then the answer that meets the requirements can be screened and obtained as the extracted electrical protocol information;

[0051] Among them, the predetermined rules may include: a predetermined format of the answer output when extracting information from the electrical protocol, and the predetermined uncertainty detection mechanism may include: the probability of all words in the answer output when extracting information from the electrical protocol, the probability of the most uncertain word in the answer and the entropy of the words in the answer, and a predetermined rule for screening answers based on the probability of all words in the answer, the probability of the most uncertain word in the answer and the entropy of the words in the answer.

[0052] To facilitate the understanding of the present invention, the specific implementation of an embodiment of a method for extracting electrical protocol information based on RAG provided by the present invention will be described in detail below with reference to the accompanying drawings.

[0053] Specifically, Figure 1As shown, during the implementation of the specific embodiment of the present invention, it can mainly include but is not limited to the following processing steps:

[0054] Step 1: Prepare electrical-related DOCX / PDF documents for automatic generation of training data sets;

[0055] Step 2: Use the data automatic annotation module to automatically generate a training data set in the required format based on existing electrical related documents;

[0056] Step 3, using the annotated data (i.e., training data set) to fine-tune the text embedding model in the RAG-based knowledge question answering system, so as to update the electrical-related information contained in the corresponding electrical-related DOCX / PDF document into the text embedding model, that is, to update the text embedding model to a text embedding model containing electrical-related information;

[0057] Specifically, the above steps 1 to 3 are performed by Figure 1 The file input and corresponding preprocessing and segmentation processes shown are used to update and adjust the text embedding model;

[0058] Step 4: Given an electrical document and a set of information to be extracted, input it as a question into Figure 1 The knowledge question answering system shown in the figure obtains a set of answers returned by the system through a series of processing;

[0059] Step 5, for a set of answers returned by the system, use rule-based wrong answer detection and uncertainty-based answer quality detection to screen out low-quality answers and reject their output, and then screen out high-quality answers and output corresponding text information.

[0060] In the processing of steps 1 to 5 above, in order to solve the problems existing in the prior art, improvements are mainly made to data annotation and to the control of the quality of answering questions of large language models. The implementation process of the specific improvements will be described in detail below with reference to the accompanying drawings.

[0061] 1. Improvements in data annotation

[0062] In the knowledge question answering system, the text embedding model usually needs to be trained based on a general training dataset. However, the existing public datasets lack electrical-related texts, which leads to poor performance of the model in distinguishing differences in electrical field texts. Manually labeling a large amount of electrical field data is not only time-consuming but also costly.

[0063] In the above Figure 1In the processing process shown, in order to solve this problem, a corresponding data set automatic generation module is specifically designed to efficiently generate a high-quality training data set in the electrical field, and then update the text embedding model based on the training data to make it contain electrical related information in the electrical field.

[0064] Specifically, the corresponding generated training data set can mainly include any one of the following two methods or both of them at the same time, where:

[0065] (1) Generate training datasets from public knowledge bases

[0066] Specifically, relevant electrical knowledge can be extracted from public knowledge bases (such as Baidu Encyclopedia, Wikipedia, etc.) and input into the GPT-3.5 model to generate a similar expression and an irrelevant expression as a training data set; that is, a training data set including electrical concepts and their corresponding similar semantic texts and irrelevant texts can be generated based on the obtained electrical concepts by constructing the prompt word Prompt and the language model ChatGPT;

[0067] The specific implementation process can be as follows Figure 2 As shown, the following steps are included:

[0068] (11) Extract electrical concepts from knowledge bases such as Baidu Encyclopedia or Wikipedia;

[0069] (12) constructing a prompt word Prompt based on the electrical concept and inputting it into ChatGPT to generate similar semantic text and irrelevant text corresponding to the electrical concept;

[0070] (13) The generated similar semantic texts and irrelevant texts and corresponding electrical concepts are used as training data sets.

[0071] (2) Generating training datasets from electrical text

[0072] Specifically, a question-answer pair can be generated from a given electrical text, and the generated answer can be used as a positive sample; then, the original model is used to search for text with low similarity as a negative sample, so as to construct a positive and negative sample pair as the training data set; that is, a training data set including positive and negative related texts and negative related texts of questions and corresponding question-answer pairs can be generated according to the obtained electrical text by constructing a prompt word Prompt and a language model ChatGPT, wherein the negative related text is obtained by searching based on a text embedding model;

[0073] The specific implementation process can be as follows Figure 3 As shown, the following steps are included:

[0074] (21) Obtaining electrical text, also known as electrical documentation;

[0075] (22) selecting a paragraph from a plurality of paragraphs included in the electrical text to construct a prompt word Prompt and inputting it into ChatGPT to generate a question-answer pair corresponding to the question;

[0076] The question-answer pair includes positively related text and negatively related text corresponding to the question; the positively related text is the answer to the question, that is, the correct answer to the question; the negatively related text refers to the text (or text block) retrieved from the text model with a low similarity to the question, that is, the text retrieved with a similarity lower than a predetermined standard is called negatively related text;

[0077] (23) The question and its corresponding positively related text and negatively related text are used as training data sets.

[0078] (II) Improvements in the quality of answering questions for large language models

[0079] When extracting key information from an electrical protocol, the quality of the answer is extremely high, and it is necessary to identify and reject wrong answers as much as possible. To this end, the present invention can adopt any one or both of the following rejection mechanisms in the specific implementation process, namely:

[0080] (1) Rule-based inspection and rejection mechanism:

[0081] That is, the answer is judged by using the format of the answer output when extracting the information in the electrical protocol as a predetermined rule, and when it does not meet the predetermined rule, the answer is rejected;

[0082] That is to say, the attribute information of the electrical cabinet usually has a fixed format, and different rules are set to check whether the format of the output answer meets the expected format; for example, if the format of a certain attribute information should be "attribute name: attribute value", the system will verify whether the format of the output attribute value corresponds to the attribute name; if it does not match, the answer will be rejected;

[0083] (2) Uncertainty-based inspection and refusal mechanism:

[0084] The uncertainty-based inspection and rejection mechanism refers to: obtaining the probability of all words in the answer output when extracting information from the electrical protocol, the probability of the most uncertain word in the answer, and the entropy of the words in the answer, and detecting and controlling the answer based on the probability of all words in the answer output when extracting information from the electrical protocol, the probability of the most uncertain word in the answer, and the entropy of the words in the answer; wherein the entropy of the words in the answer may include: the average entropy of all words in the answer and the entropy of the most uncertain word in the answer;

[0085] Specifically, each output of the large language model is based on the previous input to predict the next output word (token); the answer of the LLM (large language model) is defined as: containing J tokens, that is, J is the number of word tokens contained in the answer, p i is the probability of generating the i-th token in the answer, its uncertainty can be measured by the following two probability metrics:

[0086] (21) The probability of all words in the answer:

[0087]

[0088] (22) Probability of the least certain word in the answer:

[0089]

[0090] Entropy is used to measure the uncertainty or information content of a random variable; the entropy of a word output by a large language model can be defined as:

[0091] H i =∑ ω∈W p i (ω)logp i (ω);

[0092] Where W is the set of all possible words in the vocabulary, p i (ω) is the probability of generating word ω at the i-th token;

[0093] Based on the above formula, the uncertainty based on the entropy of the words in the answer can be measured in the following two ways:

[0094] (23) Average entropy of all words in the answer:

[0095]

[0096] (24) Entropy of the most uncertain word in the answer:

[0097]

[0098] For each LLM answer, the four uncertainties (1)-(4) above are calculated, and data is collected to train a simple predictor to predict the reliability of the answer. If the predictor determines that the answer is true, it outputs the answer; if the prediction is false, the answer is rejected.

[0099] Through the above two rejection mechanisms (1) and (2), the system can effectively identify and reject wrong or unreliable answers, ensuring the accuracy and reliability of the output answers.

[0100] The implementation of the embodiment of the present invention can utilize a large language model (such as GPT-3.5) to automatically annotate a data set, which greatly reduces time and labor costs through automated processes, thereby efficiently generating a large number of high-quality text data sets in the electrical field; thereafter, the model can be fine-tuned using the data set to enable it to have higher accuracy and effectiveness when processing electrical-related texts.

[0101] Furthermore, the implementation scheme provided in the above-mentioned embodiment of the present invention also introduces an answer quality detection mechanism, which evaluates the reliability of the output by combining manually defined rules and answer uncertainty indicators; specifically, the system ensures that the answer format meets expectations through rule checking, and uses uncertainty indicators to quantify the credibility of the answer, thereby filtering out unreliable answers and ensuring that the final output answer is accurate and credible.

[0102] In order to facilitate a further understanding of the specific implementation process of the present invention in a specific application example, the following will be combined with Figure 1 The specific implementation process of the embodiment of the present invention is described in detail. Figure 1 As shown, the implementation process of the embodiment of the present invention includes a processing process of fine-tuning the text embedding model, and a processing process of extracting electrical protocol information based on the fine-tuned text embedding model. The two processing processes will be described in detail below.

[0103] (i) The implementation process of fine-tuning the text embedding model may include the following steps:

[0104] Step 1: Use the data automatic annotation module based on Figure 2 The processing flow shown automatically generates training data in a required format, i.e., a training data set, based on existing electrical-related documents;

[0105] Step 2: Collect electrical concept-related knowledge and generate corresponding electrical texts, using the data automatic annotation module based on Figure 3 The processing flow shown generates corresponding training data, i.e., a training data set;

[0106] Step 3, using the training data set described in step 1 and step 2 as the annotation data to fine-tune the text embedding model, so that Figure 1 The text embedding model shown contains corresponding electrical related information, which facilitates the subsequent extraction and processing of electrical protocol information.

[0107] It should be noted that the execution order of the processing procedures shown in the above step 1 and step 2 is not limited in the specific implementation process.

[0108] (ii) The process of extracting electrical protocol information based on the fine-tuned text embedding model may include the following steps:

[0109] Step 1: during the process of extracting information from the electrical protocol, design questions and corresponding prompt words according to the information to be extracted;

[0110] Step 2: Input the document to be extracted, perform preliminary preprocessing on it, extract text, table and other information and divide them into blocks;

[0111] Specifically, it includes calculating word frequency and IDF (Inverse Document Frequency) values, and calculating cosine similarity based on the text embedding model; then, performing subsequent recalled text block processing;

[0112] Step 3: In the process of recalling text blocks, a recall method of multi-way retrieval plus BGE (retrieval ranking model) is used to retrieve the most similar TopK text blocks (i.e., the most similar K text blocks);

[0113] Step 4: Generate a prompt based on the selected document fragment (i.e., the most similar TopK text blocks obtained above) and the question asked;

[0114] Step 4: Input the generated prompt into the LLM (large language model) to obtain the answer content and the generation probability of each token;

[0115] Step 6: Perform rule-based wrong answer detection and uncertainty-based answer quality detection on the output results of LLM;

[0116] Specifically, the answer is detected based on predefined rules and uncertainty-based answer quality to see if it produces hallucinations, that is, inaccurate or unverified answers. And based on the detection results, it is determined whether to output the answer to ensure the accuracy of the output answer and to obtain high-quality answers after screening.

[0117] The embodiment of the present invention greatly reduces time and labor costs through automated processes, thereby efficiently generating a large number of high-quality text data sets in the electrical field, so as to adjust the text embedding model so that it has higher accuracy and effectiveness when processing electrical-related texts. At the same time, the embodiment of the present invention also introduces an answer quality detection mechanism, which controls the quality of the output answer based on rules and answer uncertainty indicators to ensure that the answer format meets expectations and can filter out unreliable answers, thereby ensuring that the final output answer is accurate and credible.

[0118] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A method for extracting electrical protocol information based on RAG, characterized in that: include: Generate a training data set based on known electrical texts and collected electrical concepts, and adjust the text embedding model based on the training data set; When extracting information from an electrical agreement, based on the adjusted text embedding model, a predetermined rule rejection mechanism, and a predetermined uncertainty detection mechanism, incorrect answers are identified and rejected, and answers that meet the requirements are screened as the extracted electrical agreement information.

2. The method according to claim 1, characterized in that The process of generating a training data set based on known electrical texts and electrical concepts includes: Generate a training data set of positively related text and negatively related text including questions and corresponding question-answer pairs by constructing a prompt word Prompt and a language model ChatGPT according to the obtained electronic text, wherein the negatively related text is obtained by searching based on a text embedding model; According to the obtained electrical concepts, a training data set including electrical concepts and their corresponding similar semantic texts and irrelevant texts is generated by constructing prompt words Prompt and language model ChatGPT.

3. The method according to claim 1, characterized in that The predetermined rules include: a preset format of an answer output when extracting information from an electrical protocol.

4. The method according to claim 1, 2 or 3, characterized in that: The predetermined uncertainty detection mechanism includes: The probability of all words in the answer, the probability of the most uncertain word in the answer, and the entropy of the words in the answer are output when extracting information from the electrical protocol, as well as pre-set rules for filtering answers based on the probability of all words in the answer, the probability of the most uncertain word in the answer, and the entropy of the words in the answer.

5. The method according to claim 4, characterized in that The likelihood of all words in the answer is determined by: Among them, J is the number of word tokens contained in the answer, and p i is the probability of generating the i-th token in the answer.

6. The method according to claim 4, characterized in that The likelihood of the least certain word in the answer is determined by: Among them, p i is the probability of generating the i-th token in the answer.

7. The method according to claim 4, characterized in that The entropy of the words in the answer includes: the average entropy of all words in the answer and the entropy of the most uncertain word in the answer, wherein: The average entropy of all words in the answer is: The entropy of the most uncertain word in the answer includes: The entropy definition of the word includes: H i =S ω∈W p i (ω)log p i (oh); W is the set of all possible words in the vocabulary, p i (ω) is the probability of generating word ω at the i-th word token.

Images