Method and system for generating text
The method employs a machine learning algorithm with neural networks to encode, vectorize, and decode text, filtering and ranking outputs for accurate and versatile text stylization, addressing the limitations of current technologies by ensuring high semantic accuracy and adaptability to various styles.
Patent Information
- Application Number
- PCT/RU2023/000178
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-02
- Filing Date
- 2023-06-15
- Publication Date
- 2024-12-05
AI Technical Summary
Current text stylization technologies face challenges in generating stylized text with high accuracy, preserving semantic content, and adapting to multiple styles, often resulting in distortion or addition of new facts, and require extensive training data and computational resources.
A method using a machine learning algorithm with a neural network to encode, vectorize, and decode source text, filtering and ranking stylized outputs based on semantic proximity and linguistic acceptability, allowing for multiple stylized texts to be generated from a single source while ensuring accuracy and versatility.
The solution significantly enhances semantic accuracy in text stylization, reduces computational complexity, and enables generation of multiple styles from a single source text, ensuring preservation of original content and adaptability to different styles.
Smart Images

Figure RU2023000178_05122024_PF_FP_ABST
Abstract
Description
METHOD AND SYSTEM FOR TEXT GENERATION AREA OF TECHNOLOGY
[0001] The claimed technical solution generally relates to the field of computer technology, and in particular to a method and system for generating text. LEVEL OF TECHNOLOGY
[0002] As a result of the functioning of the language, its varieties have developed, belonging to a certain style of text presentation, which is characterized by certain features, linguistic means, genres, etc. Thus, when publishing a text in a scientific journal, such a text will have features of a scientific style, while in informal communication the text will have features of a colloquial style, for example, informal addresses to "you", simplicity of sentence construction, use of slang, etc.
[0003] Currently, with the development of information technology, speech style transfer technologies in the field of natural language processing (NLP) have been actively developed and today they are trying to integrate it into a variety of areas. Automation of the process or part of the process of stylizing text in a certain style can significantly increase efficiency in areas such as journalism, in publishing houses, for example, text editors, content creation for media platforms and virtual assistants, etc.
[0004] However, despite the demand for this technology, there are a number of difficulties that do not allow, for example, to generate stylized text with high accuracy. Thus, the problems of text stylization are ensuring the safety of the original information, ensuring the absence of new facts in the generated texts, preserving the semantic load of the original text, etc. In addition, one of the important problems is also the possibility of ensuring the universality of the technology, allowing to generate text not only in one style, but also to provide the ability to stylize the original text in several styles, depending on the scope of application. Therefore, the creation of an effective and accurate method for automatic text generation in specified styles is a significant task.
[0005] Thus, a method for transferring text style is known from the state of the art, disclosed in the source [1]. The said method provides the ability to generate stylized text from the source text by solving the machine translation problem. In this case, the text style serves as the "language" into which the source text needs to be translated.
[0006] The disadvantages of this solution include the high complexity of implementation and the narrow focus of this solution, due to the huge set of required training data, and the impossibility of adaptation to different styles due to the peculiarities of language translation technology. In addition, this solution also does not provide high accuracy, since in the process of such "translation" the meaning of the original phrase may be lost due to the change of all the words of the original text.
[0007] The prior art also includes a method for providing logical answers that imitate the user's speech style, disclosed in Russian Federation Patent No. RU 2693332 C1 (LIMITED LIABILITY COMPANY "YANDEX"), published on 02.07.201. This method provides the ability to select a contextual answer to a question, depending on the context of the question, by analyzing the vector representation of the contextual question and searching for the closest answer from a set of answers in the database.
[0008] The disadvantages of this solution are the impossibility of generating stylized text based on the original text, high computing power costs and a large amount of memory required to form a database (DB), limited stylization by DB patterns, low accuracy of generating stylized text due to the selection of stylistic responses created in advance and saved in the DB.
[0009] Common shortcomings of existing solutions are the lack of an effective method for generating stylized text with high accuracy, ensuring the preservation of the original information and the absence of new facts in the generated texts. Also, the specified method should ensure the preservation of the semantic load of the original text. In addition, the specified method should ensure the universality of the text stylization technology, allowing to generate text not only in one style, but also to provide the ability to stylize the original text in several styles, depending on the scope of application. DISCLOSURE OF INVENTION
[0010] The declared technical solution proposes a new approach to text generation. This solution uses a machine learning algorithm that allows generating stylized text from the original text with high accuracy, ensuring the semantic proximity of the original and stylized text and eliminating the distortion of the stylized text by new facts.
[0011] Thus, the technical problem of ensuring the possibility of generating stylized text is solved.
[0012] The technical result achieved by solving this problem is an increase in the semantic accuracy of generating stylized text from the source text.
[0013] An additional technical result that appears when solving the above problem is the ability to generate multiple stylized texts from one source.
[0014] The specified technical results are achieved by implementing a method for generating text, performed by at least one computing device, and comprising the following stages: a) receiving input data containing a source text in a natural language and a target style of the generated text; b) encoding the source text, wherein during encoding at least tokenization of the text data is performed; c) vectorizing the tokens obtained in stage b); d) processing the vector representations of the tokens of the source text obtained in stage c) using a machine learning model based on a neural network trained on stylized, in accordance with a given target style, text data, during which an array of vectorized stylized texts is formed;e) performing decoding of each vectorized stylized text from the array obtained in step d), wherein during decoding at least the transformation of the vectorized stylized text into tokens and detokenization are performed; f) filter the array of stylized texts obtained in step e); g) rank the filtered stylized texts based on the pairwise distance between the original text and each of the possible stylized texts and the linguistic acceptability of each of the possible stylized texts; h) generate stylized text based on the data obtained in step g).
[0015] In one of the particular implementation options of the method, filtering of an array of stylized texts is performed using regular expressions and a morphological analyzer.
[0016] In another particular embodiment of the method, filtering of the array of stylized texts is performed based on the coincidence of proper names in the original text and each stylized text from the array.
[0017] In another particular embodiment of the method, the coincidence of proper names is determined using the number of unchanged named entities contained in the original text and the stylized text.
[0018] In another particular embodiment of the method, the number of unchanged named entities is determined based on the recognition of named entities in the source text and each stylized text from the array.
[0019] In another particular embodiment, the method additionally comprises a step of checking for the presence of text stylization.
[0020] In another particular embodiment of the method, the verification of the presence of text stylization is carried out on the basis of determining whether the replacement of target style indicators has occurred.
[0021] In another particular embodiment of the method, ranking of the filtered stylized texts is performed on the basis of an average value determined from the pairwise distance between the original text and each of the possible stylized texts and the linguistic acceptability of each of the possible stylized texts.
[0022] In addition, the stated technical results are achieved through a system of automatic generation of text in a given speech style in systems containing: at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of the text generation method. BRIEF DESCRIPTION OF DRAWINGS
[0023] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings.
[0024] Fig. 1 illustrates a block diagram of the general appearance of the claimed system.
[0025] Fig. 2 illustrates a block diagram of the implementation of the claimed method.
[0026] Fig. 3 illustrates an example of the general appearance of a computing device that ensures the implementation of the claimed solution. IMPLEMENTATION OF THE INVENTION
[0027] Below we will describe the concepts and terms necessary for understanding this technical solution.
[0028] A model in machine learning (ML) is a set of artificial intelligence methods, the characteristic feature of which is not a direct solution to a problem, but learning in the process of applying solutions to many similar problems.
[0029] Named-entity recognition (NER) is a subtask of information extraction that aims to find and classify occurrences of named entities in unstructured text into predefined categories, such as proper names, character names, organizations, locations, monetary values, percentages, etc.
[0030] Word embeddings (WEM) is a general term for various approaches to language modeling and representation learning in natural language processing that aim to match words (and possibly phrases) from some dictionary of vectors in the n-dimensional real space R_n.
[0031] Tokenization is the process of breaking text into text units or tokens (most often, these units are words, but they can also be letters, parts of a sentence, word combinations, etc.).
[0032] A language model is a probability distribution over a set of vocabulary sequences. In this patent, the term “language model” is used to describe neural network language models that are designed to model language by estimating the probability of a particular sequence of symbols.
[0033] Linguistic acceptability is a term coined by linguist Noam Chomsky. Unlike grammatical correctness, or grammaticality, which refers to the structure of language, acceptability evaluates how natural a sentence is from the point of view of a native speaker. Thus, a grammatical sentence may be unacceptable (a well-known example by L. V. Shcherba: "Glokaya kuzdra shteko budlanula bokra i kurdyachit bokryonka"), but an acceptable sentence must be grammatical. That is, linguistic acceptability allows one to determine a text sentence on a scale from possible to impossible from the point of view of a native speaker.
[0034] The claimed technical solution offers a new approach that ensures an increase in the semantic accuracy of stylized text generation, consisting in ensuring the preservation of the semantic content in the stylized text, as well as the exclusion of supplementing the stylized text with new facts and logical units of language. One of the features of the claimed technical solution is the ability to generate multiple stylized texts from a single source text, which also ensures automation of the process of generating stylized texts and significantly reduces the time of text stylization compared to manual generation of each individual stylized text. In addition, another advantage achieved when implementing the claimed technical solution is the ability to check and select the generated text using linguistic acceptability, which, as a result, ensures an increase in the accuracy of text generation due to the exclusion of "unnatural" texts.
[0035] The claimed technical solution can be implemented on a computer, in the form of an automated information system (AIS) or a machine-readable medium containing instructions for performing the above-mentioned method.
[0036] The technical solution can also be implemented as a distributed computer system or computing device.
[0037] In this solution, the term system refers to a computer system, a computer (electronic computer), a numerical control (NC) system, control), PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly defined sequence of computing operations (actions, instructions).
[0038] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs) /
[0039] The command processing unit reads and executes machine instructions (programs) from one or more data storage devices, such as random access memory (RAM) and / or read-only memory (ROM). ROM may include, but is not limited to, hard disk drives (HDD), flash memory, solid-state drives (SSD), optical storage media (CD, DVD, BD, MD, etc.), etc.
[0040] A program is a sequence of instructions intended for execution by a computer's control unit or command processing device.
[0041] The term "instructions" as used in this application may refer generally to software instructions or software commands that are written in a given programming language to implement a specific function, such as, for example, encoding and decoding texts, filtering, ranking, translating texts into a dialog system, etc. The instructions may be implemented in a variety of ways, including, for example, object-oriented methods. For example, the instructions may be implemented using the Python programming language, C++, Java, Python, various libraries (e.g., MFC; Microsoft Foundation Classes), etc. The instructions that implement the processes described in this solution may be transmitted via both wired and wireless data transmission channels, such as Wi-Fi, Bluetooth, USB, WLAN, LAN, etc.
[0042] Fig. 1 shows a general view of the text generation system 100. The system 100 includes basic functional elements such as: encoding / decoding module 101, style transfer module 102, stylized text filtering module 103, stylized text ranking module 104. The elements of the system 100 are disclosed in more detail in Fig. 3.
[0043] The text generation system 100 may represent or be part of a system, such as a server, a computer, a cloud module, etc., designed to generate text based on a source text. Thus, a text generation system may be understood as any system designed to process text. Accordingly, a text generation system may be any system for processing text data for the purpose of generating new text data.
[0044] Thus, in one particular embodiment, the text generation system 100 may interact with and / or be part of a dialogue system. The dialogue system, in turn, may represent various solutions, such as voice assistants, chat bots, robotic call centers, and other technologies that embody an automated process of communication with a user, such as user 110. It is worth noting that the generation system 100 should be understood as any automated human-machine system operating in a dialogue mode, in which it responds to each user command and, as needed, requests information from the user.
[0045] In another particular embodiment, the system 100 may be part of and / or interact with a text processing system, such as a system for creating, editing, saving and printing a document containing texts. Thus, the said system 100 may receive, via an input / output interface, such as a GUI, etc., initial text data that must be processed and, in accordance with the steps of the method 200, generate text based on the initial data. Thus, in another particular embodiment, the initial data may be voice data. To process the said data, speech is converted into text and the data is further processed. Speech-to-text conversion technologies are widely known from the state of the art.
[0046] In addition, the text generation system 100 may also be associated with digital assistants (virtual digital assistants) and intended to generate text for digital assistants taking into account the speech style of each digital assistant. Thus, digital assistants may be systems for automating interaction with the user, implemented on the basis of artificial intelligence in a dialog format (chat bot, skills for a voice assistant, etc.). Thus, in one particular embodiment, the digital assistant may be a computer system that simulates a conversation with users in a dialog format.
[0047] In this solution, text stylization is understood as text generation by transforming the accepted source text into a text that has stylistic speech features. Thus, stylistic features may be the emotional coloring of the text (cheerful, sad, etc.). In another particular embodiment, stylistic speech features may be the condition and goals of communication in some area of public activity, for example, official business activity, journalistic activity, conversational, artistic, etc. It is worth noting that text stylization may also be imparting characteristic features to the text that are inherent in the communication features of individual individuals, characters, literary heroes, etc., without limitation.Thus, in another particular embodiment, text stylization may be the transformation of the original text into a stylized one, in accordance with the specified communication style of a specific digital assistant, which has conversational features of style, for example, the use of certain pronouns that emphasize the informal / formal communication style (“You” and “You”), gender, number, etc.
[0048] The encoding / decoding module 101 can be implemented on the basis of at least one computing device equipped with the corresponding software and include a set of models for tokenization and detokenization of text, vectorization of tokenized text and transformation of tokens into text, for example, one or more machine learning models for transformation of text information into vector form, for example, BERT, ELMo, ULMFit, XLNet, RoBerta, RuGPT3 and others. In one particular embodiment, the module 101 can be implemented on the basis of the system 300, which is disclosed in more detail in Fig. 3. It is worth noting that a certain method of tokenization and vectorization depends on the selected language model on the basis of which the module 102 is implemented.For example, when using the RuGPT3 model, tokenization is performed using the BPE (Byte Pair Encoding) method, and subsequent vectorization is performed by replacing each token with its index in the language model dictionary compiled at the initial model training stage. In addition, in another particular embodiment, word tokenization can be used as a tokenization method. An example of tokenization by words and encoding words by indexes in a dictionary: 'Mom was washing the frame' ['Mom', 'washed', 'frame'] [235, 376, 1056]
[0049] Module 102 can be implemented based on at least one neural network, previously trained on specific sets of stylized texts in accordance with specified styles. For example, a generative language model such as RuGPT3, XLNet, etc. can be used as a machine learning model implementing the function of generating stylized texts in accordance with a specified style. In one particular embodiment, when implementing the claimed solution, the ML was the Russian-language generative language model RuGPT3 -Large. The model was trained on sources from different domains: Wikipedia, books, news, Russian Common Crawl, etc. The model was trained for 14 days on 128 GPUs with a context window of 1024 and an additional few days on 16 GPUs with a context of 2048. The final model has a perplexity of 13.8 on the test data set.At this stage of training, the result of training the language model was the ability to predict the probability of the next token based on the previous initial fragment of text. So, if during the training process the model often encountered a certain phrase in the training data, then when predicting the next token after the one known from the phrase, the model will predict with a high probability the token from the phrase in the training data set.
[0050] Next, to perform the actual process of generating stylized text from the source text, the trained model was further trained. The fine-tune procedure was used to further train the model. At this stage, the weights of the trained model were adjusted in accordance with the task being solved. Thus, when an initial sentence appears in the model, due to the changed weight coefficients, the most probable sentence for the model will be a paraphrase of this sentence in a certain style, i.e., the probability of continuing the text fragment in the format in which the stylized text should be generated is increased. When further training the model, datasets with various stylized texts in different speech styles were used. Thus, in one particular embodiment, the problem of stylizing the text in accordance with one of the specified speech styles was solved.The initial data for additional training included sets of texts in three specified speech styles: scientific (5500 texts), conversational (6285 texts), and business (7578 texts).
[0051] To form a dataset (training dataset) in a format suitable for further training of the model, consisting of pairs "original text" - "text in a given speech style", a paraphrase was applied to the stylized texts, for example, a paraphrase based on the generative model RuT5. The paraphrase model is known from the state of the art and is disclosed, for example, in a source available at the link on the Internet: https: / / huggingface.co / cointegrated / rut5-base-paraphraser. For each stylized text, 10 initial paraphrase variants were generated, from which 2 closest variants were then selected based on a semantic metric, for example, the LabSe metric. This metric evaluates the cosine similarity between the vector representations of sentences obtained using the model, which corresponds to semantic closeness.Thus, the final datasets for further training RuGPT3 for the task of style transfer in the specified speech styles contained 5500 (scientific), 6285 (colloquial), 7578 (business) pairs of sentences ("original text" - "text in the specified style"). In addition, a tag characterizing a specific speech style (scientific, colloquial, business) was also added to the data to ensure the ability to determine in which style the stylized text would be generated. The model was further trained for 5 epochs on the obtained data.
[0052] The obtained model was evaluated for the task of generating text in a given speech style on a test set separately for each speech style. The test data set consisted of 1100 texts for each speech style. The following metrics were used to evaluate the quality: 1) BLEU (Papineni, K.; Roukos, S.; Ward, T; Zhu, WJ (2002). BLEU: a method for automatic evaluation of machine translation (PDF). ACL-2002: 40th Annual meeting of the Association for Computational Linguistics, pp. 311-318. CiteSeerX 10.1.1.19.9416.). The BLEU algorithm compares phrases in a double translation with phrases it finds in the reference version and weights the number of matches. These matches are independent of position. A higher degree of coincidence indicates a higher degree of similarity with the reference translation and a higher score. Clarity and grammar are not taken into account. 2) The number of total N-Grams, where N-Grams were taken as sequences of words of length from 3 to 8.3) Levenshtein - Levenshtein distance (V. I. Levenshtein. Binary codes with correction of deletions, insertions and substitutions of symbols. Reports of the USSR Academies of Sciences, 1965. 163.4:845-848.) (editorial distance, editing distance). The specified metric, measuring by modulus. the difference between two sequences of characters. It is defined as the minimum number of single-character operations (namely, insertion, deletion, replacement) required to transform one sequence of characters into another. 4) Jaccard index - Jaccard index (Jaccard P. Distribution of alpine flore dans le Bassin des Dranses et dans quelques regions voisines / / Bull. Soc. Vaudoise sci. Natur. 1901. V. 37. Bd. 140. S. 241-272.), which was calculated as the number of common tokens in the stylized and original texts, divided by the union of the tokens in these two text fragments.
[0053] The results of the model estimation for each of the three styles are presented in Table 1. Table 1
[0054] Table 1 shows that the BLEU score, number of common n-grams, and Jaccard index are all above 95%. The Levenshtein distance for all styles is also less than 2 on average, indicating that the generated sentences differ from the sample answers by no more than 2 characters on average.
[0055] The module 103 may be implemented on the basis of at least one computing device and include a morphological analyzer, for example, a Unigram analyzer, an N-gram analyzer, an analyzer based on regular expressions, etc. Also, the module 103 may contain a neural network trained to solve the NER (named entity recognition) task, for example, Slovnet BERT NER, DeepPavlov BERT NER, etc., without limitation. In one particular embodiment, the module 103 may contain a heavy model with the BERT architecture, and be trained on a small manually annotated dataset.
[0056] The module for ranking 104 stylized texts can be implemented on the basis of at least one computing device equipped with the appropriate software for calculating the Levenshtein distance and linguistic acceptability. Thus, the said module 104 is designed with the possibility of implementing an algorithm for calculating pairwise distances between the original text and stylized texts and linguistic acceptability.
[0057] It will be apparent to one skilled in the art that, although the modules described above are presented as separate devices, said modules may also be combined into a single device, such as system 300.
[0058] Fig. 2 shows a block diagram of a method 200 for generating text, which is disclosed step by step in more detail below. Said method 200 consists of performing steps aimed at processing various digital data. The processing is typically performed by a system, such as system 100, which may also represent, for example, a server, a computer, a mobile device, a computing device, etc.
[0059] At step 210, the system 100 receives input data containing a source text in a natural language and a target style. Thus, the input data can be received from a dialog system, a text processing system and / or directly into the system 100, via data transmission channels, such as the Internet. The source text can represent, for example, a dialog replica, such as an answer to a user question, a question to the user, a sentence subject to stylization, etc. The target style can represent certain characteristic features of a speech style, for example, a conversational style of communication, which is characterized, for example, by the presence of informal appeals to the user, a business style of communication, which is characterized by formal appeals, etc. It is obvious to a person skilled in the art that stylization can be used both as replicas of digital assistants and in the process of processing and / or paraphrasing text.Also, in one particular embodiment, the target style may indicate the emotional coloring of the text, for example, sad, cheerful, neutral, etc. The target style may be determined by the settings of the dialogue system. Thus, the dialogue system may contain a set of digital assistants, each of which has its own communication style. When starting a dialogue, the user can select a specific assistant in the system settings. The specified data on the selected assistant can also be transmitted to the system 100. In addition, in another particular embodiment, the selection of the target style can be carried out for the text paraphrasing system.
[0060] At step 220, the source text is encoded, wherein during the encoding, at least tokenization of the text data is performed. The specified step 220 can be performed by module 101. The input text can be divided into tokens. In this solution, a token is understood to be a sequence of characters in the text that is significant for analysis. In another particular embodiment, text tokenization can be performed using the BPE (Byte Pair encoding) algorithm. In another particular embodiment, tokenization can be split into words by the space between words. Then, a dictionary of tokens of a fixed size (for example, 30,000 tokens) is compiled, where each token is associated with its index in the dictionary. Example of tokenization for words: <request> ']
[0061] At the specified step 230, vectorization of tokenized texts is performed. As mentioned above, the tokenization method depends on the language model used in module 102 at step 240. For example, when using the RuGPT3 language model, each token is assigned its index in the dictionary. Thus, the tokenized text fragment (list of tokens) after vectorization is mapped to a vector of token data indices in the dictionary. Example of vectorization when tokenizing by words:
[0062] Method 100 then proceeds to step 240.
[0063] At step 240, the vector representations of the source text tokens are processed using module 103, during which an array of vectorized stylized texts is formed. As mentioned above, the neural network-based machine learning model was further trained on stylized texts in accordance with the specified target style. At step 240, a vector representation of the source text is input to the model. At the output, the model generates several variants of stylized texts (candidates) in the form of vector representations. These vector representations are saved in an array of stylized texts. The number of candidates generated by the model depends on the sampling methods used. In one particular embodiment, the following sampling criteria were used for the model: top P sampling (top_p=0.92), top K sampling (top_k=50), temperature sampling (temperature=0.85). These sampling methods are disclosed in more detail in the source, found in Internet at the link: https: / / towardsdatascience.com / how-to-sample-from-language-models-682bceb97277 . The general principle of the model is the ability to predict the probability of the next token in a certain context. Thus, at the first stage of training, the language model is able to predict the probability of the next token based on the previous initial fragment of text. To implement the ability of the specified model to generate stylized texts, its weights are then changed so that the probabilities for the next token that the model predicts correspond to the current text stylization task. The specified change in weights is carried out based on the training data set. The retrained model is then able to generate from the original text a set of stylized texts with varying degrees of probability of tokens in this text.Since the generated set of stylized texts can be very large, and the probability distribution in some of them will be too small, such candidates (generated stylized texts) can be cut off based on the sampling criteria specified above. As a result, an array of stylized texts is formed at the output of the model.
[0064] It is worth noting that providing the ability to generate an array of stylized texts, rather than a single stylized text, increases the accuracy of text stylization, since it provides the ability to further check and process all variants of the stylized text to select the most semantically close to the original text from the array, as well as the linguistically acceptable text. In addition, generating an array of stylized texts allows you to exclude the addition of new facts and / or incorrect changes to certain words based on further processing of such texts.
[0065] At step 250, the generated array of vectorized stylized texts is fed to module 101. At said step 250, each vectorized stylized text from the array is decoded, and during decoding, at least the vectorized stylized text is converted into tokens and detokenized. For example, during said process, each fixed-length vector is matched to a token by the dictionary index based on its dimensionality, which allows each vector to be represented as a token. The detokenization process is the inverse process to tokenization and consists of combining tokens into text. As a result, After completing this step 250, the array of vectorized stylized texts is transformed into an array of stylized texts in natural language. Example: Original text [ 'Here's what I found on your request] An array of stylized texts: ['Here's what I found for your request', 'Here's what I found for your request', 'Here's what I found for your request', 'What I found for your request'...]
[0066] At step 260, the array of stylized texts is filtered. At the specified step 260, the specified array is filtered using module 103. Thus, at this stage, those candidates that do not meet the specified characteristic stylistic features are excluded. Thus, for example, when generating a stylized text in a business style, at least the following criterion for addressing (formally addressed as "you", not as "you") will be used. The specified filtering also increases the accuracy of text stylization. Regular expressions written on the basis of the te library and the MorphAnalyzer morphological analyzer from the pymorphy2 library are used to check for correctness. At the output of this module, we have a filtered list of possible candidates that have passed the correctness check. Example: [''Here's what I found for the request 'What I found for your request'...].
[0067] In addition, in one particular embodiment, the filtering of the array of stylized texts is performed based on the coincidence of named entities (e.g., proper names) in the original text and each stylized text from the array. For this purpose, an algorithm for recognizing named entities is performed in the original text and each of the stylized texts, for example, using module 103, and a comparison is made of the number of unchanged named entities contained in the original text and the stylized text. As indicated above, at step 240, an array of stylized texts is generated. In order to increase the semantic accuracy between the stylized and original text, the stylized text should not distort the facts and should not be supplemented with new logical connections.In turn, the specified array of stylized texts may contain relevant stylized texts that satisfy the characteristic stylistic features, but such texts will be incorrect, for example, due to a change in the facts of such text. Example: Original text: I think you should read Alexander Sergeevich Pushkin's work "Eugene Onegin" Incorrect stylized text: I think you should read the work of Alexander Vasilyevich Pushkin "Eugene Onegin"
[0068] As can be seen from the example, the generated text satisfies the stylistic features of the official style (addressing as “You”), i.e. it is relevant, however, it changes the facts of the text (the name of the author of the work).
[0069] To solve this problem, an approach was proposed that consists of comparing named entities in the source text and each stylized text from the set. For this purpose, in the first step, the named entities (e.g., names, toponyms, and organizations) are recognized in the source text and stored in memory, for example, in the memory of the system 100, as a data file. In the next step, the named entities are recognized in each stylized text from the array. After this, the unchanged entities are compared between the source and stylized texts. Accordingly, candidates that differ from the source text in named entities are discarded, which additionally increases the semantic accuracy of the text stylization by eliminating changes / additions of facts to the stylized text.
[0070] Thus, at step 260, the array of stylized texts is filtered.
[0071] At step 270, the filtered stylized texts are ranked and the best stylized text is selected, where the selection of the best text is based on two factors: a) the pairwise distance between the original text and each of the possible stylized texts; b) an assessment of the linguistic acceptability of the candidates from the point of view of the Russian language.
[0072] As mentioned above, linguistic acceptability characterizes the meanings assigned to a particular text, reflecting the possibility or impossibility of the text's existence from the point of view of a native speaker.
[0073] So, the following requirements are imposed on linguistic acceptability. The text must be meaningful - an example that makes sense, semantically understandable to the listener. Consistent, without logical errors. The text must be grammatical - an example constructed correctly, according to the rules of grammar. The rules of grammar may include, for example, word order, cases, and everything that is syntax. Accordingly, acceptable text must be both meaningful and grammatical.
[0074] Let's consider the first criterion for ranking texts.
[0075] The distance between a candidate and the original text is estimated by comparing the symbolic proximity to the original text using the Levenshtein distance. To do this, the pairwise distances between the original neutral phrase or text and each of the possible candidates are calculated. The Levenshtein distance is calculated using the distance function from the Levenshtein library. The resulting set of distances is then normalized using the minimax transformation. (https: / / www.bigdataschool.ru / blog / %D0%BD%D0%BE%D1 %80%D0%BC%D0%B0% D0%BB%D0%B8%D0%B7%D0%B0%Dl%86%D0%B8%Dl%8F%D0%BC%D0%B5% D 1 %82%D%BE%D0%B4%D 1 %8B%D 1 %81 %D 1 %80%D0%B5%D0%B4%D 1 %81 %D l%82%D0%B2%D0%B0-data-preparation.html) and is reduced to the range [0, 1].
[0076] To assess the linguistic acceptability values of candidates from the point of view of the Russian language, the ruRoBERTa-large-rucola model (https: / / huggingface.co / RussianNLP / ruRoBERTa-large-rucola) is used. This model is a ruRoBERTa-large model that was fine-tuned for the task of assessing the grammatical acceptability of text on the RuCoLa dataset (https: / / github.com / RussianNLP / RuCoLA). This model determines the value of linguistic acceptability from 0 to 1 for each sentence (stylized text), from the point of view of the Russian language (0 - unacceptable, 1 - absolutely acceptable). To further train the model, a dataset was used containing marked examples that are acceptable and unacceptable in accordance with the criteria of linguistic acceptability.Using this model, for each candidate from the set, its linguistic acceptability (grammatical acceptability) is calculated from the point of view of the Russian language and the score inverse to grammatical acceptability, which is calculated as unacceptability = 1 - acceptability. This value reflects the degree of linguistic unacceptability and also varies from 0 to 1 (the lower, the better). It is worth noting that the value of linguistic acceptability essentially reflects the probability of the correctness of the model's prediction.
[0077] Thus, for each candidate from the set, two scores from 0 to 1 are obtained. The first reflects the closeness to the original sentence, the second - the linguistic unacceptability of the candidate. For each candidate, the average of these two assessments is taken as the final assessment, and the candidate with the minimum value of the final assessment is selected as the best. As a result, we obtain a stylized text, on the one hand close to the original text fragment, on the other hand with high linguistic acceptability, which accordingly increases the accuracy of the entire stylization algorithm and reduces the likelihood of adding new facts and generating linguistically unacceptable texts.
[0078] Let's look at several texts as an example. Example: I found this in the application - unacceptable
[0079] According to the linguistic acceptability assessment, the value will be 0.27, which characterizes the unacceptability of this sentence.
[0080] As noted above, introducing an additional ranking criterion increases the accuracy of the generated text styling and also reduces the likelihood of adding new facts.
[0081] Let's consider an example of ranking candidates in a business style for the text "Yesterday at two the store was closed". The following texts were received as candidates: 1) The store was closed at two o'clock yesterday. 2) The store was closed at two o'clock yesterday afternoon. 3) The supermarket was closed at two o'clock yesterday afternoon.
[0082] Let us define the criterion of semantic similarity (the Levenshtein distance normalized using the minimax transformation) for each of the given examples. For the first text, the Levenshtein distance will be - 10, for the second - 12, for the third - 18. Applying the minimax transformation to these values, we obtain a set of values: 0, 0.25 and 1, respectively.
[0083] Next, we determine the linguistic acceptability values for each candidate, for example, using the linguistic acceptability model described above.
[0084] For the first candidate, according to the prediction of the linguistic acceptability model, the value will be - acceptability - 0.15 - nonacceptability - 1 - 0.15 = 0.85.
[0085] In this case, for the second and third texts the value of linguistic acceptability will be 0.87 and 0.84, and non-acceptability - 1 - 0.87 = 0.13 and 1 - 0.84 = 0.16.
[0086] As a result, we obtain that the final score for the first candidate is equal to (0 + 0.85) / 2 = 0.425, for the second - (0.25 + 0.13) / 2 = 0.19, for the third - (1 + 0.16) / 2 = 0.58 and as a result, the second candidate with the lowest value of the final score will be selected, which reflects the high linguistic acceptability of this candidate and high semantic closeness to the original text.
[0087] As can be seen from the obtained values, the normalized Levenshtein distance has low values for both the first and second candidates, while for the third candidate the value is higher due to the replacement of the word "magaz" with "supermarket", which is not necessary. At the same time, the linguistic unacceptability of the first candidate clearly shows the unacceptability of this candidate.
[0088] Thus, when using only the semantic similarity criterion, the ranked candidates, and therefore the final generated text, will have lower accuracy.
[0089] Therefore, ranking candidates using the average of two criteria additionally improves accuracy and eliminates incorrect candidates.
[0090] In addition, in one particular embodiment, the ranking of candidates may be performed according to a threshold value for the linguistic acceptability criterion. For example, the threshold of linguistic acceptability may be 0.5. Therefore, when ranking, all candidates with a value below the threshold value will be automatically discarded and will not be taken into account in the ranking. In the event that all candidates do not satisfy the threshold value, then the candidate with the best value among those having it may be selected.
[0091] In addition, in one particular embodiment of the claimed solution, the text obtained as a result of ranking can be additionally checked for the presence of stylization. The presence of text stylization is checked based on determining whether the target style indicators have been replaced. Thus, for example, the system 100 can additionally contain a stylization checking module implemented on the basis of a computing device. The specified module is designed with the ability to check for a pair (original text, stylized text) whether the stylization replaces the target style indicators, for example, the number in the address Ty / Vy (required for conversational / business style) compared to the original phrase (there are many neutral phrases for which replacement is not required). This module is designed with the ability to replace only if at least one change occurs. Thus, if a replacement occurs, then at the output we get the stylized text, and if no replacement occurs, then the original, which is neutral and suitable for any style.
[0092] At step 280, stylized text is generated based on the data obtained at step 270.
[0093] Thus, at the specified stage 280, the final stylized text is generated. The stylized text can be generated based on the ranking results obtained at stage 270. Thus, the system 100 can select the best text from the set of ranked candidates and generate the selected text in the input / output interface.
[0094] Thus, for example, when stylizing a text fragment, the generated stylized text can be stored in the memory of the system 100 as a file and sent to the dialog system or to the input / output interface, for example, via a data transmission channel for subsequent display to the user, for example, using the input / output interface of the dialog system or the input / output interface of the system 100.
[0095] Thus, the above materials described a system and method for generating text stylized in a given speech style, ensuring high semantic accuracy of generating stylized text.
[0096] In addition, it is worth noting that the implementation of the claimed solution also ensures the universality of text stylization, allowing to generate text not only in one style, but also to provide the ability to stylize the original text in several styles, depending on the scope of application. This feature eliminates the need for separate formation of a unique stylized text for each style. Thanks to the use of the claimed solution, it is possible to submit one original text and generate different stylized texts on its basis.
[0097] Now let's look at one example of the implementation of the declared technical solution.
[0098] One of the possible applications of system 100 is to stylize an initially neutral response to a given speech style (scientific, conversational, business, etc.). At the user's request, the dialog system generates an initially neutral text (initial text). The specified method 200 provides the ability to generate from the specified neutral text, a stylized text in accordance with the specified style (for example, to turn the generated text into a more scientific / business one). One of the advantages of this approach is the flexibility in choosing a style, the generation of a text response is a computationally complex operation, for which it is necessary to train a large model. The declared solution, in turn, allows stylizing the generated initial response using system 100 to suit different styles.
[0099] Fig. 3 shows an example of a general view of a computing system (300) that ensures the implementation of the claimed method or is part of a computer system, for example, modules 101-103, a server, a personal computer, part of a computing cluster that processes the necessary data for implementing the claimed technical solution.
[0100] In general, the system 300 comprises components such as: one or more processors 301, at least one memory 302, data storage means 303, input / output interfaces 304, I / O means 305, network interaction means 306, which are connected via a universal bus.
[0101] The processor 301 performs the basic computing operations necessary for processing the data when performing the method 200. The processor 301 executes the necessary machine-readable instructions contained in the RAM 302.
[0102] Memory 302 is usually implemented as RAM and contains the necessary software logic to provide the required functionality.
[0103] Data storage means 303 can be implemented in the form of HDD, SSD disks, RAID array, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. Means 303 allow long-term storage of various types of information, such as generated stylized replicas, user identifiers, digital assistant identifiers, etc.
[0104] To organize the operation of the components of the 300 system and to organize the operation of external connected devices, various types of I / O interfaces are used. 304. The choice of appropriate interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.
[0105] The choice of interfaces 304 depends on the specific implementation of system 300, which can be implemented on the basis of a wide range of devices, for example, a personal computer, mainframe, laptop, server cluster, thin client, smartphone, server, etc.
[0106] The following may be used as I / O data means 305: keyboard, joystick, display (touch display), monitor, touch display, touchpad, mouse, light pen, stylus, touch panel, trackball, speakers, microphone, augmented reality means, optical sensors, tablet, light indicators, projector, camera, biometric identification means (retina scanner, fingerprint scanner, voice recognition module), etc.
[0107] Network interaction means 306 are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, a WLAN / Wi-Fi module, a Bluetooth module, a BLE module, an NFC module, IrDa, an RFID module, a GSM modem, etc. With the help of means 305, data exchange is organized between, for example, a system 300, presented in the form of a server, and a user's computing device, on which the received data can be displayed via a wired or wireless data transmission channel, for example, a WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM.
[0108] The specific selection of elements of system 300 for the implementation of various software and hardware architectural solutions can vary while maintaining the required functionality provided.
[0109] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology. Thus, the scope of the present technical solution is limited only by the scope of the attached formula. SOURCES OF INFORMATION 1. An Empirical Study on Multi-Task Learning for Text Style Transfer and Paraphrase Generation, Pawei Bujnowskia, Kseniia Ryzhovac, Hyungtak Choib, Katarzyna Witkowskad, Jaroslaw Piersaa, Tymoteusz Krumholca and Katarzyna Beksa. Found on the Internet at: https: / / aclanthology.Org / 2020.coling-industry.6.pdf, 04 / 20 / 2022.
Claims
FORMULA 1. A method for generating text, performed by at least one computing device, and comprising the steps of: a) receiving input data containing source text in a natural language and a target style of the generated text; b) encoding the source text, wherein during encoding at least tokenization of the text data is performed; c) vectorizing the tokens obtained in step b); d) processing the vector representations of the tokens of the source text obtained in step c) using a machine learning model based on a neural network trained on text data stylized in accordance with a given target style, during which an array of vectorized stylized texts is formed;e) decoding each vectorized stylized text from the array obtained in step d), wherein the decoding comprises at least converting the vectorized stylized text into tokens and detokenizing; f) filtering the array of stylized texts obtained in step e); g) ranking the filtered stylized texts based on the pairwise distance between the original text and each of the possible stylized texts and the linguistic acceptability of each of the possible stylized texts; h) generating stylized text based on the data obtained in step g)-; 2. The method according to paragraph 1, characterized in that the filtering of the array of stylized texts is performed using regular expressions and a morphological analyzer.
3. The method according to paragraph 1, characterized in that the filtering of the array of stylized texts is performed on the basis of the coincidence of proper names in the original text and each stylized text from the array. 25 4. The method according to paragraph 3, characterized in that the coincidence of proper names is determined using the number of unchanged named entities contained in the original text and the stylized text.
5. The method according to paragraph 4, characterized in that the number of unchanged named entities is determined on the basis of recognition of named entities in the source text and each stylized text from the array.
6. The method according to paragraph 1, characterized by the fact that it additionally contains the step of checking for the presence of text stylization.
7. The method according to paragraph 7, characterized in that the verification of the presence of text stylization is carried out on the basis of determining whether the target style indicators have been replaced.
8. The method according to paragraph 1, characterized in that the ranking of the filtered stylized texts is performed on the basis of an average value determined from the pairwise distance between the original text and each of the possible stylized texts and the linguistic acceptability of each of the possible stylized texts.
9. A text generation system comprising: • at least one processor; • at least one memory connected to the processor, which contains machine-readable instructions that, when executed by at least one processor, ensure the execution of the method according to any one of claims 1-8.
Citation Information
Patent Citations
Method of animating sms-messages
RU2631164C2
Method and system for digital assistant text generation
RU2796208C1
Broad-Coverage Normalization System For Social Media Language
US20130173258A1
Adaptive parser-centric text normalization
US20150186355A1
Machine translation system
US20170220559A1
