Text filling method and device

Abstract

This specification provides a text completion method and apparatus. The text completion method includes: determining an initial text and answer indices corresponding to regions in the initial text to be filled; processing the regions in the initial text to be filled based on a masked language model to obtain processed target text, wherein the masked language model employs a sparse attention mechanism; segmenting the target text and then using the masked language model to obtain target vectors for the segmented target words; determining the probability value of each word in the vocabulary relative to the target vector, and determining the target word to be filled based on the probability value and the answer index. Specifically, the text completion method uses a masked language model with a sparse attention mechanism, combined with a vocabulary, to achieve effective and correct prediction of answers for first-letter fill-in-the-blank questions.

Description

Technical Field

[0001] This specification relates to the field of natural language processing technology, and in particular to a text completion method. This specification also relates to a text completion device, a computing device, and a computer-readable storage medium. Background Technology

[0002] In recent years, artificial intelligence technology has developed rapidly, and natural language processing, which uses deep learning to process some dialogue understanding in our lives, has become a popular technology.

[0003] Natural Language Processing (NLP) is an important field in computer science and artificial intelligence. It studies various theories and methods that enable effective communication between humans and computers using natural language.

[0004] Transformer-based models have become one of the most successful deep learning models in NLP. For natural language processing tasks, the Bidirectional Encoder Representation from Transformers (BERT) neural network model is typically used. Currently, the task of filling in the blanks with the first letter is usually handled by the BERT model. However, due to the full attention mechanism used by the BERT model, one of its core limitations is its squared dependence on the sequence length, which means it can only support input sequences with a maximum length of 512. This makes it impossible to process extremely long texts with a length greater than 512 in the first letter fill-in-the-blank scenario, thus failing to achieve effective and correct prediction of the answer.

[0005] To address the aforementioned issues, how to process extremely long texts exceeding 512 characters in a first-letter fill-in-the-blank scenario and achieve effective and accurate prediction of the first-letter answer has become a pressing problem for technical personnel. Summary of the Invention

[0006] In view of this, embodiments of this specification provide a text completion method. This specification also relates to a text completion device, a computing device, and a computer-readable storage medium to address the technical deficiencies existing in the prior art.

[0007] According to a first aspect of the embodiments of this specification, a text completion method is provided, including:

[0008] Determine the initial text and the answer index corresponding to the region of words to be filled in the initial text;

[0009] The target text is obtained by processing the word-to-be-filled region in the initial text based on a masked language model, wherein the masked language model employs a sparse attention mechanism.

[0010] After segmenting the target text into words, the target vector of the target words in the segmented target text is obtained through the masked language model;

[0011] Determine the probability value of each word in the vocabulary relative to the target vector, and determine the target word to fill in based on the probability value and the answer index.

[0012] According to a second aspect of the embodiments of this specification, a text completion device is provided, comprising:

[0013] The initial text determination module is configured to determine the initial text and the answer index corresponding to the word-to-be-filled region in the initial text;

[0014] The target text determination module is configured to process the word-to-be-filled region in the initial text based on a masked language model to obtain the processed target text, wherein the masked language model adopts a sparse attention mechanism.

[0015] The vector acquisition module is configured to segment the target text into words and then pass the masked language model to obtain the target vectors of the target words in the segmented target text.

[0016] The target word filling determination module is configured to determine the probability value of each word in the vocabulary relative to the target vector, and determine the target word filling based on the probability value and the answer index.

[0017] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0018] Memory and processor;

[0019] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the above-described text filling method.

[0020] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the above-described text filling method.

[0021] The text filling method and apparatus provided in this specification include: determining an initial text and answer indices corresponding to the regions of words to be filled in the initial text; processing the regions of words to be filled in the initial text based on a masked language model to obtain processed target text, wherein the masked language model employs a sparse attention mechanism; segmenting the target text and obtaining target vectors of target words in the segmented target text through the masked language model; determining the probability value of each word in the vocabulary relative to the target vector, and determining the target words to be filled based on the probability values ​​and the answer indices.

[0022] Specifically, the text completion method uses a masked language model with a sparse attention mechanism to reduce the quadratic dependence on sequence length to linear, enabling the masked language model to support input sequences greater than 512. This solves the problem of processing ultra-long texts with a length greater than 512 in the first letter completion scenario, and combines a vocabulary to achieve effective and correct prediction of the answer in the first letter completion. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a text filling method provided in one embodiment of this specification;

[0024] Figure 2 This is a flowchart illustrating a text filling method for English initial letter scenarios provided in one embodiment of this specification;

[0025] Figure 3 This is a schematic diagram of the structure of a text filling device provided in one embodiment of this specification;

[0026] Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0027] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0030] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0031] BigBird: Transformer-based models (such as BERT) have become one of the most successful deep learning models in NLP. Unfortunately, due to their full attention mechanism, one of their core limitations is a squared dependency on sequence length, supporting only input sequences with a maximum length of 512. To address this issue, the BigBird model employs a sparse attention mechanism that reduces this squared dependency to linear, supporting input sequences with a maximum length of 4096, while also showing significant performance improvements on mask language modeling tasks.

[0032] Mask language model: This refers to randomly masking (replacing words with "[MASK]") some words in the input sequence (usually sentences as the basic unit), and then having the modified sequence predict which words were masked, such as the Big Bird model.

[0033] Mask Language Head Classifier: The vocabulary classifier in a masked language model, such as the output layer of the Big Bird model, is a basic function of the masked language model, used to determine the mapping from a vector dimension to a vocabulary dimension; in the masked language model, the Mask Language Head classifier predicts which word originally occupied the position of the replaced [MASK].

[0034] First letter fill-in-the-blank: Given a sentence or short passage, purposefully remove several words to create blanks, keeping only the first letter of the removed words, which serves as both a hint and a constraint; then spell the words completely according to the meaning of the sentence or short passage to make it coherent.

[0035] This specification provides a text completion method, and also relates to a text completion device, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0036] See Figure 1 , Figure 1 A flowchart of a text filling method according to an embodiment of this specification is shown, which specifically includes the following steps:

[0037] Step 102: Determine the initial text and the answer index corresponding to the word-filling area in the initial text.

[0038] Specifically, the text-filling method provided in the embodiments of this specification can be applied to English fill-in-the-blank scenarios, such as English initial letter fill-in-the-blank scenarios; it can also be applied to Chinese fill-in-the-blank scenarios, such as Chinese radical fill-in-the-blank scenarios, or fill-in-the-blank scenarios in other languages. The specific application scenario varies, and different adaptations can be made to the pre-trained masked language model and the accompanying vocabulary. This specification does not impose any limitations on this. For ease of understanding, the embodiments of this specification will focus on the application of the text-filling method to English initial letter fill-in-the-blank scenarios in a detailed description.

[0039] In the context of applying the text-filling method to English first-letter fill-in-the-blank scenarios, the initial text can be understood as an English short passage or English dialogue; the word-to-fill area can be understood as the blank area formed by the first letter of a word or phrase after removing it from the initial text, and words need to be filled into this blank area later; the answer index can be understood as the first letter retained in the word-to-fill area, so that the words filled into the blank area later can form the correct word or phrase with the first letter, so that the initial text can be restored to completeness.

[0040] Taking "In our solar s_, eight planest circle around the sun" as an example, "In oursolar s_, eight planest circle around the sun." can be understood as the initial text, "_" can be understood as the area containing the words to be filled in the initial text, and "s" in "s_" can be understood as the answer index corresponding to the area containing the words to be filled in the initial text.

[0041] In English fill-in-the-blank scenarios, each word field needs to be filled with a word corresponding to a reserved first letter (answer index). This first letter serves as a hint and constraint for the word to be filled in the field. Therefore, to ensure the correctness of the subsequent word selection, each word field must have a corresponding answer index. The specific implementation method is as follows:

[0042] The process of determining the initial text and the answer index corresponding to the region of words to be filled in the initial text includes:

[0043] Get the initial text containing the area to be filled in and the answer index;

[0044] Determine the number of regions in the area to be filled in, and the number of indices in the answer index;

[0045] If the number of regions and the number of indexes are the same, determine the answer index corresponding to the region to be filled in.

[0046] In practice, the initial text containing one, two, or more regions to be filled in and answer indexes is first obtained; then the number of regions to be filled in and the number of answer indexes contained in the initial text are determined; finally, the number of regions to be filled in and the number of answer indexes are compared. If the two numbers are the same, then it can be determined that there is a corresponding answer index at each region to be filled in.

[0047] If the number of the two is different, there may be cases where there are extra or missing word regions to be filled in, or extra or missing answer indexes. Regardless of which of the above situations causes the difference between the number of word regions to be filled in and the number of answer indexes, it will result in a mismatch between the word regions to be filled in and the answer indexes, which will affect the accuracy of the target words to be filled in when selecting word regions to be filled in.

[0048] Therefore, in order to avoid affecting the correctness of the target words to be filled in the subsequent selection of the word-filling areas, after obtaining the initial text containing the word-filling areas and the answer index, the correspondence between the word-filling areas and the answer index can be pre-judged based on the number of areas to be filled in and the number of indexes of the answer index, so as to ensure that there is a corresponding answer index at each word-filling area, thereby improving the correctness of the target words to be filled in the subsequent selection.

[0049] Continuing with the previous example, the initial text is: "In our solar s_, eight planest circle around the sun." This initial text contains one region to be filled in: "_", and one answer index: "s". Therefore, we can determine that the number of regions to be filled in is the same as the number of indexes in the answer index. If there are two regions to be filled in and one answer index, then we can determine that the number of regions to be filled in and the number of indexes in the answer index are not the same. Alternatively, if there is one region to be filled in and two answer indices, then we can also determine that the number of regions to be filled in and the number of indexes in the answer index are not the same.

[0050] Step 104: Process the word-to-be-filled region in the initial text based on the masked language model to obtain the processed target text, wherein the masked language model adopts a sparse attention mechanism.

[0051] Specifically, the process of processing the region to be filled in the initial text based on the masked language model to obtain the processed target text includes:

[0052] Based on the mask language model, the word region to be filled in the initial text is replaced with multiple preset masks of the mask language model, and the replaced initial text is determined as the target text.

[0053] The masked language model is a model that employs a sparse attention mechanism, such as the Big Bird model. However, it is not limited to the Big Bird model; it can also be any other model that employs a sparse attention mechanism and can perform natural language processing on very long texts (texts greater than 512 characters). For ease of understanding, this specification uses the Big Bird model as an example in all embodiments.

[0054] In practice, after determining the regions to be filled in the initial text and the answer index corresponding to each region, each region to be filled in the initial text is replaced with multiple preset masks of the mask language model, such as three preset masks, and the preset mask is [MASK]. The replaced initial text is then determined as the target text.

[0055] Continuing with the previous example, if “In our solar s_, eight planest circle around the sun.”, the default mask for the mask language model is [MASK].

[0056] Then, based on the masked language model, after replacing each region of the word to be filled in in the initial text with three preset masks, the target text is: "In our solar s[MASK][MASK][MASK],eight planest circle aroundthe sun."

[0057] In practical applications, the number of preset masks corresponding to each word-to-be-filled area can be set according to the actual application, and this manual does not impose any restrictions on this.

[0058] In this embodiment of the specification, each region of words to be filled in in the initial text is replaced with multiple special tokens [MASK] of the Big Bird model. Subsequently, the multiple [MASK] method can be used to solve the problem that the answer is more than one token, so as to improve the effectiveness and accuracy of selecting target words for filling in.

[0059] Step 106: After segmenting the target text into words, use the masked language model to obtain the target vector of the target words in the segmented target text.

[0060] For a detailed introduction to the mask language model, please refer to the above embodiments and terminology explanations, which will not be repeated here.

[0061] Specifically, the step of segmenting the target text and then using the masked language model to obtain the target vector of the target word in the segmented target text includes:

[0062] The target text is segmented based on a vocabulary list to obtain the text segmentation of the target text;

[0063] The text segmentation is input into a masked language model to obtain the segmentation vectors of the text segmentation;

[0064] Determine the mask vector of each preset mask corresponding to the region to be filled from the word segmentation vector;

[0065] The mask vector of each preset mask is used as the target vector of the target word in the target text after word segmentation.

[0066] In this process, each region to be filled in is replaced with multiple preset masks, and each preset mask serves as a target word segmentation. The mask vector of each preset mask serves as the target vector of each target word segmentation. Therefore, the mask vectors of the multiple preset masks corresponding to each region to be filled in can be understood as the target vectors of the multiple target words corresponding to each region to be filled in.

[0067] A vocabulary can be understood as the English vocabulary of a masked language model. This vocabulary allows for the combination of any English word, phrase, or word class. In practical applications, segmenting the target text based on the vocabulary to obtain the target text's word segments can be understood as inputting the target text into the masked language model, which then segments the target text according to its vocabulary to obtain the target text's word segments. Furthermore, the target word segments described below are determined from these word segments and are a subset of the word segments, specifically all instances of [MASK] selected from the word segments.

[0068] In practice, the target text is segmented based on a vocabulary list to obtain multiple segmented text segments. Each segmented text segment is then input into a pre-trained masked language model (such as the Big Bird model) to obtain a segmentation vector for each segmented text segment. From these multiple segmentation vectors, the mask vectors of all preset masks (i.e., the vectors of [MASK]) are selected. This preset mask is used as the target segment, and its mask vector is used as the target vector for the target segment in the segmented target text.

[0069] In the embodiments of this specification, the Big Bird model can be used to obtain the word segmentation vector of each word in the target text after word segmentation. The [MASK] mask vector is selected from the word segmentation vector to determine the probability value of each word in the vocabulary relative to it based on the [MASK] mask vector, so as to ensure the accurate selection of the target words in the subsequent filling.

[0070] Step 108: Determine the probability value of each word in the vocabulary relative to the target vector, and determine the target word to fill in based on the probability value and the answer index.

[0071] For a detailed explanation of the vocabulary, please refer to the above embodiments, which will not be repeated here.

[0072] Specifically, determining the probability value of each word in the vocabulary relative to the target vector, and determining the target word to fill in based on the probability value and the answer index, includes:

[0073] Based on multiple preset masks corresponding to the region to be filled, and the mask vector of each preset mask, the target vectors of multiple target word segments corresponding to the region to be filled are determined;

[0074] Each target vector corresponding to the region to be filled is passed through the word classifier of the masked language model to calculate the probability value of each word in the word list relative to each target vector;

[0075] The target word is determined based on the probability value and the answer index.

[0076] In practice, firstly, multiple preset masks and mask vectors for each region to be filled are determined. Based on the association between the preset masks and the target words, the target vectors for multiple target words corresponding to each region to be filled are obtained.

[0077] Then, for each target vector corresponding to each region to be filled in, the probability value of each word in the vocabulary relative to each target vector is calculated by the word classifier of the masked language model; finally, the target word to be filled in is determined based on the probability value and the answer index.

[0078] Continuing with the previous example, if the region to be filled in is replaced with three preset masks [MASK][MASK][MASK], then the target vectors for the multiple target word segments corresponding to the region to be filled in will be the target vectors of the three [MASK][MASK][MASK]. Then, each [MASK] target vector is passed through the word classifier of the mask language model to calculate the probability value of each word in the word list relative to each [MASK] target vector. Finally, the target word to be filled in is determined based on the probability value of each [MASK] target vector and the answer index.

[0079] In the embodiments of this specification, by combining the target vector output by the Big Bird model with the word classifier of the mask language model, the probability value of each word in the word list relative to each target vector of the word to be filled can be accurately obtained. Furthermore, the Big Bird model can significantly improve the performance on the Mask Language Model task.

[0080] In practical applications, each region to be filled in corresponds to multiple target vectors. Therefore, it is necessary to accurately obtain the target words for the region to be filled in by combining the probability value of each word in the vocabulary relative to each target vector in the region to be filled in with the answer index. The specific implementation method is as follows:

[0081] The process of determining the target word to fill in based on the probability value and the answer index includes:

[0082] Determine the first target vector corresponding to the region to be filled, and the first probability value of each word in the vocabulary relative to the first target vector;

[0083] Determine other target vectors corresponding to the word region to be filled, and a second probability value of each word in the vocabulary relative to the other target vectors, wherein the other target vectors are any vectors other than the first target vector;

[0084] Based on the first probability value, the second probability value, and the answer index, the target word to be filled in for the word-filling area is determined.

[0085] In practical applications, each word-to-be-filled region is replaced with multiple [MASK], so each word-to-be-filled region will correspond to multiple target vectors of [MASK]. The first target vector of the word-to-be-filled region can be understood as the target vector that is immediately adjacent to the first letter (answer index) to be filled, such as the leftmost target vector.

[0086] Specifically, firstly, the first target vector corresponding to each word-to-be-filled region is obtained, as well as the first probability value of each word in the vocabulary relative to the first target vector; then, the other target vectors of each word-to-be-filled region excluding the first target vector are obtained, as well as the second probability value of each word in the vocabulary relative to the other target vectors; finally, based on the first probability value, the second probability value, and the answer index, the target word to be filled in for each word-to-be-filled region is determined.

[0087] In the text filling method provided in the embodiments of this specification, in order to solve the problem that the target word is too long and exceeds one token, multiple [MASK] will be used to replace the word filling area. Subsequently, when selecting the target word based on multiple [MASK], it is also necessary to accurately select the target word in the word filling area based on the probability value of each word in the vocabulary relative to the target vector of each [MASK] and the answer index.

[0088] In specific implementation, determining the target word for the word-filling area based on the first probability value, the second probability value, and the answer index includes:

[0089] Based on the first probability value, the words in the vocabulary are arranged in descending order to form a sorted vocabulary.

[0090] The first initial word to fill in is determined from the sorted word list based on the answer index;

[0091] The word with the highest probability value in the vocabulary list is used as the second initial word.

[0092] If the second initial word filling is not a preset word filling, the target word filling for the word filling area is determined based on the first initial word filling and the second initial word filling.

[0093] Among them, preset words are special words that are pre-set in the vocabulary list according to actual needs. For example, setting preset words to be... <empty>This indicates that no content needs to be filled in this position.

[0094] In practice, the first probability value of each word in the vocabulary relative to the first target vector of the region to be filled is obtained. Then, the words in the vocabulary are arranged in descending order according to the size of the first probability value, that is, the word with the largest first probability value is arranged first, and the word with the smallest first probability value is arranged last, so as to form a sorted vocabulary.

[0095] Then, based on the answer index, the first initial word is selected from the sorted word list, and the word with the second highest probability value in the word list is used as the second initial word. If the second initial word is not the preset word, the target word for the word to be filled in is determined based on the first and second initial words.

[0096] In practical applications, the target word for each word-filling region may be a word larger than one token. In this case, based on multiple [MASKs], obtaining the word corresponding to each [MASK] and using the word corresponding to each [MASK] as the target word can more accurately determine the target word for each word-filling region. The first [MASK] of each word-filling region will always have a corresponding word, so the correct or best word can be selected from the word list based on the first probability value. However, if the target word for a word-filling region is only one token, the remaining [MASKs] may not have corresponding words. Therefore, when determining the target word for a word-filling region, it is necessary to determine whether there are corresponding words in the other [MASKs].

[0097] Specifically, the steps for determining the first initial word to fill in based on the answer index from the sorted vocabulary are as follows:

[0098] The step of determining the first initial word to fill in based on the answer index from the sorted vocabulary list includes:

[0099] Match the answer index with the first word in the sorted word list;

[0100] In the case of a match, the first word in the sorted vocabulary that matches the answer index is used as the first initial word; or

[0101] If there is no match, the answer index is matched with the next word in the sorted word list until a word matching the answer index is determined from the sorted word list as the first initial word to fill in.

[0102] The words in the sorted word list are arranged in descending order according to their first probability value. Therefore, the first word in the sorted word list can be understood as the word with the highest first probability value.

[0103] In practice, the answer index is matched with the first word in the sorted word list. If a match is found, the first word in the sorted word list that matches the answer index, i.e. the word with the highest probability value, is used as the first initial word to fill in.

[0104] In the case of a mismatch, the answer index is matched with the second word below the first word. If a match is found, the second word is used as the first initial word. If no match is found, the answer index is matched with the third word below the second word. This process continues until a word matching the answer index can be selected from the sorted word list as the first initial word.

[0105] The step of matching the answer index with the first word in the sorted word list includes:

[0106] The answer index is matched with the first letter of the first word in the sorted word list.

[0107] Continuing with the previous example, if the answer index is "s", then the first letter of the first word in the sorted word list must also be "s".

[0108] In this embodiment of the specification, the answer index is filtered by the first letter of each word in the sorted word list to quickly select the correct or best first initial word or target word to fill in the word area.

[0109] In another possible embodiment, after determining the maximum second probability value in the vocabulary relative to the other target vectors based on the second probability value, the method further includes:

[0110] If the second initial word filling is a preset word filling, the target word filling for the word filling area is determined based on the first initial word filling.

[0111] For a detailed explanation of the preset word filling, please refer to the above embodiments, which will not be repeated here.

[0112] Based on preset word filling <empty>This means that if no content needs to be filled in this position, then the first initial word to fill in is the target word for the area to be filled in.

[0113] In this embodiment of the specification, the target word to be filled in the word-filling area is a token. Then, the word with the highest second probability value corresponding to the target vector of the other [MASK] in the word-filling area is the preset word, that is, the position is empty. At this time, the first initial word can be used as the correct target word to be filled in the word-filling area.

[0114] The text completion method provided in the embodiments of this specification reduces the quadratic dependence on sequence length to linear by using a masked language model with a sparse attention mechanism. This enables the masked language model to support input sequences greater than 512, solving the problem of processing ultra-long texts with a length greater than 512 in the first letter completion scenario. It also combines a vocabulary to achieve effective and correct prediction of the answer in the first letter completion.

[0115] Meanwhile, the text filling method provided in the embodiments of this specification uses a multi-MASK approach, which can solve the problem that the target word in the word-filling area exceeds one token, and the addition of first letter filtering can further ensure the effectiveness and accuracy of the target word filling.

[0116] The following is in conjunction with the appendix Figure 2 Taking the text-filling method provided in this manual as an example in the application of English initial letter scenarios, the text-filling method will be further explained. Figure 2 The flowchart illustrates a text completion method for English initial letter scenarios provided in an embodiment of this specification, specifically including the following steps:

[0117] Step 202: Obtain the initial English first letter fill-in-the-blank text, which contains at least one word-filling area and the first letter of the target answer for each word-filling area.

[0118] Step 204: Replace each word region to be filled in in the initial English initial letter fill-in-the-blank text with three identical BigBird model special tokens - [MASK] to form the target English initial letter fill-in-the-blank text.

[0119] Step 206: Input the target English first letter fill-in-the-blank text into the Big Bird model after word segmentation to obtain the vector of each token (text segment) of the target English first letter fill-in-the-blank text.

[0120] Step 208: Extract the vector of all [MASK] tokens from the vector of each token (text segmentation) of the target English first letter fill-in text after word segmentation.

[0121] Step 210: Pass the vector of each [MASK] through the Mask Language Head classifier to calculate the probability value of each word in the vocabulary relative to the vector of each [MASK].

[0122] Step 212: Determine the vectors of the three [MASK] corresponding to each region to be filled, and the probability value of each word in the vocabulary relative to the vector of each of the three [MASK] vectors.

[0123] Step 214: For the first [MASK] vector corresponding to each word-filling region, filter the selectable words using the first letter of the target answer for that word-filling region to determine the first initial answer.

[0124] Specifically, determine the probability value of each word in the vocabulary relative to the first [MASK] vector, select the word with the highest probability value, and if the first letter of the word with the highest probability value is the same as the first letter of the target answer in the word area to be filled, then determine that word as the first initial answer.

[0125] If the first letter of the word with the highest probability value is different from the first letter of the target answer in the word area to be filled, then continue to filter other words with the highest probability value based on the first letter of the target answer in the word area to be filled, until the first initial answer is determined.

[0126] Step 216: For each region to be filled in, determine whether the vectors of other [MASK] exist as a second initial answer.

[0127] Specifically, if the word with the highest probability value in the vocabulary is " <empty>If the value is ',' it means that no content needs to be filled in at this position, that is, the vector of [MASK] does not have a corresponding second initial answer, and it can be removed from the obtained answer.

[0128] If the word with the highest probability value in the vocabulary is not " <empty>If the symbol "", it means that content needs to be filled in at this position. In this case, the word with the highest probability value in the vocabulary list can be used as the second initial answer.

[0129] Step 218: Determine the target answer for each word-filling area based on the first initial answer and / or the second initial answer.

[0130] The text completion method provided in this specification is applied to the English first-letter fill-in-the-blank scenario. By employing a masked language model with a sparse attention mechanism, the quadratic dependence on sequence length is reduced to linear, enabling the masked language model to support input sequences greater than 512. This solves the problem of processing ultra-long texts with a length greater than 512 in the English first-letter fill-in-the-blank scenario, and, combined with a vocabulary, achieves effective and correct prediction of answers in English first-letter fill-in-the-blank questions.

[0131] Meanwhile, the text filling method provided in the embodiments of this specification uses a multi-mask approach, which can solve the problem that the target answer in the area to be filled exceeds one token, and the addition of initial letter filtering can further ensure the effectiveness and accuracy of the target answer.

[0132] Corresponding to the above method embodiments, this specification also provides embodiments of a text completion device. Figure 3 A schematic diagram of a text-filling device according to an embodiment of this specification is shown. Figure 3 As shown, the device includes:

[0133] The initial text determination module 302 is configured to determine the initial text and the answer index corresponding to the word-to-be-filled region in the initial text;

[0134] The target text determination module 304 is configured to process the word-to-be-filled region in the initial text based on a masked language model to obtain the processed target text, wherein the masked language model adopts a sparse attention mechanism.

[0135] The vector acquisition module 306 is configured to segment the target text and then use the masked language model to obtain the target vector of the target segment in the segmented target text;

[0136] The target word filling determination module 308 is configured to determine the probability value of each word in the vocabulary relative to the target vector, and determine the target word filling based on the probability value and the answer index.

[0137] Optionally, the initial text determination module 302 is further configured to:

[0138] Get the initial text containing the area to be filled in and the answer index;

[0139] Determine the number of regions in the area to be filled in, and the number of indices in the answer index;

[0140] If the number of regions and the number of indexes are the same, determine the answer index corresponding to the region to be filled in.

[0141] Optionally, the target text determination module 304 is further configured to:

[0142] Based on the mask language model, the word region to be filled in the initial text is replaced with multiple preset masks of the mask language model, and the replaced initial text is determined as the target text.

[0143] Optionally, the vector acquisition module 306 is further configured to:

[0144] The target text is segmented based on a vocabulary list to obtain the text segmentation of the target text;

[0145] The text segmentation is input into a masked language model to obtain the segmentation vectors of the text segmentation;

[0146] Determine the mask vector of each preset mask corresponding to the region to be filled from the word segmentation vector;

[0147] The mask vector of each preset mask is used as the target vector of the target word in the target text after word segmentation.

[0148] Optionally, the target word filling determination module 308 is further configured to:

[0149] Based on multiple preset masks corresponding to the region to be filled, and the mask vector of each preset mask, the target vectors of multiple target word segments corresponding to the region to be filled are determined;

[0150] Each target vector corresponding to the region to be filled is passed through the word classifier of the masked language model to calculate the probability value of each word in the word list relative to each target vector;

[0151] The target word is determined based on the probability value and the answer index.

[0152] Optionally, the target word filling determination module 308 is further configured to:

[0153] Determine the first target vector corresponding to the region to be filled, and the first probability value of each word in the vocabulary relative to the first target vector;

[0154] Determine other target vectors corresponding to the word region to be filled, and a second probability value of each word in the vocabulary relative to the other target vectors, wherein the other target vectors are any vectors other than the first target vector;

[0155] Based on the first probability value, the second probability value, and the answer index, the target word to be filled in for the word-filling area is determined.

[0156] Optionally, the target word filling determination module 308 is further configured to:

[0157] Based on the first probability value, the words in the vocabulary are arranged in descending order to form a sorted vocabulary.

[0158] The first initial word to fill in is determined from the sorted word list based on the answer index;

[0159] The word with the highest probability value in the vocabulary list is used as the second initial word.

[0160] If the second initial word filling is not a preset word filling, the target word filling for the word filling area is determined based on the first initial word filling and the second initial word filling.

[0161] Optionally, the target word filling determination module 308 is further configured to:

[0162] Match the answer index with the first word in the sorted word list;

[0163] In the case of a match, the first word in the sorted vocabulary that matches the answer index is used as the first initial word; or

[0164] If there is no match, the answer index is matched with the next word in the sorted word list until a word matching the answer index is determined from the sorted word list as the first initial word to fill in.

[0165] Optionally, the target word filling determination module 308 is further configured to:

[0166] The answer index is matched with the first letter of the first word in the sorted word list.

[0167] Optionally, the device further includes:

[0168] The target word acquisition module is configured to determine the target word for the word-to-be-filled region based on the first initial word when the second initial word is a preset word.

[0169] The text completion device provided in the embodiments of this specification reduces the quadratic dependence on sequence length to linear by using a masked language model with a sparse attention mechanism. This enables the masked language model to support input sequences greater than 512, solving the problem of processing ultra-long texts with a length greater than 512 in the first letter completion scenario. It also combines a vocabulary to achieve effective and correct prediction of the answer in the first letter completion.

[0170] Meanwhile, the text filling device provided in this specification uses a multi-mask method, which can solve the problem that the target word in the word-filling area exceeds one token, and the addition of first letter filtering can further ensure the effectiveness and accuracy of the target word filling.

[0171] The above is an illustrative scheme of a text-filling device according to this embodiment. It should be noted that the technical solution of this text-filling device and the technical solution of the above-described text-filling method belong to the same concept. For details not described in detail in the technical solution of the text-filling device, please refer to the description of the technical solution of the above-described text-filling method.

[0172] Figure 4 A structural block diagram of a computing device 400 according to an embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0173] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0174] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0175] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 400 can also be a mobile or stationary server.

[0176] The processor 420 is configured to execute the following computer-executable instructions to implement the steps of the above-described text filling method.

[0177] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described text filling method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described text filling method.

[0178] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the above-described text filling method.

[0179] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-described text filling method belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the above-described text filling method.

[0180] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0181] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0182] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this specification is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this specification. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this specification.

[0183] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0184] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. These embodiments have been selected and specifically described in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.< / empty> < / empty> < / empty> < / empty>

Claims

1. A text completion method, characterized in that, include: Determine the initial text and the answer index corresponding to the word-filling area in the initial text, wherein the word-filling area is the blank area formed by retaining the first letter after removing a certain word or phrase from the initial text, and the answer index is the first letter retained in the word-filling area; The word-to-be-filled region in the initial text is replaced with multiple preset masks of the mask language model based on the mask language model, and the replaced initial text is determined as the target text. The mask language model adopts a sparse attention mechanism. After segmenting the target text, the target vector of the target segmented word in the target text is obtained through the mask language model. Each preset mask is used as a target segmented word, and the mask vector of each preset mask is used as the target vector of each target segmented word. Determine the probability value of each word in the vocabulary relative to the target vector, and determine the target word to fill in based on the probability value and the answer index.

2. The text completion method according to claim 1, characterized in that, The process of determining the initial text and the answer index corresponding to the region of words to be filled in the initial text includes: Get the initial text containing the area to be filled in and the answer index; Determine the number of regions in the area to be filled in, and the number of indices in the answer index; If the number of regions and the number of indexes are the same, determine the answer index corresponding to the region to be filled in.

3. The text completion method according to claim 1, characterized in that, The step of segmenting the target text into words and then using the masked language model to obtain the target vector of the target words in the segmented target text includes: The target text is segmented based on a vocabulary list to obtain the text segmentation of the target text; The text segmentation is input into a masked language model to obtain the segmentation vectors of the text segmentation; Determine the mask vector of each preset mask corresponding to the region to be filled from the word segmentation vector; The mask vector of each preset mask is used as the target vector of the target word in the target text after word segmentation.

4. The text completion method according to claim 3, characterized in that, The process of determining the probability value of each word in the vocabulary relative to the target vector, and determining the target word to fill in based on the probability value and the answer index, includes: Based on multiple preset masks corresponding to the region to be filled, and the mask vector of each preset mask, the target vectors of multiple target word segments corresponding to the region to be filled are determined; Each target vector corresponding to the region to be filled is passed through the word classifier of the masked language model to calculate the probability value of each word in the word list relative to each target vector; The target word is determined based on the probability value and the answer index.

5. The text completion method according to claim 4, characterized in that, The process of determining the target word to fill in based on the probability value and the answer index includes: Determine the first target vector corresponding to the region to be filled, and the first probability value of each word in the vocabulary relative to the first target vector; Determine other target vectors corresponding to the word region to be filled, and a second probability value of each word in the vocabulary relative to the other target vectors, wherein the other target vectors are any vectors other than the first target vector; Based on the first probability value, the second probability value, and the answer index, the target word to be filled in for the word-filling area is determined.

6. The text completion method according to claim 5, characterized in that, Determining the target word for the word-filling region based on the first probability value, the second probability value, and the answer index includes: Based on the first probability value, the words in the vocabulary are arranged in descending order to form a sorted vocabulary. The first initial word to fill in is determined from the sorted word list based on the answer index; Based on the second probability value, determine the maximum second probability value in the vocabulary relative to the other target vectors; If the maximum second probability value has a corresponding second initial word in the word list, the target word for the word to be filled in the word region is determined based on the first initial word and the second initial word.

7. The text completion method according to claim 6, characterized in that, The step of determining the first initial word to fill in based on the answer index from the sorted vocabulary list includes: Match the answer index with the first word in the sorted word list; In the case of a match, the first word in the sorted vocabulary that matches the answer index is used as the first initial word; or If there is no match, the answer index is matched with the next word in the sorted word list until a word matching the answer index is determined from the sorted word list as the first initial word to fill in.

8. The text completion method according to claim 7, characterized in that, The step of matching the answer index with the first word in the sorted word list includes: The answer index is matched with the first letter of the first word in the sorted word list.

9. The text completion method according to claim 6, characterized in that, After determining the maximum second probability value in the vocabulary relative to the other target vectors based on the second probability value, the method further includes: If the maximum second probability value does not have a corresponding second initial word in the word list, the target word for the word to be filled in the word region is determined based on the first initial word.

10. A text-filling device, characterized in that, include: The initial text determination module is configured to determine the initial text and the answer index corresponding to the word-to-be-filled area in the initial text, wherein the word-to-be-filled area is the blank area formed by retaining the first letter after removing a certain word or phrase from the initial text, and the answer index is the first letter retained in the word-to-be-filled area; The target text determination module is configured to replace the word-to-be-filled region in the initial text with multiple preset masks of the mask language model based on the mask language model, and determine the replaced initial text as the target text, wherein the mask language model adopts a sparse attention mechanism; The vector acquisition module is configured to segment the target text and then use the mask language model to obtain the target vector of the target segment in the segmented target text, wherein each preset mask is a target segment and the mask vector of each preset mask is the target vector of each target segment. The target word filling determination module is configured to determine the probability value of each word in the vocabulary relative to the target vector, and determine the target word filling based on the probability value and the answer index.

11. A computing device, characterized in that, It includes a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the text filling method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer instructions, characterized in that, When executed by the processor, this instruction implements the steps of the text filling method according to any one of claims 1 to 9.

Images