Text processing method and device

Abstract

The embodiments of the present specification provide a text processing method and device, wherein the text processing method includes: determining initial words of an initial text in an initial vocabulary; processing the initial text based on preset processing rules to obtain candidate words of the processed initial text; comparing the initial words with the candidate words, and taking the candidate words that do not match the initial words as target words; and calculating the word representation of the target words based on preset representation generation rules.

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of computer technology, and in particular to a text processing method. Background Art

[0002] With the development of science and technology and people's demand for convenient human-computer interaction, machine learning has been widely used in the field of natural language processing. For example, using a trained language generation model, some objects are converted into accurate natural language, that is, using relatively accurate sentences to describe these objects, which can be voice, pictures or text, etc. The construction of the vocabulary in the language generation model refers to the process of obtaining words from text expectations in an automatic or semi-automatic manner, which directly determines the performance of the natural language processing system. In practical applications, after the pre-trained model vocabulary is fixed, it cannot be adjusted for subsequent task data. Therefore, there is an over-segmentation of specific domain nouns or sentences, which makes it impossible to accurately determine the word representation of words in the vocabulary, resulting in poor text processing results. Summary of the invention

[0003] In view of this, the present specification provides a text processing method. One or more embodiments of the present specification also relate to a text processing device, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a text processing method is provided, including:

[0005] Determining initial words of the initial text in the initial vocabulary;

[0006] Processing the initial text based on a preset processing rule to obtain candidate words of the processed initial text;

[0007] Comparing the initial word with the candidate words, and taking the candidate words that do not match the initial word as target words;

[0008] The word representation of the target word is calculated based on a preset representation generation rule.

[0009] According to a second aspect of the embodiments of this specification, there is provided a text processing device, including:

[0010] A determination module, configured to determine an initial word of an initial text in an initial vocabulary;

[0011] An acquisition module is configured to process the initial text based on a preset processing rule to obtain candidate words of the processed initial text;

[0012] a comparison module, configured to compare the initial word with the candidate words, and use the candidate words that do not match the initial word as target words;

[0013] The calculation module is configured to calculate the word representation of the target word based on a preset representation generation rule.

[0014] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0015] Memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the text processing method when executing the computer-executable instructions.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any one of the text processing methods are implemented.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned text processing method.

[0019] A text processing method provided by an embodiment of the present specification determines initial words of an initial text in an initial vocabulary; processes the initial text based on preset processing rules to obtain candidate words of the processed initial text; compares the initial words with the candidate words, and uses the candidate words that do not match the initial words as target words; and calculates the word representation of the target word based on preset representation generation rules.

[0020] Specifically, by comparing the initial words of the initial text with the candidate words in the processed initial text, the target words that do not match the initial words and the candidate words are screened out, and then based on the preset representation generation rules, the word representation of the target words in the processed initial text can be calculated, which is convenient for subsequent adaptation to the needs of different application scenarios for the word representation of the target words in the initial text. In this way, not only can the word representations of more words in the initial text be obtained, but it is also convenient for subsequent training of text processing models based on target words, word representations of target words, and initial words in the initial vocabulary that meet the scenario requirements, so that the effect of the text processing model is enhanced, and it can also adapt to the needs of different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a text processing method provided by an embodiment of this specification;

[0022] Figure 2 It is a schematic diagram of a text processing method for implementing word representation migration provided by an embodiment of this specification;

[0023] Figure 3 It is a schematic diagram of a model application of a text processing method provided by an embodiment of this specification;

[0024] Figure 4 is a structural schematic diagram of a text processing device provided by an embodiment of this specification;

[0025] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0026] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0027] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] First, the terms involved in one or more embodiments of this specification are explained.

[0030] Pre-trained model: a model trained on large-scale unlabeled data.

[0031] Vocabulary: When a natural language neural network model processes a sentence, it will first map the words into mathematical representations. This process uses a vocabulary as a query.

[0032] In actual applications, the pre-trained model vocabulary is usually fixed and cannot be adjusted for task data in other subsequent project scenarios. There is excessive segmentation of domain-specific nouns or sentences, which makes it difficult for the pre-trained model to understand sentences in actual project scenarios and cannot effectively process sentences, thus restricting the effectiveness of the pre-trained model. For the above scenarios, specific optimizations can be made to subsequent task data, such as retraining a new vocabulary with other data or training a new vocabulary with downstream data. However, for this solution, resources will be consumed for tedious vocabulary retraining in each specific project scenario, which is not universal. At the same time, completely abandoning the initial vocabulary will also affect model performance.

[0033] Based on this, the text processing method provided in the embodiment of this specification proposes a pluggable word list migration text processing method for the problem of subword segmentation differences in language generation models, and also proposes a new pre-training model application process. When faced with the situation where words in the new word list do not appear in the original word list, a word representation generator can be used to generate mathematical representations of these new words. The word representation generator will comprehensively analyze the mathematical representations of old words that are similar to the new word morphology, and calculate the mathematical representations of new words through a neural network model, thereby allowing downstream actual project scenarios to use the desired word list; and when training the word representation generator, the old words will be randomly split and combined in the pre-training model to generate new words. By calculating the training loss value of the pre-training model and the loss value of the original sentence identifier and the sentence after the split and combination, the model parameters of the representation generator are jointly trained and updated. This method enables the model to flexibly select a specific word list in the actual project scenario, thereby improving the effect of the model in the project scenario.

[0034] It should be noted that the pre-trained model involved in the text processing method provided in the embodiments of this specification can understand any text processing model, such as a translation model, etc. The specific description of the following embodiments is described in detail by taking the text translation model as an example.

[0035] In this specification, a text processing method is provided. This specification also relates to a text processing apparatus, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0036] In order to obtain more model training data and make the text processing model adaptable to different project scenarios, the text processing method provided in this embodiment proposes a method for generating word representations of target words. The preset representation generation rules can be used to generate corresponding word representations for different target words, and then a vocabulary table adapted to the project scenario is generated. The vocabulary table can also be applied to the text processing model. For specific implementation methods, please refer to Figure 1 , Figure 1 A flowchart of a text processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0037] Step 102: Determine the initial words of the initial text in the initial vocabulary.

[0038] The initial vocabulary can be understood as the words of the initial training data in the pre-training model; the initial text can be understood as the text sentence to be processed.

[0039] It should be noted that this embodiment uses the detailed processing of the initial text in the initial training data in the pre-trained model as an example. According to the following method, the same processing can be performed on multiple texts at the same time, and this embodiment does not impose too many restrictions on this.

[0040] In actual applications, the server can divide an initial text, which can be understood as a text sentence in the first application scenario of the pre-trained model, and determine the initial words corresponding to the text sentence after the division in the initial vocabulary of the pre-trained model. The process of processing the text sentence to obtain the initial words can be divided based on the text processing rules in the first application scenario. This embodiment does not impose any limitation on this specific division method.

[0041] It should be noted that the first application scenario can be understood as the first application scenario in which the pre-trained model can be applied. For example, the pre-trained model is a text translation model, and the first application scenario in which it can be applied is a scientific translation scenario, which means that the vocabulary in the text translation model in this scenario tends to be more scientific.

[0042] For example, the first application scenario is a scientific text processing scenario, and the first initial text selected is "soda contains a large amount of carbon dioxide". The pre-trained model of the server can divide the initial text, and the initial words finally obtained are "soda", "in", "contains", "a large amount", "two", "oxidation", and "carbon".

[0043] Step 104: Process the initial text based on preset processing rules to obtain candidate words of the processed initial text.

[0044] Among them, the preset processing rules can be understood as rules for dividing the initial text and / or merging words. Each different application scenario can be adapted to different processing rules, and this embodiment does not make too many restrictions on this.

[0045] In actual applications, in order to adapt to different project application scenarios, the initial text can be processed again according to the preset processing rules to obtain candidate words for the processed initial text, so as to facilitate the subsequent screening of words suitable for another project application scenario from the candidate words and realize a new vocabulary suitable for the project application scenario.

[0046] Specifically, the processing of the initial text based on a preset processing rule to obtain candidate words of the processed initial text includes:

[0047] The words in the initial text are split and / or merged based on preset processing rules to obtain candidate words of the initial text.

[0048] In actual applications, in order to adapt to the second application scenario of the pre-trained model, where the second application scenario can be a life translation scenario, the words in the initial text are split and / or merged based on the preset processing rules under the second application scenario. Splitting can be understood as splitting the characters in a group of words. For example, the word "soda" can be split into "steam" and "water". Merging can be understood as merging two adjacent groups of words. For example, the two words "two" and "oxidation" are merged into "dioxidation". Therefore, the words after splitting and / or merging are used as candidate words for the initial text.

[0049] The text processing method provided in the embodiment of this specification processes the initial text again through preset processing rules to obtain candidate words, so as to facilitate the subsequent screening of words different from the initial words from the candidate words in the second application scenario, and then obtain the word representation of the word.

[0050] Step 106: Compare the initial word with the candidate words, and use the candidate words that do not match the initial word as target words.

[0051] In order to determine the translation words required for the initial text in the second application scenario, which are different from those in the first application scenario, in actual applications, it is necessary to compare the initial words in the first application scenario with the candidate words in the second application scenario, screen out the candidate words that do not match the initial words as the target words, and then determine the word representation of the target word, so as to facilitate the subsequent update of the word representation of the target word to the initial vocabulary, so that the initial vocabulary is suitable for the first application scenario as well as the second application scenario.

[0052] Step 108: Calculate the word representation of the target word based on a preset representation generation rule.

[0053] Among them, the preset representation generation rule can be understood as the word representation rule for calculating words, which can be a representation generator or a representation calculation formula. This embodiment does not limit this. The following embodiment will provide three word representation calculation methods to determine the word representation of the target word, but the embodiments of this specification are not limited to the following three calculation methods.

[0054] In order to determine the accurate word representation of the target word, the target word can be divided into two cases, one is the target word that can be further split and is called the first target word, and the other is the target word that cannot be split and is called the second target word. Then, for the above two cases, the word representations of different target words can be calculated respectively according to the representation mean; specifically, before calculating the word representation of the target word based on the preset representation generation rule, it also includes:

[0055] Determine a word representation of each word segment in the first target word, and determine a first representation mean based on the word representation of each word segment; and / or

[0056] A word representation of an initial word having an association relationship with a second target word is determined, and a second representation mean is determined based on the word representation of the initial word having an association relationship with the second target word.

[0057] Among them, the first target word can be understood as a target word that can be further split into participles. For example, the target word "soda" can be split into two participles, namely "steam" and "water".

[0058] Among them, the second target word can be understood as a target word that cannot be further split into participles, such as the target word "water", which cannot be further split into participles.

[0059] In specific implementation, the pre-training model can determine the word representation of each participle in the first target word, and then determine the average value of the word representation of each participle as the first representation mean. In actual applications, the word representation of the first target word can be determined based on the average value of the word representation of each participle. Since the second target word cannot be split into participles, the pre-training model can determine the word representation of the initial word with which the second target word has an associated relationship, and then determine the second representation mean based on the word representation of each initial word with which the second target word has an associated relationship. In actual applications, the word representation of the second target word can be obtained based on the average value of the word representation of the initial words with which the second target word has an associated relationship.

[0060] For example, if the first target word is "soda", the pre-trained model can determine the word representation of each word segment "steam" and the word representation of "water", which can be expressed as h'(steam) and h'(water), respectively. Therefore, the first representation mean h'(steam) of "soda" is the average of h'(steam) and h'(water). If the second target word is "water", the initial words associated with "water" can be determined from the initial word list as "ice water", "boiled water", and "in water", and then the word representation of "ice water", the word representation of "boiled water", and the word representation of "in water" can be determined, which can be identified as h'(ice water), h'(boiled water), and h'(in water), respectively. Therefore, the second representation mean h'(water) of "water" is the average of h'(ice water), h'(boiled water), and h'(in water).

[0061] It should be noted that the first representation mean and the second representation mean determined in this embodiment facilitate the subsequent calculation of the representation value of the target word. The following three calculation methods for the word representation of the target word are all based on the calculation of the above-mentioned first representation mean and the second representation mean. For details, please see the implementation method of the following embodiment.

[0062] The calculation method of the word representation of the first target word is as follows: the mean of the word representation of each word segment in the first target word is used as the word representation of the first target word, and the mean of the word representation of the initial words associated with the second target word is used as the word representation of the second target word.

[0063] Specifically, the step of calculating the word representation of the target word based on a preset representation generation rule includes:

[0064] Calculate the word representation of the first target word based on the first representation mean; and / or

[0065] A word representation of the second target word is calculated based on the second representation mean.

[0066] In practical applications, the word representation of the first target word can be obtained based on the mean of the first representation, and the word representation of the second target word can be obtained based on the mean of the second representation. Specifically, the word representation calculation method of the first target word can refer to formula (1):

[0067]

[0068] Among them, G(w) represents the word representation value of the first target word, E(w') is the first representation mean of the first target word, S_m(w) is the set of words with similar morphology to the first target word w; the word representation calculation method of the second target word can refer to the word representation calculation formula of the first target word, which will not be elaborated here.

[0069] The text processing method provided in the embodiments of this specification calculates the word representation of the target word by the mean of the word representations of the segmented words in the target word or the mean of the word representations associated with the target word, and can accurately determine the word representation of the target word, making it easier to subsequently add the word representation of the target word to the initial word list for updating.

[0070] The second method of calculating the word representation of the target word: not only through the mean of the word representation of the segmented words or the mean of the word representation of the initial words related to the target word, but also based on the semantic information of the target word, the weight value of each segmented word or the weight value of the initial words related to the target word can be determined, and then the weight ratio of the mean calculation of different words can be adjusted to achieve a more accurate calculation of the word representation of the target word.

[0071] Specifically, the step of calculating the word representation of the target word based on a preset representation generation rule includes:

[0072] Determine a first weight value for each word segment in the first target word based on the semantic information, and calculate a word representation of the first target word based on the first representation mean and the first weight value of each word segment; and / or

[0073] A second weight value of an initial word associated with a second target word is determined based on the semantic information, and a word representation of the second target word is calculated based on the second representation mean and the second weight value.

[0074] In practical applications, the first weight value of each participle in the first target word can be determined through the semantic information of the first target word. For example, if the first target word is "soda", it can be known through the semantic information that the weight value of the participle "water" in the first target word is higher than that of the participle "steam". Therefore, the first weight value of each participle is determined based on the initial word list, and the word representation of the first target word is calculated through the first representation mean and the first weight value. The specific calculation method can refer to formula (2):

[0075]

[0076] Among them, G(w) represents the word representation value of the first target word, E(w') is the first representation mean of the first target word, S_m(w) is the set of words with similar morphology to the first target word w, and W is a trainable parameter matrix.

[0077] In practical applications, the second weight value of the initial word associated with the second target word can be determined through the semantic information of the second target word. For example, the second target word is "water", and the initial words associated with the second target word are "ice water", "boiled water" and "in the water". According to the voice information, it can be known that the semantics of "ice water" and "boiled water" are higher than the weight value of the initial word "in the water". Therefore, based on the initial word list, the second weight value of each initial word associated with the second target word can be determined, and the word representation of the second target word can be calculated by the second representation mean and the second weight value. The specific calculation method can refer to the above formula (2), which will not be elaborated here.

[0078] The text processing method provided in the embodiments of this specification is a word representation calculation method based on the attention mechanism. The weight of each participle or initial word that has an associated relationship with the target word is determined, and then the word representation of the target word is calculated based on the weight and the representation mean. The semantic weight information is added in the above manner, so that the expression meaning of the target word is more accurate, and the method of calculating the word representation of the target word is also more effective.

[0079] The third method for calculating the word representation of the target word is as follows: when calculating the word segmentation weight value of the target word based on semantic information or calculating the weight value of the initial word that is associated with the target word, the attention mechanism is integrated into the position information, thereby not only realizing the weight ratio of different words, but also indicating the position information of the word through the position information, making the calculation method of the word representation of the target word more accurate.

[0080] Specifically, the step of calculating the word representation of the target word based on a preset representation generation rule includes:

[0081] Determine the position information of each participle in the first target word, determine a first weight value of each participle in the first target word based on the position information and semantic information, and calculate the word representation of the first target word based on the first representation mean and the first weight value of each participle; and / or

[0082] Determine the position information of the second target word in the initial word that has an associated relationship with the second target word, determine the second weight value of the initial word that has an associated relationship with the second target word based on the position information and semantic information, and calculate the word representation of the second target word based on the second representation mean and the second weight value.

[0083] In practical applications, after splitting the first target word, the position information of each segmentation can be determined, and the first weight value of each segmentation in the first target word can be determined based on the position information and semantic information of each segmentation. For example, for the first target word "steam", the segmentation "steam" and the segmentation "water" after splitting, the position of the segmentation "steam" is the first position of the first target word, and the position of the segmentation "water" is the last position of the first target word. Therefore, in the process of determining the first weight value of the segmentation "steam", it is necessary to consider the position information of the segmentation "steam", determine the first weight value of the segmentation based on the position information and semantic information, and then calculate the word representation of the first target word according to the first weight value of each segmentation and the first representation mean. The specific calculation method can refer to formula (3):

[0084]

[0085] Among them, G(w) represents the word representation value of the first target word, E(w') is the first representation mean of the first target word, S_m(w) is a set of words with similar morphology to the first target word w, W is a trainable parameter matrix, and I is the location information of the segmentation.

[0086] In practical applications, after determining the initial words that are associated with the second target word, the position of the second target word in the initial words can be determined. For example, the second target word is "water", and the initial words that are associated with the second target word are "ice water", "boiling water", and "in water". Therefore, it can be determined that the position information of the second target word in the initial words is at the last, last, and first place respectively. Therefore, based on the position information and semantic information, the second weight value of each initial word that is associated with the second target word is determined, and the word representation of the second target word is calculated based on the second representation mean and the second weight value. The specific calculation method can refer to the above formula (3), which will not be described in detail here.

[0087] It should be emphasized that compared with the two methods of only considering the mean, only considering the weight value of semantic factors and the mean, the third method mentioned above has higher accuracy, and in different task requirements, any of the above methods of calculating word representation can be selected, and this embodiment does not limit this.

[0088] The text processing method provided in the embodiments of this specification incorporates the attention mechanism into the position information of the word in the process of determining the word representation of the target word, which can more accurately express the weight value of the word to achieve a more accurate calculation result of the word representation.

[0089] The preset representation generation rule in the above embodiment may be a word representation generator in practical applications. In order to continuously enhance the calculation accuracy of the word representation of the target word, the parameters in the preset representation generation rule may be adjusted. Specifically, after the word representation of the target word is calculated based on the preset representation generation rule, the method further includes:

[0090] Determining a word representation loss value for a target word;

[0091] Determine the candidate word representations of the initial text based on the word representations of the candidate words, and determine the initial word representations of the initial text based on the word representations of the initial words;

[0092] Determining a text representation loss value of the initial text based on the candidate word representations of the initial text and the initial word representations of the initial text;

[0093] Based on the word representation loss value and the text representation loss value, the parameters of the preset representation generation rule are adjusted.

[0094] The word representation loss value of the target word can be understood as the loss value of the target word calculated by the calculation formula in the above embodiment.

[0095] Among them, the candidate word representation of the initial text can be understood as the word representation of the initial text composed of candidate words. For example, the initial text is "Soda contains a large amount of carbon dioxide", and its candidate words are "soda", "water", "in", "contains", "a large amount", "dioxide", and "carbon", then the candidate word representation of the initial text is the word representation of all the above candidate words.

[0096] Among them, the initial word representation of the initial text can be understood as the word representation of the initial text composed of initial words. Using the above example, the initial words are "soda", "medium", "containing", "a large amount", "two", "oxidation", and "carbon", then the initial word representation of the initial text is the word representation of all the above initial words.

[0097] Specifically, this embodiment calculates the loss value of the preset representation generation rule through two aspects. On the one hand, the word representation loss value of each target word is calculated, and on the other hand, the loss value between the split and combined sentence and the original sentence is calculated as the text representation loss value of the original sentence (the text representation loss value of the initial text), and the two loss values ​​are added as the overall loss value of the preset representation generation rule, and the corresponding parameters in the preset representation generation rule are adjusted based on the overall loss value.

[0098] In practical applications, in order to reduce the above two loss values, the calculation expression of the word representation loss value of the target word can be referred to:

[0099]

[0100] The calculation method of the text representation loss value of the initial text can be referred to:

[0101]

[0102] Then, the overall loss value representing the generation rule is preset:

[0103]

[0104] The text processing method provided in the embodiment of this specification calculates the loss value of the target word and the loss value of the representation value of the text where the target word is located and the representation value of the initial word, and then determines the overall loss value of the preset representation generation rule, so as to adjust the parameters in the preset representation generation rule and improve the calculation accuracy of the preset representation generation rule.

[0105] Based on the above determination of the word representation of the target word, the target word and the word representation of the target word can also be updated to the initial word list to adapt the initial word list to the second application scenario; specifically, after calculating the word representation of the target word based on the preset representation generation rule, it also includes:

[0106] Determine candidate words from the initial word list based on scenario requirements, and form a target word list based on the candidate words and / or the target words; or

[0107] The target words are added to the initial vocabulary to obtain a target vocabulary.

[0108] In actual applications, there are two ways to determine the target vocabulary. One is to determine the alternative words that meet the scenario requirements from the initial vocabulary according to the scenario requirements, and the target words and / or the alternative words determined according to the above embodiments together form the target vocabulary. The target vocabulary determined by this method can not only meet the scenario requirements, but also facilitate the subsequent training of models adapted to the application scenarios. The other method is to add all the target words determined in the above embodiments to the initial vocabulary to obtain the target vocabulary. This method forms a target vocabulary that can be applied to both the first application scenario and the second application scenario, that is, based on the target vocabulary, text processing can be implemented in the first application scenario, and corresponding text processing can also be implemented in the second application scenario. It should be noted that the text processing method provided in this embodiment has more than the above two methods for determining the target vocabulary, and no limitation is made here.

[0109] Through the above-mentioned embodiments of the text processing method provided in this specification, it can be seen that the word representation migration process is the process of forming a target vocabulary, the purpose of which is to process the initial vocabulary, and then to form a vocabulary associated with different application scenarios to adapt to more application scenarios, see Figure 2 , Figure 2 A schematic diagram showing the text processing method provided in the embodiment of this specification to achieve word representation migration.

[0110] Figure 2 The first line in is the original word representation (the word representation of the initial word). The original initial text is "Soda contains a lot of carbon dioxide". Each initial word is randomly divided from the initial text. The word representation of each initial word can be represented as E(soda), E(in), E(contains), E(a lot), E(di), E(oxidation), and E(carbon). When model training is performed based on the word representation of the above initial words, the word representation of the initial words can be encoded to train the pre-training model (text processing model). Figure 2 The last line in is an illustration of the word representation migration process. The word representations corresponding to the candidate words after the initial text is randomly split or merged, the word representations of each candidate word can be represented as G(steam), G(water), E(in), E(contains), E(a lot), G(dioxide), and E(carbon), respectively. It can be seen that the word representations of the initial words do not have the word representations of G(steam), G(water), and G(dioxide). Therefore, in the process of migrating the vocabulary, it is necessary to calculate the word representations of the target words "steam", "water", and "dioxide", and then G(steam), G(water), and G(dioxide) can be calculated based on the above-mentioned word representation calculation method of the target words, and then the word representations of the target words are input into the pre-trained model as training data, that is, the target words and the word representations of the target words are used as training data, so as to achieve the adaptation of the trained text processing model to another application scenario.

[0111] Based on this, after the word representation of the target word can be calculated based on the preset representation generation rule, the model training can be realized; specifically, after the word representation of the target word is calculated based on the preset representation generation rule, it also includes:

[0112] determining a word representation of the initial word;

[0113] A target text processing model is trained based on the initial word, the word representation of the initial word, the target word, and the word representation of the target word.

[0114] In practical applications, the pre-trained model can determine the word representation of each initial word in the initial text, and the above-mentioned method of determining the target word and the above-mentioned method of calculating the word representation of the target word can be applied to this embodiment and will not be elaborated here. The specific implementation method can refer to the above-mentioned embodiment. Based on this, the target text processing model can be trained based on the initial words, the word representations of the initial words, the target words and the word representations of the target words. The target text processing model can be obtained by unsupervised data training or supervised training data. This embodiment does not impose any limitations on this.

[0115] The text processing method provided in the embodiments of this specification trains a text processing model by determining target words, word representations of target words, initial words, and word representations of initial words. This method enables the model to flexibly select specific vocabularies in actual project scenarios, thereby improving the effectiveness of the model in project scenarios.

[0116] Furthermore, the training of the target text processing model based on the initial word, the word representation of the initial word, the target word and the word representation of the target word includes:

[0117] Training an initial text processing model based on the initial words and word representations of the initial words;

[0118] The target word and the word representation of the target word are input into the initial text processing model, and the initial text processing model is trained to obtain a target text processing model.

[0119] In practical applications, in the process of pre-training the model, the initial text processing model can be trained based on the initial words and the word representations of the initial words. The initial text processing model may only be suitable for a small part of the application scenarios, and there may be problems such as excessive segmentation of some specific domain proper nouns or sentences, resulting in the initial text processing model having a poor effect on text processing and lack of universality. Based on this, after the initial text is randomly split or merged again to obtain target words different from the initial words, this embodiment can use the word representation generator tool to calculate the word representation of the target word, and then input the target word and the word representation of the target word into the initial text processing model, train the initial text processing model, and finally obtain the target text processing model. Based on this method, the target word list required by each project can be extracted according to different project scenarios and data needs, and then the project-specific model can be realized, and the project-specific model can be used to train under the project scenario data to obtain a target text processing model that can be actually delivered.

[0120] The text processing method provided in the embodiments of this specification obtains a target text processing model by training an initial text processing model, so as to facilitate subsequent text processing in different project scenarios through the target text processing model. Based on the vocabulary migration method proposed by the text processing method provided in the embodiments of this specification, the model vocabulary can be conveniently replaced with a vocabulary optimized for the project scenario, and at the same time, the advantage of quickly converting a single pre-trained model into a specific project-specific model can be achieved.

[0121] The following combines the attached Figure 3 , taking the application of the text processing method provided in this specification in a text processing model as an example, to further illustrate the text processing method. Among them, Figure 3 FIG. shows a schematic diagram of the model application of a text processing method provided in an embodiment of this specification.

[0122] Figure 3 It can be divided into two major parts. The first part is the pre-training part, and the second part is the model application part. For the convenience of understanding, this embodiment takes the model as a text processing model as an example for illustration.

[0123] First, in the pre-training part, the pre-trained model is trained with pre-training data. Among them, the pre-training data can be understood as text sentences. Based on the intermediate layer of the pre-trained model, the text sentence is split into at least two initial words, such as Figure 3 the initial words "reporter", "worker", "motorcycle", "author", and "vehicle" after division in. The above initial words are input into the representation generator in the pre-training part. The representation generator performs random splitting or merging processing on the above text sentence again. At least two candidate words can be determined from the processed text sentence. By matching the candidate words with the initial words, the target words "person" and "motorcycle" can be determined. Furthermore, the representation generator calculates the corresponding word representations for the target words "person" and "motorcycle". At the same time, the target words can be added to the vocabulary in the model application part to obtain a target vocabulary suitable for the project scenario. Among them, there are two ways to obtain the target vocabulary. One is to screen out the words that meet the requirements of the project scenario from the initial vocabulary in the pre-training part based on the project scenario requirements. These words and the target words together constitute the target vocabulary. The other is to incorporate the target words into all the words in the initial vocabulary to obtain the target vocabulary; Figure 3The downstream data in can be understood as project data under different application scenarios, and the downstream model can be understood as a text processing model that adjusts the pre-trained model according to the target vocabulary to be suitable for the project scenario. In actual applications, after the downstream model obtains the target vocabulary, it can perform text processing on the downstream data, and use the word representation generator to calculate the word representation of the target words in the target vocabulary, thereby realizing the subsequent model processing process of the downstream data. It should be noted that the process of generating word representation by the word representation generator can be regarded as a process executed by a certain layer in the model, or it can be a process executed by a representation generator tool independent of the model. This embodiment does not impose too many restrictions on this.

[0124] The text processing method provided in the embodiments of this specification determines the target words applicable to the project scenario, and calculates the word representation of the target words with the help of a word representation generator, thereby training a text processing model applicable to the project scenario, which is applied to text processing of project data, thereby improving the accuracy and efficiency of text processing.

[0125] Corresponding to the above method embodiment, this specification also provides a text processing device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a text processing device provided by an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0126] A determination module 402 is configured to determine an initial word of an initial text in an initial word list;

[0127] The obtaining module 404 is configured to process the initial text based on a preset processing rule to obtain candidate words of the processed initial text;

[0128] A comparison module 406 is configured to compare the initial word with the candidate words, and use the candidate words that do not match the initial word as target words;

[0129] The calculation module 408 is configured to calculate the word representation of the target word based on a preset representation generation rule.

[0130] Optionally, the device further comprises:

[0131] A training module configured to determine a word representation of the initial word;

[0132] A target text processing model is trained based on the initial word, the word representation of the initial word, the target word, and the word representation of the target word.

[0133] Optionally, the device further comprises:

[0134] an acquisition module configured to determine candidate words from the initial vocabulary based on scenario requirements, and form a target vocabulary based on the candidate words and / or the target words; or

[0135] The target words are added to the initial vocabulary to obtain a target vocabulary.

[0136] Optionally, the training module is further configured to:

[0137] Training an initial text processing model based on the initial words and word representations of the initial words;

[0138] The target word and the word representation of the target word are input into the initial text processing model, and the initial text processing model is trained to obtain a target text processing model.

[0139] Optionally, the obtaining module 404 is further configured to:

[0140] The words in the initial text are split and / or merged based on preset processing rules to obtain candidate words of the initial text.

[0141] Optionally, the device further comprises:

[0142] a representation mean determination module, configured to determine a word representation of each word segment in the first target word, and determine a first representation mean based on the word representation of each word segment; and / or

[0143] A word representation of an initial word having an association relationship with a second target word is determined, and a second representation mean is determined based on the word representation of the initial word having an association relationship with the second target word.

[0144] Optionally, the calculation module 408 is further configured to:

[0145] Calculate the word representation of the first target word based on the first representation mean; and / or

[0146] A word representation of the second target word is calculated based on the second representation mean.

[0147] Optionally, the calculation module 408 is further configured to:

[0148] Determine a first weight value for each word segment in the first target word based on the semantic information, and calculate a word representation of the first target word based on the first representation mean and the first weight value of each word segment; and / or

[0149] A second weight value of an initial word associated with a second target word is determined based on the semantic information, and a word representation of the second target word is calculated based on the second representation mean and the second weight value.

[0150] Optionally, the calculation module 408 is further configured to:

[0151] Determine the position information of each participle in the first target word, determine a first weight value of each participle in the first target word based on the position information and semantic information, and calculate the word representation of the first target word based on the first representation mean and the first weight value of each participle; and / or

[0152] Determine the position information of the second target word in the initial word that has an associated relationship with the second target word, determine the second weight value of the initial word that has an associated relationship with the second target word based on the position information and semantic information, and calculate the word representation of the second target word based on the second representation mean and the second weight value.

[0153] Optionally, the device further comprises:

[0154] An adjustment module configured to determine a word representation loss value of a target word;

[0155] Determine the candidate word representations of the initial text based on the word representations of the candidate words, and determine the initial word representations of the initial text based on the word representations of the initial words;

[0156] Determining a text representation loss value of the initial text based on the candidate word representations of the initial text and the initial word representations of the initial text;

[0157] Based on the word representation loss value and the text representation loss value, the parameters of the preset representation generation rule are adjusted.

[0158] The text processing device provided in the embodiments of the present specification compares the initial words of the initial text with the candidate words in the processed initial text, screens out the target words that do not match the initial words and the candidate words, and then calculates the word representation of the target words in the processed initial text based on the preset representation generation rules, so as to facilitate the subsequent adaptation to the needs of different application scenarios for the word representation of the target words in the initial text. In this way, not only can the word representations of more words in the initial text be obtained, but also it is convenient to train the text processing model based on the target words, the word representations of the target words and the initial words in the initial vocabulary that meet the scene requirements, so that the effect of the text processing model is enhanced, and it can also adapt to the needs of different application scenarios.

[0159] The above is a schematic scheme of a text processing device of this embodiment. It should be noted that the technical scheme of the text processing device and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the text processing device can be referred to the description of the technical scheme of the above text processing method.

[0160] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0161] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0162] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0163] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0164] The processor 520 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned text processing method when executed by the processor.

[0165] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above text processing method.

[0166] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned text processing method when executed by a processor.

[0167] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above text processing method.

[0168] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned text processing method.

[0169] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be referred to the description of the technical scheme of the above text processing method.

[0170] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0172] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0173] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0174] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A text processing method, comprising: Determining initial words of an initial text in an initial vocabulary, wherein the initial text is a text sentence in a first application scenario of a pre-trained model, the first application scenario being a first application scenario to which the pre-trained model can be applied, and determining the initial words of the initial text in the initial vocabulary comprises: dividing the text sentence based on a text processing rule in the first application scenario, and determining the initial words corresponding to the text sentence after the text sentence is divided in the initial vocabulary of the pre-trained model; Processing the initial text based on a preset processing rule to obtain candidate words of the processed initial text, wherein the processing the initial text based on the preset processing rule to obtain candidate words of the processed initial text includes: splitting and / or merging words in the initial text based on the preset processing rule in the second application scenario to obtain candidate words of the initial text; Comparing the initial word with the candidate words, and taking the candidate words that do not match the initial word as target words; The word representation of the target word is calculated based on preset representation generation rules, wherein the word representation of the target word is calculated based on preset representation generation rules, including: generating a mathematical representation of the target word through a word representation generator, the word representation generator comprehensively analyzing the mathematical representation of the initial words that are morphologically similar to the target word; and calculating the mathematical representation of the target word through a neural network model to adapt to different application scenarios.

2. The text processing method according to claim 1, after calculating the word representation of the target word based on the preset representation generation rule, further comprising: determining a word representation of the initial word; A target text processing model is trained based on the initial word, the word representation of the initial word, the target word, and the word representation of the target word.

3. The text processing method according to claim 1, after calculating the word representation of the target word based on the preset representation generation rule, further comprising: Determining candidate words from the initial word list based on scenario requirements, and forming a target word list based on the candidate words and / or the target words; or The target words are added to the initial vocabulary to obtain a target vocabulary.

4. The text processing method according to claim 2, wherein the training of the target text processing model based on the initial word, the word representation of the initial word, the target word and the word representation of the target word comprises: Training an initial text processing model based on the initial words and word representations of the initial words; The target word and the word representation of the target word are input into the initial text processing model, and the initial text processing model is trained to obtain a target text processing model.

5. The text processing method according to claim 1, wherein the processing of the initial text based on a preset processing rule to obtain candidate words of the processed initial text comprises: The words in the initial text are split and / or merged based on preset processing rules to obtain candidate words of the initial text.

6. The text processing method according to any one of claims 1 to 5, before calculating the word representation of the target word based on the preset representation generation rule, further comprising: Determine a word representation of each participle in the first target word, and determine a first representation mean based on the word representation of each participle; and / or A word representation of an initial word having an association relationship with a second target word is determined, and a second representation mean is determined based on the word representation of the initial word having an association relationship with the second target word.

7. The text processing method according to claim 6, wherein the step of calculating the word representation of the target word based on a preset representation generation rule comprises: Calculate a word representation of the first target word based on the first representation mean; and / or A word representation of the second target word is calculated based on the second representation mean.

8. The text processing method according to claim 6, wherein the step of calculating the word representation of the target word based on a preset representation generation rule comprises: Determine a first weight value of each participle in a first target word based on semantic information, and calculate a word representation of the first target word based on the first representation mean and the first weight value of each participle; and / or A second weight value of an initial word associated with a second target word is determined based on the semantic information, and a word representation of the second target word is calculated based on the second representation mean and the second weight value.

9. The text processing method according to claim 6, wherein the step of calculating the word representation of the target word based on a preset representation generation rule comprises: Determine the position information of each participle in the first target word, determine a first weight value of each participle in the first target word based on the position information and semantic information, and calculate the word representation of the first target word based on the first representation mean and the first weight value of each participle; and / or Determine the position information of the second target word in the initial word that has an associated relationship with the second target word, determine the second weight value of the initial word that has an associated relationship with the second target word based on the position information and semantic information, and calculate the word representation of the second target word based on the second representation mean and the second weight value.

10. The text processing method according to claim 8, after calculating the word representation of the target word based on the preset representation generation rule, further comprising: Determining a word representation loss value for a target word; Determine the candidate word representations of the initial text based on the word representations of the candidate words, and determine the initial word representations of the initial text based on the word representations of the initial words; Determining a text representation loss value of the initial text based on the candidate word representation of the initial text and the initial word representation of the initial text; Based on the word representation loss value and the text representation loss value, the parameters of the preset representation generation rule are adjusted.

11. A text processing device, comprising: A determination module is configured to determine initial words of an initial text in an initial vocabulary, wherein the initial text is a text sentence in a first application scenario of a pre-trained model, the first application scenario being a first application scenario to which the pre-trained model can be applied, and the determination module is further configured to: divide the text based on a text processing rule in the first application scenario, and determine the initial words corresponding to the text sentence after the text sentence is divided in the initial vocabulary of the pre-trained model; an obtaining module configured to process the initial text based on a preset processing rule to obtain candidate words of the processed initial text, wherein the obtaining module is further configured to: split and / or merge words in the initial text based on a preset processing rule in a second application scenario to obtain candidate words of the initial text; a comparison module, configured to compare the initial word with the candidate words, and use the candidate words that do not match the initial word as target words; A calculation module is configured to calculate the word representation of the target word based on a preset representation generation rule, wherein the calculation module is further configured to: generate a mathematical representation of the target word through a word representation generator, the word representation generator comprehensively analyzes the mathematical representation of the initial words that are morphologically similar to the target word; and calculate the mathematical representation of the target word through a neural network model to adapt to different application scenarios.

12. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the text processing method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the text processing method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, wherein: When the computer program is executed in a computer, the computer is caused to execute the steps of the text processing method according to any one of claims 1 to 10.

Images