Text smoothing method and device, equipment, storage medium and program product

Abstract

The invention discloses a text smoothing method and device, equipment, a storage medium and a program product, and relates to the technical field of information processing, and the method comprises the following steps: carrying out at least one level of attention mechanism-based coding on each segmented word in a target text to obtain a coding feature of each segmented word; wherein in each hierarchy, the attention range of each segmented word comprises the segmented word, a target preceding word segmented word in the preceding word and a target following word segmented word in the following word, the number of the target preceding word segmented words is at most a first number corresponding to the hierarchy, and the number of the target following word segmented words is at most a second number corresponding to the hierarchy; for each segmented word, a classification result of the segmented word is obtained based on the coding feature of the segmented word, the segmented word result at least ensures whether the segmented word needs to be smooth or not, the segmented word needing to be smooth in the target text is deleted based on the classification result, and the smoothed text is obtained. According to the method and the device, the text smoothing accuracy is improved, namely, the probability of wrong text smoothing is reduced.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and in particular to a text smoothing method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition technology has made great progress and is widely used in smart homes, voice assistants, smart customer service and other fields. However, due to the complexity and diversity of speech signals, the text output by the speech recognition system usually contains various unfluencies. For example, in oral communication, people often have incoherent components such as filling pauses, repetitions, corrections, and restarts. These components are often misidentified as valid speech information during the speech recognition process, making the recognized text difficult to understand and not conducive to the development of downstream tasks (such as translation, entity recognition and other natural language processing tasks). Therefore, it is necessary to smoothly process the text output by the speech recognition system to make the speech recognition results more accurate and fluent.

[0003] The current text smoothing method has a high probability of false smoothing. Therefore, how to reduce the probability of false smoothing of text has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, the present application provides a text smoothing method, apparatus, device, storage medium and program product to reduce the probability of text smoothing errors. The specific scheme is as follows:

[0005] The first aspect of the present application provides a text smoothing method, the method comprising:

[0006] Perform at least one level of attention-based encoding on each segmentation in the target text to obtain encoding features of each segmentation; wherein, in each level, the attention scope of each segmentation includes: the segmentation and the target preceding segmentation in its preceding context and the target following segmentation in its following context, the number of the target preceding segmentation is at most the first number corresponding to the level, and the number of the target following segmentation is at most the second number corresponding to the level;

[0007] For each segmentation, a classification result of the segmentation is obtained based on the encoding feature of the segmentation, and the classification result at least represents whether the segmentation needs to be smoothed;

[0008] Based on the classification result, the word segmentations that need to be smoothed in the target text are deleted to obtain a smoothed text.

[0009] In a possible implementation, encoding each word segment in the target text based on at least one level of attention mechanism includes:

[0010] Performing multiple levels of attention-based encoding on each word segment in the target text;

[0011] Among them, at different levels, the attention scope of the same participle is the same or different.

[0012] In a possible implementation, when the attention scopes of the same participle are different at at least two levels, the different attention scopes of the same participle at different levels include: the same participle has different target context participles at different levels.

[0013] In a possible implementation, encoding each word segment in the target text based on at least one level of attention mechanism includes:

[0014] For each sentence in the target text, a target number of word segments representing spaces are added to the end of the sentence to obtain an extended text; the target number is the maximum second number among the second numbers corresponding to the multiple levels;

[0015] Performing at least one level of encoding based on the attention mechanism on each word segment in the extended text to obtain encoding features of each word segment in the extended text;

[0016] The encoding features of each word segment of the target text are extracted from the encoding features of each word segment of the extended text.

[0017] In a possible implementation, the classification result indicates whether the segmentation word needs to be smoothed and the smoothing type of the segmentation word, and the smoothing type is a reduplicated word that needs to be smoothed or an modal particle that needs to be smoothed. The process of determining the segmentation word that needs to be smoothed in the target text based on the classification result includes:

[0018] For each participle, if the participle is a number and the classification result of the participle indicates that the participle is a reduplication that needs to be smoothed, obtain the next participle of the participle and its classification result;

[0019] If the next participle of the participle is not a number, and the classification result of the next participle of the participle indicates that the next participle of the participle is a reduplicated word that needs smoothing, the participle is determined to need smoothing, otherwise the participle is updated to not need smoothing.

[0020] In a possible implementation, the classification result includes: a first probability that the segmentation needs to be smooth, a second probability that the segmentation is a reduplicated word that needs to be smooth, and a third probability that the segmentation is a modal particle that needs to be smooth; and a process of determining the segmentation that needs to be smooth in the target text based on the classification result includes:

[0021] For each participle, if the classification result of the participle indicates that the participle is a reduplicated word that needs smoothing, and any of the following conditions is met, the participle is updated to not need smoothing;

[0022] Condition 1: the difference between the original score corresponding to the second probability and the original score corresponding to the first probability is greater than a threshold, and the classification result of the previous participle of the participle indicates that the previous participle does not need to be smooth, and the classification result of the next participle of the participle indicates that the next participle does not need to be smooth;

[0023] Condition 2: The participle belongs to the reduplication of ancient poetry;

[0024] Condition three: The participle is a reduplication of colloquialisms or idioms.

[0025] In a possible implementation, after determining the segmentation words that need to be smooth in the target text based on the classification result, and before deleting the segmentation words that need to be smooth in the target text, the method further includes:

[0026] Traversing the target text to find consecutively repeated segmented words;

[0027] The first n-1 consecutively repeated participles among the found consecutively repeated participles are determined as the participles that need to be smoothed; n is the number of consecutive repetitions of the participle; n is an integer greater than 2.

[0028] A second aspect of the present application provides a text smoothing device, the device comprising:

[0029] The encoding unit is used to perform at least one level of attention-based encoding on each segmentation in the target text to obtain encoding features of each segmentation; wherein, in each level, the attention scope of each segmentation includes: the segmentation and its target preceding segmentation in the preceding context and the target following segmentation in the following context, the number of the target preceding segmentation is at most the first number corresponding to the level, and the number of the target following segmentation is at most the second number corresponding to the level;

[0030] A classification unit, configured to obtain, for each segmented word, a classification result of the segmented word based on the encoding feature of the segmented word, wherein the classification result at least indicates whether the segmented word needs to be smoothed;

[0031] A smoothing unit is used to delete the segmented words that need to be smoothed in the target text based on the classification result to obtain a smoothed text.

[0032] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the text smoothing method of the above-mentioned first aspect or any implementation of the first aspect.

[0033] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0034] The memory is used to store computer programs;

[0035] The processor is used to execute the computer program so that the electronic device can implement the text smoothing method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0036] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the text smoothing method of the above-mentioned first aspect or any implementation of the first aspect.

[0037] By means of the above technical scheme, the text smoothing method, device, equipment, storage medium and program product provided by the present application perform at least one level of attention-based encoding on each segmentation in the target text to obtain the encoding features of each segmentation; wherein, at each level, the attention range of each segmentation includes the segmentation and its target preceding segmentation in the preceding context and the target following segmentation in the following context, the number of target preceding segmentations is at most the first number corresponding to the level, and the number of target following segmentations is at most the second number corresponding to the level; for each segmentation, the classification result of the segmentation is obtained based on the encoding features of the segmentation, and the segmentation result at least guarantees whether the segmentation needs to be smoothed, and the segmentations that need to be smoothed in the target text are deleted based on the classification result to obtain the smoothed text. When the present application performs attention-based encoding on each segmentation in the target text, the attention range of each segmentation is no longer the entire sentence where the segmentation is located, but is determined according to the maximum attention range of the preceding context (i.e., the first number) and the maximum attention range of the following context (i.e., the second number), thereby improving the accuracy of smoothing, that is, reducing the probability of incorrect smoothing of the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0039] Figure 1 A flowchart for implementing the text smoothing method provided in this application;

[0040] Figure 2 A flowchart for implementing the attention-based encoding of each word segment in the target text at each level provided in this application;

[0041] Figure 3 An example diagram of a smooth model using a transformer structure provided in this application;

[0042] Figure 4 A flowchart for implementing the present application to determine the word segmentation that needs to be smooth in the target text based on the classification results;

[0043] Figure 5 Another implementation flow chart of the text smoothing method provided in this application;

[0044] Figure 6 A structural schematic diagram of a text smoothing device provided in this application;

[0045] Figure 7 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0046] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0047] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0048] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0049] The current text smoothing method is prone to mis-smoothing. For example, when dealing with reduplication, words that are important to semantic information in the sentence may be mis-smoothed. For example, for the sentence "Should we drink a glass of ice cola or ice soda water?", the existing text smoothing method will smooth out the word "ice" in "ice cola", but it is obvious that the word "ice" in "ice cola" does not belong to reduplication. In order to improve the smoothing accuracy, this application is proposed.

[0050] like Figure 1 As shown, it is a flowchart of an implementation of the text smoothing method provided in an embodiment of the present application, including:

[0051] Step S101: Perform at least one level of attention-based encoding on each segmentation in the target text to obtain encoding features of each segmentation; wherein, at each level, the attention scope of each segmentation includes: the segmentation and its target preceding segmentation in its preceding context and the target following segmentation in its following context, the number of target preceding segmentation is at most the first number corresponding to the level, and the number of target following segmentation is at most the second number corresponding to the level.

[0052] The target text refers to the text to be smoothed. The target text can be obtained by performing speech recognition on real-time collected speech data, or by performing speech recognition on pre-stored speech data. The text obtained by speech recognition can be called spoken text.

[0053] When encoding each word segment in the target text, the present application adopts an encoding based on the attention mechanism. The present application has found that in the prior art, for each word segment, regardless of the length of the sentence in which the word segment is located, the sentence in which the word segment is located is used as the attention scope of the word segment. This global attention mechanism has two defects: First, for the text smoothing task, the farther the context relationship, the smaller the impact, and the closer the context, the greater the impact; when a word is far enough from the current discriminant word, it can be basically determined that the former has an impact on the latter (that is, the current discriminant word) of 0; there are very few cases where punctuation judgment is affected across 100 words, so the global perspective will increase useless computational burden. Second, global attention means that in the text smoothing task, the smoothing judgment of the current word will take into account the farther words, especially for reduplication. If the word at the current position is the same as the farther away word, it is very likely that the model will be mis-smoothed, which will greatly affect the text semantics.

[0054] In the present application, the attention range of a participle is no longer a fixed sentence in which the participle is located, but the maximum length of the preceding context (i.e., the first number, that is, the preceding context pays attention to at most the first number of participles) and the maximum length of the following context (i.e., the second number, that is, the following context pays attention to at most the second number of participles). In this way, for any participle, if the preceding context of the any participle in the sentence in which the any participle is located is n participles, and the following context of the any participle is m participles, then, if n is less than or equal to the first number, the preceding context attention range of the any participle is the n participles in the preceding context; if n is greater than the first number, the preceding context attention range of the any participle is the first number of participles in the preceding context that are closest to the any participle; similarly, if m is less than or equal to the second number, the following context attention range of the any participle is the m participles in the following context; if m is greater than the second number, the following context attention range of the any participle is the second number of participles in the following context that are closest to the any participle.

[0055] The context of each participle refers to the participles before it in the sentence where it is located; the context of each participle refers to the participles after it in the sentence where it is located. Based on this, in each sentence, the context of the first participle is empty, and the context of the last participle is empty.

[0056] Based on this application, at each level, the attention scopes of different participles in the same sentence may be the same or different. For example, taking "Should we drink a glass of ice cola or ice soda water?" as an example, the result of the sentence segmentation is "Should we drink a glass of ice cola or ice soda water?" In the prior art, for each participle in the sentence, the attention scope of the participle is the 11 participles "Should we drink a glass of ice cola or ice soda water?" In the present application, assuming that the first quantity corresponding to a certain level is 7 and the second quantity is 1, then for the participle "drink", its context is the three participles "do we want", and its context is the participle "a cup", that is to say, based on the present application, the attention scope of "drink" at this level is the five participles "do we want to drink a cup"; similarly, for the second participle "ice", its context is the seven participles "do we want to drink a cup of ice cola or", and its context is the participle "soda water", that is to say, based on the present application, the attention scope of the second "ice" at this level is the nine participles "do we want to drink a cup of ice cola or ice soda water".

[0057] Step S102: For each word segment, a classification result of the word segment is obtained based on the encoding feature of the word segment, and the classification result at least represents whether the word segment needs to be smoothed.

[0058] Each segmented word can be input into a classification module to obtain a classification result of the segmented word. The classification result can not only characterize whether the segmented word needs to be smooth, but also characterize the smooth type of the segmented word. The smooth type can include but is not limited to the following two types: reduplicated words that need to be smooth, and modal particles that need to be smooth.

[0059] Step S103: based on the classification result, the word segments that need to be smoothed in the target text are deleted to obtain a smoothed text.

[0060] The word segmentation that needs to be smooth in the target text may be determined at least based on the classification result, or the word segmentation that needs to be smooth in the target text may be determined in combination with the classification result and other information.

[0061] Delete the word segmentations that need to be smoothed in the target text to obtain a smoothed text.

[0062] The text smoothing method provided in the embodiment of the present application performs at least one level of attention-based encoding on each word segment in the target text. The attention scope of each word segment is no longer the entire sentence in which the word segment is located, but is determined based on the maximum attention scope of the previous text (i.e., the first quantity) and the maximum attention scope of the following text (i.e., the second quantity), thereby improving the accuracy of text smoothing, that is, reducing the probability of incorrect text smoothing.

[0063] In an optional embodiment, an implementation method of performing at least one level of attention-based encoding on each word segment in the target text may be:

[0064] Each word in the target text is encoded based on multiple levels of attention mechanism.

[0065] Among them, at different levels, the attention scope of the same participle is the same or different.

[0066] Optionally, the attention scope of the same participle in different levels is the same.

[0067] Alternatively, the attention scope of the same participle is different at different levels.

[0068] Alternatively, when each word segment in the target text is encoded based on the attention mechanism at least three levels, the attention scope of the same word segment is the same at at least two levels, and the attention scope of at least one level is different from that of other levels.

[0069] For example, each word in the target text is encoded based on the attention mechanism at four levels, and the attention range of the same word in different levels is the same; or, the attention range of the same word in different levels is different; or, the attention range of the same word in two levels is the first attention range, and the attention range of the other two levels is the second attention range, and the first attention range is different from the second attention range; or, the attention range of the same word in three levels is the third attention range, and the attention range of the other level is the fourth attention range, and the fourth attention range is different from the third attention range.

[0070] In an optional embodiment, when the attention scopes of the same participle are different at at least two levels, the attention scopes of the same participle at different levels are different including:

[0071] The same participle has different participles in target contexts at different levels.

[0072] Optionally, the target context participles of the same participle at different levels may be the same or different.

[0073] As an example, the attention ranges of the various levels are the same, wherein the first number is 7 and the second number is 1 for each level.

[0074] As an example, the attention ranges of the various levels are the same, wherein the first number is 7 and the second number is 2 for each level.

[0075] As an example, the attention ranges of the various levels are the same, wherein the first number of each level is 6 and the second number is 1.

[0076] As an example, the attention ranges of each level are the same, wherein the first number is 6 and the second number is 2 for each level.

[0077] As an example, the attention ranges of the various levels are the same, wherein the first number of each level is 8 and the second number is 1.

[0078] As an example, the attention ranges of the various levels are the same, wherein the first number is 8 and the second number is 3 for each level.

[0079] As an example, the attention range of the same word at different levels is different, where in the first two levels, the first number of each level is 7 and the second number is 1; in other levels, the first number of each level is 7 and the second number is 0 (that is, the attention range does not include the following text). Based on this, the computational burden can be reduced while ensuring the accuracy of text smoothness.

[0080] As an example, the attention scope of the same participle at different levels is different, wherein, as the level goes deeper, the corresponding first quantity gradually decreases, and the second quantity also gradually decreases.

[0081] As an example, the same participle has different attention scopes at different levels, wherein as the level goes deeper, the corresponding first quantity remains unchanged, while the second quantity gradually decreases.

[0082] The above examples are merely illustrative and do not constitute a limitation of the present application. The smaller the second number, the higher the smoothing accuracy. Therefore, different first numbers and second numbers can be set according to actual accuracy requirements.

[0083] In an optional embodiment, a flowchart for implementing the attention-based encoding of each word in the target text at each level is as follows: Figure 2 As shown, it may include:

[0084] Step S201: For each word segmentation, a weight matrix corresponding to the word segmentation is calculated based on the target feature of the word segmentation, the target feature of the target previous word segmentation of the word segmentation, and the target feature of the target following word segmentation of the word segmentation.

[0085] The weight matrix represents the attention weight of the segmentation to each segmentation within its attention scope (including: the segmentation and its target previous segmentation and target following segmentation).

[0086] Among them, at the first level, the target feature of the word segmentation is the embedded feature of the word segmentation, the target feature of the target previous word segmentation of the word segmentation is the embedded feature of the target previous word segmentation of the word segmentation, and the target feature of the target following word segmentation of the word segmentation is the embedded feature of the target following word segmentation of the word segmentation.

[0087] In order to improve the efficiency of smooth processing, when obtaining the word embedding features of the segmented words, you can first obtain the word id of the segmented words, and then use the word id to find the embedding features of the word in the preset dictionary. If the word id of the segmented word cannot be found in the dictionary, you can use the preset word (for example, <unk>) as the word id of the word.

[0088] The embedding feature of the word segmentation can be obtained by fusing the word embedding feature and the position embedding feature of the word segmentation. For example, the embedding feature of the word segmentation is obtained by adding the word embedding feature and the position embedding feature of the word segmentation. The embedding features of other word segmentations are obtained in the same way and will not be described here.

[0089] At a level other than the first one, the target feature of the word segmentation is the encoding feature of the word segmentation obtained by encoding at the previous level, the target feature of the target previous word segmentation of the word segmentation is the encoding feature of the target previous word segmentation of the word segmentation obtained by encoding at the previous level, and the target feature of the target following word segmentation of the word segmentation is the encoding input feature of the target following word segmentation of the word segmentation obtained by encoding at the previous level.

[0090] Step S202: Based on the above weight matrix, the word segmentation and its target preceding word segmentation and target following word segmentation are merged to obtain the encoding feature of the word segmentation.

[0091] The encoding feature of the word can be obtained by weighted summing the word and its target preceding word and target following word.

[0092] In an optional embodiment, the above-mentioned encoding of each segmentation in the target text based on at least one level of attention mechanism is performed to obtain encoding features of each segmentation; for each segmentation, the process of obtaining the classification result of the segmentation based on the encoding feature of the segmentation may include:

[0093] Each word segment in the target text is encoded based on the attention mechanism at least one level through the encoding module of the smooth model to obtain the encoding features of each word segment.

[0094] For each word segmentation, the classification result of the word segmentation is obtained based on the encoding features of the word segmentation through the classification module of the smooth module.

[0095] The smooth model uses text as training samples and uses the label of whether each word in the training sample needs smoothing as the training label.

[0096] Alternatively, the smoothing model is trained by using text as training samples, and taking whether each word in the training sample needs smoothing and the smoothing type when smoothing is needed as labels.

[0097] Optionally, in the embodiments of the present application, the smoothing model may adopt a model with a transformer structure, or a model with other structures, which is not specifically limited in the present application.

[0098] like Figure 3 As shown, an example diagram of a smooth model using a transformer structure provided in an embodiment of the present application. In this example diagram, the smooth model uses multiple (i.e., N) levels of attention-based encoding. Each level can use single-head attention encoding or multi-head attention encoding, which is not specifically limited in this application.

[0099] Among them, the positional encoding module is used to encode the position of each word in the input text to obtain the position embedding feature, and the word embedding module is used to obtain the word embedding feature of each word. The feedforward network layer is the feed forward module, and the normalization and residual connection module is the add&norm module. The attention layer can be a multi-head self-attention layer (mult-head attention layer) or a single-head self-attention layer.

[0100] When performing self-attention encoding at each level, a mask mechanism can be used to set the words that are not in the attention range of the word segmentation to zero, so as to achieve encoding based on the attention mechanism only according to the word segmentations within the attention range corresponding to the level.

[0101] The output of the linear classification layer includes the scores of each segmentation in different dimensions of each layer, where the score of the first dimension of each segmentation is the original score representing whether the segmentation needs to be smoothed, the score of the second dimension is the original score representing whether the segmentation is a reduplication, and the score of the third dimension is the original score representing whether the segmentation is an interjection.

[0102] The linear classification layer can also normalize the original scores of the three dimensions to obtain the probability distribution of the three dimensions. For example, the linear classification layer can input the original scores of the three dimensions into softmax for processing to obtain the probability distribution of the three dimensions, P0, P1, and P2; among which, P0 of each participle represents the first probability that the participle needs to be smooth; P1 of each participle represents the second probability that the participle is a reduplicated word that needs to be smooth, and P2 of each participle represents the third probability that the participle is an adverb that needs to be smooth. Based on this, the output of the linear classification layer can also include the probability distribution of each participle in different dimensions in each layer.

[0103] Softmax is a component of the linear classification layer.

[0104] In the case where the classification result only indicates whether the corresponding segmentation word needs to be smoothed, P0 can be compared with the target probability. If P0 is greater than the target probability, it is determined that the corresponding segmentation word needs to be smoothed. Otherwise, it is determined that the corresponding segmentation word does not need to be smoothed.

[0105] When the classification result indicates whether the corresponding participle needs to be smoothed and the type of smoothing that needs to be smoothed, P0 can be compared with the target probability. If P0 is greater than the target probability, it is determined that the corresponding participle needs to be smoothed. Then, P1 and P2 are compared. If P1 is greater than P2, it is determined that the corresponding participle is a reduplicated word that needs to be smoothed. Otherwise, it is determined that the corresponding participle is an modal particle that needs to be smoothed.

[0106] In an optional embodiment, the labels of the training text are automatically marked. Specifically:

[0107] The acquired text to be marked can be preliminarily cleaned to remove punctuation marks and special symbols to obtain the cleaned text.

[0108] For each sentence in the cleaned text, the sentence is segmented to obtain a segmented sentence. The segmentation method can adopt an existing segmentation scheme, which is not specifically limited in this application.

[0109] For each sentence after word segmentation, according to the modal particle marking rules, the modal particles in the sentence are marked with the first mark (indicating that the corresponding word is a modal particle that needs to be smoothed); according to the reduplication marking rules, the reduplication in the sentence is marked with the second mark (indicating that the corresponding word is a reduplication that needs to be smoothed), and other participles are not marked (indicating that the participle is neither a modal particle nor a reduplication and does not need to be smoothed).

[0110] Among them, marking the first mark on the modal particles in the sentence according to the modal particle marking rule includes: traversing the participles in the sentence, and for each participle traversed to, judging whether the participle is a preset modal particle, if so, marking the first mark on the participle, otherwise, not marking the first mark on the participle. The preset modal particles may include but are not limited to the following: "oh", "oh", "uh", "um", "ah", "ah", "la", "ha", "wow", "ha", "um hum", etc.

[0111] For example, "*Oh / F I remember what you said is that *Well / F Let's do part of the task"; another example is "Who should be in Baoji *Ah / F". In the above two examples, "*" and the adjacent " / F" constitute the first marker, and the content between * and / F is the modal particle, such as "Oh" and "Ah".

[0112] According to the reduplicated word annotation rules, the second annotation of reduplicated words in a sentence includes: searching for continuously repeated content in the segmented sentence according to the regular expressions corresponding to various repetition types, and annotating the first n - 1 occurrences of the continuously repeated content that is searched and repeated n times as the second segmentation. The repetition types can include, but are not limited to: reduplicated words of a single segmented word that are repeated two or more times (for example, in abccd, c is the single segmented word that appears repeatedly); reduplicated words of two segmented words that are repeated two or more times (for example, in ABAB, AB is the two segmented words that appear repeatedly).

[0113] For example, "Then there is *no problem / E no problem, so there is no such thing as doing it"; another example, "We don't have *ah / F *we / E we don't even have the tools, so how can we calculate"; and another example, "Can you stop *saying / E *saying / E *saying / E *saying / E *saying / E saying so much". In the above three examples, "*" and the adjacent " / E" form the second annotation, and the content between * and / E is the reduplicated word that needs to be smoothed, such as "no problem", "we", and "saying".

[0114] Correct the modal particles and reduplicated words that should not be smoothed in specific scenarios to obtain the labels of the text.

[0115] In some scenarios, some modal particles should not be smoothed. For example, modal particles in entity words (for example, "Ha" in "Harry Potter" should not be annotated as a modal particle; "Ha" in "Show Hand" should not be marked as a modal particle); in addition to entity words, modal particles in some other words should also not be smoothed. For example, "Ha" in "Hello" should not be marked as a modal particle, "He" in "乐呵呵" should not be marked as a modal particle, and so on.

[0116] In some scenarios, some reduplicated words should not be smoothed. For example, the first "He" in "乐呵呵" should not be marked as a reduplicated word, repeated digits in phone numbers should not be marked as reduplicated words, repeated words in ancient poems should not be marked as reduplicated words (for example, the first "Ji", the first "唧唧", and the third "Ji" in "唧唧复唧唧" should not be marked as reduplicated words), and repeated words in common sayings or idioms should not be marked as reduplicated words (for example, the first "Xin" in "欣欣向荣" should not be marked as a reduplicated word, and the first "Mi" in "甜蜜蜜" should not be marked as a reduplicated word).

[0117] In order to avoid words in specific scenarios being mistakenly marked as modal particles or reduplications, it is necessary to compare modal particles and reduplications marked based on rules. Optionally, it is possible to search whether there is any preset text in a preset text list in the sentence where the marked modal particle or reduplication is located, and the preset text list includes any of the following preset texts: preset words, preset ancient poems, preset proverbs, preset idioms, etc. If there is a preset text in the sentence where the modal particle or reduplication is located, and the marked modal particle or reduplication exists in the found preset text, the mark of the marked modal particle or reduplication is deleted, otherwise, the mark of the marked modal particle or reduplication remains unchanged.

[0118] When training the smooth model, the embedded features of the word segmentations of the training samples in the training set can be input into the smooth model to obtain the classification results output by the smooth model. The classification results represent whether each word segmentation in the training sample needs to be smoothed, or the classification results represent whether each word segmentation in the training sample needs to be smoothed, and the type of smoothing (modal particle or reduplication) when smoothing is required; with the goal of the classification results being close to the labels of the training samples, the parameters of the smooth model are updated.

[0119] By automatically labeling tags, the efficiency of obtaining training sets and the data range of training sets used to train smooth models can be improved (for example, training texts in different fields can be quickly obtained), thereby improving the training efficiency and generalization ability of smooth models.

[0120] Furthermore, after the smoothness model is trained using the training set, statistical analysis can be performed to determine the training indicators of the smoothness model in each smoothness type (for example, accuracy, recall rate, etc.). If the training indicators of the smoothness model in a certain smoothness type do not meet the qualification conditions, the training set can be expanded. At this time, some texts can be extracted from a text set not used for text smoothness, and the extracted texts can be processed to generate texts with modal particles or reduplications as training samples and added to the training set used for text smoothness.

[0121] As an example, the extracted text can be preliminarily cleaned to remove punctuation and special symbols to obtain the cleaned text; randomly select one or two segmented words in the cleaned text, repeat them, obtain text with reduplication, and then label the text with reduplication (see the above embodiment for specific implementation) and add it to the training set for text smoothness. And / or, randomly select a position in the cleaned text, add an interjection to the position, obtain text with interjection, and then label the text with interjection (see the above embodiment for specific implementation) and add it to the training set for text smoothness. For example:

[0122] Original text (i.e. extracted text): I really want to eat barbecue and drink beer!

[0123] Preliminary cleaning and labeling: I really want to eat barbecue and drink beer *ah / F

[0124] Data augmentation and annotation: I really want to eat / E Eat barbecue and drink beer / F

[0125] By specifically expanding the training set for text smoothing and using the expanded training set to train the smoothing model, the smoothing effect of the smoothing model in specific smoothing types (repetition of words or modal particles) can be qualitatively improved.

[0126] In an optional embodiment, an implementation method of performing at least one level of attention-based encoding on each word segment in the target text may be:

[0127] For each sentence in the target text, a target number of word segments representing a space is added to the end of the sentence to obtain an extended text. The target number is the maximum second number among the second numbers corresponding to each level.

[0128] This application study found that because the back field of view of the word segmentation (i.e., the attention range below) is k (the second number), during reasoning, the last k word segmentations of the sentence will lack the back field of view, which will cause the final word to be misjudged as a reduplication, resulting in the final word being incorrectly smoothed. In addition, the last two words will be repeated, and when the penultimate word needs to be smoothed, it will not be correctly judged and smoothed. It has been determined through experiments that when reasoning, adding k empty words with all embedded features of 0 to the end of the sentence can effectively solve the problem of incorrect smoothing of reduplication of the final word and the problem of the final word not being able to be smoothed.

[0129] Each word segment in the extended text is encoded based on the attention mechanism to obtain the encoding features of each word segment in the extended text.

[0130] Each word segment in the extended text includes each word segment in the target text and k extended words segment.

[0131] The encoding features of each word in the target text are extracted from the encoding features of each word in the extended text.

[0132] Alternatively, the encoding features of the k extended words in the extended text may be deleted.

[0133] In an alternative embodiment, since the repetition of numbers is different from the repetition of ordinary words and its repetition is meaningful in scenarios such as representing phone numbers, etc., it is necessary to separately process the word segmentation of numbers. Based on this, the classification result of each word segmentation in this application characterizes whether the word segmentation needs to be smoothed and the type of smoothing when the word segmentation needs to be smoothed, where the type of smoothing is: an overlapping word that needs to be smoothed, or an interjection that needs to be smoothed. Correspondingly, a flowchart of an implementation for determining the word segmentations that need to be smoothed in the target text based on the classification result provided by the embodiment of this application is as Figure 4 shown, and may include:

[0134] Step S401: For each word segmentation, if the word segmentation is a number and the classification result of the word segmentation characterizes that the word segmentation is an overlapping word that needs to be smoothed, obtain the next word segmentation of the word segmentation and its classification result.

[0135] Step S402: If the next word segmentation of the word segmentation is not a number and the classification result of the next word segmentation of the word segmentation characterizes that the next word segmentation of the word segmentation is an overlapping word that needs to be smoothed, determine that the word segmentation needs to be smoothed; otherwise, update the word segmentation to not need to be smoothed.

[0136] For example, based on the classification result of "我 想要 一 个 一 个 手枪” (I want one by one pistol), we can get: "我 想要 *一 / E*个 / E 一 个 手枪”. For the first number "一” (one), the classification result of the word segmentation characterizes that the word segmentation is an overlapping word that needs to be smoothed, then obtain the next word segmentation "个” (piece) of the word segmentation and its classification result. The next word segmentation "个” (piece) of the first number "一” (one) is not a number, and the classification result characterizes that "个” (piece) is an overlapping word that needs to be smoothed, then determine that the first number "一” (one) is a word segmentation that needs to be smoothed, and keep it marked as an overlapping word that needs to be smoothed.

[0137] For another example, based on the classification result of "房间 号 是 七 七 零 九” (The room number is 7709), we can get: "房间 号 是 *七 / E 七 零 九”. For the first number "七” (seven), the classification result of the word segmentation characterizes that the word segmentation is an overlapping word that needs to be smoothed, then obtain the next word segmentation "七” (seven) of the analysis. Since this word segmentation is still a number, the first number "七” (seven) does not need to be smoothed, so delete the mark of the first number "七” (seven), that is, there is no word segmentation that needs to be smoothed in the sentence "房间 号 是 七 七 零 九” (The room number is 7709).

[0138] In an alternative embodiment, the classification result of each word segmentation includes: the first probability that the word segmentation needs to be smoothed, the second probability that the word segmentation is an overlapping word that needs to be smoothed, and the third probability that the word segmentation is an interjection that needs to be smoothed; the process of determining the word segmentations that need to be smoothed in the target text based on the classification result includes:

[0139] For each segmented word, if the classification result of the segmented word indicates that the segmented word is a reduplicated word that needs to be smoothed, and meets any of the following conditions, update the segmented word to not need to be smoothed.

[0140] Condition 1: The difference between the original score corresponding to the second probability and the original score corresponding to the first probability is greater than the threshold, and the classification result of the previous segmented word of this segmented word indicates that the previous segmented word does not need to be smoothed, and the classification result of the next segmented word of this segmented word indicates that the next segmented word does not need to be smoothed.

[0141] Condition 2: The segmented word belongs to reduplicated words in ancient Chinese poems. Optionally, it is possible to search in the sentence where the marked reduplicated word is located for any preset text in the first preset text list, and the first preset text list includes preset ancient Chinese poems. If there is a preset text in the sentence where the marked reduplicated word is located, and the marked reduplicated word exists in the found preset text, it means that the marked reduplicated word does not need to be smoothed. Therefore, delete the mark of the marked reduplicated word. For example, the first "Xiao" character in "The wind blows shrill over the Yi River cold" should not be smoothed.

[0142] Condition 3: The segmented word belongs to reduplicated words in common sayings or idioms. Optionally, it is possible to search in the sentence where the marked reduplicated word is located for any preset text in the second preset text list, and the second preset text list includes any of the following preset texts: preset common sayings, preset idioms, etc. If there is a preset text in the sentence where the marked reduplicated word is located, and the marked reduplicated word exists in the found preset text, it means that the marked reduplicated word does not need to be smoothed. Therefore, delete the mark of the marked reduplicated word. For example, the first "Hu" character in "silly" should not be smoothed, and the first "Tian" character and the first "Mi" character in "sweet and honeyed" should not be smoothed either.

[0143] For each segmented word, if the classification result of the segmented word indicates that the segmented word is a reduplicated word that needs to be smoothed, but the segmented word does not meet any of the above three conditions, determine that the segmented word needs to be smoothed.

[0144] In an optional embodiment, since the smoothing model cannot guarantee 100% recall rate, and long reduplicated words of the "AAA" type (the number of repetitions is greater than or equal to 3) obviously cause semantic redundancy, therefore, this application adds a re-judgment smoothing mechanism for long reduplicated words of the "AAA" type to ensure that such problems that need to be smoothed can be smoothed 100% (such as: *you / E *you / E *you / E You look really nice today). The processing complexity of this re-judgment smoothing mechanism for long reduplicated words is O(n), where n is the number of reduplicated words. Based on this, after determining the segmented words that need to be smoothed in the target text based on the classification result and before deleting the segmented words that need to be smoothed in the target text, this application further includes:

[0145] Traverse the target text to find segmented words with consecutive repetition times greater than 2.

[0146] The first n-1 consecutively repeated participles among the n consecutively repeated participles found are determined as the participles that need to be smoothed. n is the number of consecutive repetitions of the participle; n is an integer greater than 2.

[0147] That is to say, after determining the word segmentation that needs to be smooth in the target text based on the classification results, if a reduplication with a continuous repetition number greater than 2 is found in the target text, no matter what the classification results of the first n-1 reduplications among the reduplications with a continuous repetition number of n are, the first n-1 reduplications will be determined as the word segmentation that needs to be smooth.

[0148] like Figure 5 As shown, another implementation flowchart of the text smoothing method provided in the embodiment of the present application may include:

[0149] Step S501: add two empty participles (recorded as empty participles) to the end of the smooth text to obtain an extended text.

[0150] Step S502: Perform six-level attention-based encoding on each word in the extended text to obtain encoding features of each word in the extended text. The encoding features of each word in the extended text include encoding features of each word in the text to be smoothed and encoding features of two empty words.

[0151] The attention scope of each segmentation includes: the segmentation and the target segmentation in the previous context and the target segmentation in the next context. The attention scopes of the first two levels are the same, and the attention scopes of the last four levels are the same. In the first two levels, the number of target segmentation in the previous context is at most 7, and the number of target segmentation in the next context is at most 2; in the last four levels, the number of target segmentation in the previous context is at most 7, and the number of target segmentation in the next context is 0.

[0152] That is to say, in the first two levels, if the number of participles in the previous context of the participle (i.e., the participles before the participle in the sentence where the participle is located) is less than or equal to 7, then the target previous context participle of the participle is the participle in the previous context of the participle; if the number of participles in the previous context of the participle is greater than 7, then the target previous context participle of the participle is the 7 participles closest to the participle in the previous context. Similarly, if the number of participles in the following context of the participle (i.e., the participles after the participle in the sentence where the participle is located) is less than or equal to 2, then the number of target following context participles of the participle is the number of participles in the following context of the participle; if the number of participles in the following context of the participle is greater than 2, then the target following context participle of the participle is the 2 participles closest to the participle in the following context.

[0153] Step S503: For each word segment in the text to be smoothed, a classification result of the word segment is obtained based on the encoding features of the word segment, and the classification result includes: a first original score representing whether the word segment needs to be smoothed and a corresponding first probability, a second original score representing whether the word segment is an interjection that needs to be smoothed and a corresponding second probability, and a third original score representing whether the word segment is a reduplication that needs to be smoothed and a corresponding third probability.

[0154] The first probability is obtained by normalizing the first original score, the second probability is obtained by normalizing the second original score, and the third probability is obtained by normalizing the third original score.

[0155] Step S504: for each word segment in the text to be smoothed, determine whether the word segment is a number. If it is a number, proceed to step S505, otherwise proceed to step S506.

[0156] Step S505: Execute a separate processing mechanism for digital reduplication: if the classification result of the participle indicates that the participle is a reduplication that needs to be smoothed, obtain the next participle of the participle and its classification result; if the next participle of the participle is not a number, and the classification result of the next participle of the participle indicates that the next participle of the participle is a reduplication that needs to be smoothed, determine that the participle needs to be smoothed, otherwise update the participle to not need to be smoothed.

[0157] Step S506: If the classification result of the segmentation indicates that the segmentation does not need to be smooth, determine that the segmentation does not need to be smooth; if the classification result of the segmentation indicates that the segmentation is a modal particle that needs to be smooth, determine that the segmentation is a modal particle that needs to be smooth. If the classification result of the segmentation indicates that the segmentation is a reduplicated word that needs to be smooth, proceed to step S507.

[0158] Step S507: Execute the reduplication fallback judgment mechanism: if the segmentation word meets any of the following conditions, update the segmentation word to not require smoothing.

[0159] Condition 1: The difference between the second original score and the first original score is greater than a threshold (for example, 8), and the classification result of the previous participle of the participle indicates that the previous participle does not need to be smooth, and the classification result of the next participle of the participle indicates that the next participle does not need to be smooth.

[0160] Condition 2: The participle is a reduplication of ancient poetry.

[0161] Condition three: The participle is a reduplication of colloquialisms or idioms.

[0162] If the participle does not meet any of the above three conditions, the participle is kept as a reduplicated word that needs to be smooth. After all the participles are judged based on steps S504-S507, the process proceeds to step S508.

[0163] Step S508: Execute the smoothing mechanism for AAA type long reduplicated words again: traverse the text to be smoothed to find the segmented words with a continuous repetition number greater than 2. The first n-1 continuous repeated segmented words among the n continuously repeated segmented words found are determined as the segmented words that need to be smoothed; n is the continuous repetition number of the segmented words; n is an integer greater than 2.

[0164] Step S509: deleting the segmented words that need to be smoothed in the text to be smoothed, to obtain the smoothed text.

[0165] Corresponding to the method embodiment, the present application also provides a text smoothing device. A structural schematic diagram of the text smoothing device provided in the embodiment of the present application is shown in FIG. Figure 6 As shown, it may include:

[0166] Coding unit 601, classification unit 602 and smoothing unit 603;

[0167] The encoding unit 601 is used to perform at least one level of attention-based encoding on each segmentation in the target text to obtain encoding features of each segmentation; wherein, in each level, the attention scope of each segmentation includes: the segmentation and the target preceding segmentation in its preceding context and the target following segmentation in its following context, the number of the target preceding segmentation is at most the first number corresponding to the level, and the number of the target following segmentation is at most the second number corresponding to the level;

[0168] The classification unit 602 is used to obtain, for each segmentation, a classification result of the segmentation based on the encoding feature of the segmentation, wherein the classification result at least indicates whether the segmentation needs to be smoothed;

[0169] The smoothing unit 603 is used to delete the segmented words that need to be smoothed in the target text based on the classification result to obtain a smoothed text.

[0170] The text smoothing device provided in the embodiment of the present application, when performing at least one level of attention-based encoding on each word segment in the target text, the attention range of each word segment is no longer the entire sentence in which the word segment is located, but is determined based on the maximum attention range of the previous text (i.e., the first number) and the maximum attention range of the following text (i.e., the second number), thereby improving the accuracy of smoothing, that is, reducing the probability of incorrect text smoothing.

[0171] In an optional embodiment, when the encoding unit 601 performs at least one level of attention-based encoding on each word segment in the target text, it is used to:

[0172] Performing multiple levels of attention-based encoding on each word segment in the target text;

[0173] Among them, at different levels, the attention scope of the same participle is the same or different.

[0174] In an optional embodiment, when the attention scopes of the same participle are different at at least two levels, the different attention scopes of the same participle at different levels include: the same participle has different target context participles at different levels.

[0175] In an optional embodiment, when the encoding unit 601 performs at least one level of attention-based encoding on each word segment in the target text, it is used to:

[0176] For each sentence in the target text, a target number of word segments representing spaces are added to the end of the sentence to obtain an extended text; the target number is the maximum second number among the second numbers corresponding to each level;

[0177] Performing at least one level of attention-based encoding on each word segment in the extended text to obtain encoding features of each word segment in the extended text;

[0178] The encoding features of each word segment of the target text are extracted from the encoding features of each word segment of the extended text.

[0179] In an optional embodiment, the classification result indicates whether the segmentation word needs to be smoothed and the smoothing type of the segmentation word, and the smoothing type is a reduplicated word that needs to be smoothed or an interjection that needs to be smoothed; when the smoothing unit 603 determines the segmentation word that needs to be smoothed in the target text based on the classification result, it is used to:

[0180] For each participle, if the participle is a number and the classification result of the participle indicates that the participle is a reduplication that needs to be smoothed, obtain the next participle of the participle and its classification result;

[0181] If the next participle of the participle is not a number, and the classification result of the next participle of the participle indicates that the next participle of the participle is a reduplicated word that needs smoothing, the participle is determined to need smoothing, otherwise the participle is updated to not need smoothing.

[0182] In an optional embodiment, the classification result includes: a first probability that the segmentation needs to be smooth, a second probability that the segmentation is a reduplicated word that needs to be smooth, and a third probability that the segmentation is a modal particle that needs to be smooth; when the smoothing unit 603 determines the segmentation that needs to be smooth in the target text based on the classification result, it is used to:

[0183] For each participle, if the classification result of the participle indicates that the participle is a reduplication that needs smoothing and meets any of the following conditions, the participle is updated to not need smoothing; Condition 1: the difference between the original score corresponding to the second probability and the original score corresponding to the first probability is greater than the threshold, and the classification result of the previous participle of the participle indicates that the previous participle does not need smoothing, and the classification result of the next participle of the participle indicates that the next participle does not need smoothing; Condition 2: the participle is a reduplication of ancient poetry; Condition 3: the participle is a reduplication of colloquialisms or idioms.

[0184] In an optional embodiment, after the smoothing unit 603 determines the segmentation words that need to be smoothed in the target text based on the classification result and before deleting the segmentation words that need to be smoothed in the target text, it further includes:

[0185] Traversing the target text to find word segments with a number of consecutive repetitions greater than 2;

[0186] The first n-1 consecutively repeated participles among the found consecutively repeated participles are determined as the participles that need to be smoothed; n is the number of consecutive repetitions of the participle; n is an integer greater than 2.

[0187] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a structural schematic diagram of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application can be a terminal device (such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (can be a single server, can be a server cluster, or can be a cloud server, etc.). Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0188] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0189] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0190] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any text smoothing method provided in the embodiment of the present application.

[0191] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any text smoothing method provided in the embodiment of the present application.

[0192] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0193] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0194] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0195] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0196] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0197] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / unk>

Claims

1. A text smoothing method, characterized in that: The method comprises: Perform at least one level of attention-based encoding on each segmentation in the target text to obtain encoding features of each segmentation; wherein, in each level, the attention scope of each segmentation includes: the segmentation and the target preceding segmentation in its preceding context and the target following segmentation in its following context, the number of the target preceding segmentation is at most the first number corresponding to the level, and the number of the target following segmentation is at most the second number corresponding to the level; For each segmentation, a classification result of the segmentation is obtained based on the encoding feature of the segmentation, and the classification result at least represents whether the segmentation needs to be smoothed; Based on the classification result, the word segmentations that need to be smoothed in the target text are deleted to obtain a smoothed text.

2. The method according to claim 1, characterized in that The encoding of each word segment in the target text based on the attention mechanism at least one level includes: Performing multiple levels of attention-based encoding on each word segment in the target text; Among them, at different levels, the attention scope of the same participle is the same or different.

3. The method according to claim 2, characterized in that When the attention scopes of the same participle are different at at least two levels, the different attention scopes of the same participle at different levels include: the same participle has different participles in target contexts at different levels.

4. The method according to claim 1, characterized in that The encoding of each word segment in the target text based on the attention mechanism at least one level includes: For each sentence in the target text, a target number of word segments representing spaces are added to the end of the sentence to obtain an extended text; the target number is the maximum second number among the second numbers corresponding to each level; Performing at least one level of encoding based on the attention mechanism on each word segment in the extended text to obtain encoding features of each word segment in the extended text; The encoding features of each word segment of the target text are extracted from the encoding features of each word segment of the extended text.

5. The method according to claim 1, characterized in that The classification result indicates whether the participle needs to be smoothed and the smoothing type of the participle, wherein the smoothing type is a reduplicated word that needs to be smoothed or an interjection that needs to be smoothed; The process of determining the word segmentation that needs to be smooth in the target text based on the classification result includes: For each participle, if the participle is a number and the classification result of the participle indicates that the participle is a reduplication that needs to be smoothed, obtain the next participle of the participle and its classification result; If the next participle of the participle is not a number, and the classification result of the next participle of the participle indicates that the next participle of the participle is a reduplicated word that needs smoothing, the participle is determined to need smoothing, otherwise the participle is updated to not need smoothing.

6. The method according to claim 1, characterized in that The classification result includes: a first probability that the segmentation needs to be smooth, a second probability that the segmentation is a reduplicated word that needs to be smooth, and a third probability that the segmentation is an interjection that needs to be smooth; and a process of determining the segmentation that needs to be smooth in the target text based on the classification result includes: For each word, if the classification result of the word indicates that the word is a reduplication that needs smoothing, and any of the following conditions is met, the word is updated to not need smoothing; Condition 1: the difference between the original score corresponding to the second probability and the original score corresponding to the first probability is greater than a threshold, and the classification result of the previous participle of the participle indicates that the previous participle does not need to be smooth, and the classification result of the next participle of the participle indicates that the next participle does not need to be smooth; Condition 2: The participle belongs to the reduplication of ancient poetry; Condition three: The participle is a reduplication of colloquialisms or idioms.

7. The method according to any one of claims 1 to 6, characterized in that: After determining the segmentation words that need to be smooth in the target text based on the classification result, and before deleting the segmentation words that need to be smooth in the target text, the method further includes: Traversing the target text to find words with a continuous repetition number greater than 2; The first n-1 consecutively repeated participles among the n consecutively repeated participles found are determined as the participles that need to be smoothed; n is the number of consecutive repetitions of the participle; n is an integer greater than 2.

8. A text smoothing device, characterized in that: The device comprises: The encoding unit is used to perform at least one level of attention-based encoding on each segmentation in the target text to obtain encoding features of each segmentation; wherein, in each level, the attention scope of each segmentation includes: the segmentation and its target preceding segmentation in the preceding context and the target following segmentation in the following context, the number of the target preceding segmentation is at most the first number corresponding to the level, and the number of the target following segmentation is at most the second number corresponding to the level; A classification unit, configured to obtain, for each segmented word, a classification result of the segmented word based on the encoding feature of the segmented word, wherein the classification result at least indicates whether the segmented word needs to be smoothed; A smoothing unit is used to delete the segmented words that need to be smoothed in the target text based on the classification result to obtain a smoothed text.

9. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the text smoothing method as claimed in any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the text smoothing method as described in any one of claims 1 to 7.

11. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the text smoothing method as described in any one of claims 1 to 7.

Images