Method, apparatus, device and storage medium for text processing

Abstract

According to embodiments of the present disclosure, methods, apparatuses, devices and storage media for text processing are provided. In the method, a plurality of first feature representations are generated based on a plurality of text units in a first language original text, the plurality of first feature representations indicating semantics of the respective text units; the plurality of first feature representations are combined with a second feature representation to form a sequence of feature representations, the second feature representation indicating whether each text unit in the plurality of text units is a boundary of a text unit group; and based on the sequence of feature representations, a converted text in a second language is generated. In this way, the accuracy and efficiency of text conversion can be significantly improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and particularly to methods, apparatus, devices and computer-readable storage media for text processing. Background Technology

[0002] Text-to-speech (TTS) is a speech synthesis application that converts text into natural speech output. A typical TTS system consists of a front-end system and a back-end system. The front-end system is responsible for converting the input text into intermediate results, such as text segmentation, prosody (used to control speech rhythm and intonation), phoneme information, etc., and then sending these intermediate results to the back-end system. The back-end system generates sound that closely matches these results based on existing recording data. For example, the back-end system can first summarize some audio features (e.g., Mel spectrum) from the audio, and then use a training algorithm to train a converter, i.e., an acoustic model, to map the front-end results to these audio features. The back-end system can then use a vocoder to restore these audio features into audible sound.

[0003] Chinese dialect TTS systems can be used to convert dialect text into dialect speech. However, these systems are usually only usable by dialect experts because Chinese dialects often differ greatly from Standard Mandarin in terms of vocabulary, idioms, grammar, and usage, making it difficult even for native speakers to input correct sentences. Summary of the Invention

[0004] In a first aspect of this disclosure, a text processing method is provided. The method includes generating multiple first feature representations based on multiple text units in an original text of a first language, the multiple first feature representations indicating the semantics of the corresponding text units; combining the multiple first feature representations with second feature representations to form a feature representation sequence, the second feature representation indicating whether each text unit in the multiple text units is a boundary of a text unit group; and generating converted text in a second language based on the feature representation sequence.

[0005] In a second aspect of this disclosure, a method for text processing is provided. The method includes acquiring training text pairs, the training text pairs comprising source text in a first language and target text in a second language; converting the source text into converted text in the second language using a conversion model, comprising: generating multiple first feature representations based on multiple text units in the source text, the multiple first feature representations indicating the semantics of corresponding text units; combining the multiple first feature representations with second feature representations to form a feature representation sequence, the second feature representation indicating whether each text unit in the multiple text units is a boundary of a text unit group; generating converted text based on the feature representation sequence; and training the conversion model based at least on the differences between the converted text and the target text.

[0006] In a third aspect of this disclosure, a text processing apparatus is provided. The apparatus includes a feature generation module configured to generate a plurality of first feature representations based on a plurality of text units in an original text of a first language, the plurality of first feature representations indicating the semantics of the corresponding text units; a feature combination module configured to combine the plurality of first feature representations with second feature representations to form a feature representation sequence, the second feature representations indicating whether each text unit in the plurality of text units is a boundary of a group of text units; and a conversion module configured to generate converted text in a second language based on the feature representation sequence.

[0007] In a fourth aspect of this disclosure, a text processing apparatus is provided. The apparatus includes a text acquisition module configured to acquire training text pairs, the training text pairs including source text in a first language and target text in a second language; a text conversion module configured to convert the source text into converted text in the second language using a conversion model, the text conversion module being further configured to: generate multiple first feature representations based on multiple text units in the source text, the multiple first feature representations indicating the semantics of corresponding text units; combine the multiple first feature representations with second feature representations to form a feature representation sequence, the second feature representation indicating whether each text unit in the multiple text units is a boundary of a text unit group; and generate converted text based on the feature representation sequence; and a training module configured to train the conversion model based at least on the differences between the converted text and the target text.

[0008] In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the methods of the first or second aspect.

[0009] In a sixth aspect of this disclosure, a computer-readable storage medium is provided. A computer program is stored on the medium, which, when executed by a processor, implements the method of the first or second aspect.

[0010] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0013] Figure 2A A flowchart illustrating the processing procedure of a Chinese dialect TTS front-end according to some embodiments of the present disclosure is shown;

[0014] Figure 2B A flowchart is shown in the word-to-sound conversion process in Chinese dialect TTS front-end processing according to some embodiments of the present disclosure;

[0015] Figure 3 A flowchart of a text processing procedure according to some embodiments of the present disclosure is shown;

[0016] Figure 4 A schematic diagram of an example architecture for a text conversion model according to some embodiments of the present disclosure is shown;

[0017] Figure 5 A flowchart illustrating a model training process according to some embodiments of the present disclosure is shown;

[0018] Figure 6A and Figure 6B A schematic diagram illustrating the changes in the target alignment matrix before and after alignment dimension transformation according to some embodiments of the present disclosure is shown;

[0019] Figure 7 A block diagram of a text processing apparatus according to some embodiments of the present disclosure is shown;

[0020] Figure 8 A block diagram of a text processing apparatus according to some embodiments of the present disclosure is shown; and

[0021] Figure 9 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0024] The term "text unit" as used herein refers to the basic unit of text processing, the granularity of which can be changed and set according to the application. For example, a text unit can include words, phrases, characters, combinations thereof, or any other unit that appears in natural language expressions. In some examples, a text unit is also referred to as a token.

[0025] As used herein, a “text unit group” refers to a collection of multiple text units. For example, where the text units are words, a text unit group can include words, phrases, or sentences, etc.

[0026] The term "feature representation" as used herein refers to a representation that reflects a certain characteristic of text or text units, which is extracted from or can be generated based on text or text units.

[0027] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0028] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0029] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0030] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0032] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0033] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0034] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0035] As mentioned above, due to the strong particularities of the words, idiomatic expressions, grammar, and usage of Chinese dialects, dialect TTS systems are usually only used by dialect experts. The existing main solution is to directly reuse the Mandarin front-end in dialect TTS tasks. Since there are no word segmentation tools and vocabulary lists for dialects, the Mandarin dictionary is usually supplemented and used instead of the dialect dictionary. Such dialect TTS systems usually work well when inputting Chinese dialects that are similar in expression to Mandarin. This is because the dialect recordings are made based on dialect texts, and the acoustic models used by the back-end systems are usually trained based on dialect texts and audio; while the Mandarin front-end lacks support for dialects and predicts information such as word segmentation, prosody, and phonemes of dialects based on Mandarin expression habits, resulting in inaccurate front-end results.

[0036] However, when users use dialect TTS, they usually input dialect texts with inaccurate expressions or directly input Mandarin texts. If the input text has a large difference in expression from the dialect text used during training, the acoustic model of the back-end system will have problems of inadaptability, and the generated audio quality and naturalness will be relatively poor. For example, in Cantonese, "你先吃饭" is expressed as "你食饭先". If the TTS system performs audio synthesis based on "你食饭先", it can generate a more authentic and fluent Cantonese audio. However, because users are not familiar with Cantonese, they input expressions like "你先吃饭" or "你吃饭先" which are not very authentic into the TTS system, and the back-end model has not seen such expressions during training, so the authenticity and fluency of the generated Cantonese audio will be greatly reduced.

[0037] Figure 1 Fig. 100 shows an example environment 100 in which embodiments of the present disclosure can be implemented.

[0038] As Figure 1 shown, the example environment 100 includes an electronic device 110, which includes a text conversion system 115. The text conversion system 115 is configured to convert a text (referred to as "original text") 120 in one language (referred to as "first language", e.g., Mandarin Chinese) into a text (referred to as "converted text") 122 in another language (referred to as "second language", e.g., Chinese dialect). For illustrative purposes only, in Figure 1 it is shown that the original text is the Mandarin text "他说要给大家做点好吃的,显显本领", and the second text is the Cantonese dialect text "佢话要煮啲好嘢畀大家食,骚吓quali ".

[0039] It should be understood that the text conversion system 115 can be used for conversion between any two language texts. For example, the text conversion system 115 can convert Mandarin text to other dialect texts, convert dialect text to Mandarin text, convert one dialect text to another dialect text, or convert Chinese text to texts in other languages, and so on.

[0040] The text conversion system 115 can perform text conversion operations in any suitable manner. In environment 100, text conversion can be implemented using a conversion model 130. The conversion model 130 can be any suitable machine learning model, such as a Transformer model.

[0041] Electronic device 110 can be a terminal device or a server device. Terminal devices can be various portable or fixed terminals, such as tablets, desktop computers, laptops, mobile phones, and general electronic devices. Server devices can be centralized servers, distributed servers, mainframes, edge electronic devices, or cloud computing.

[0042] It should be understood that the text conversion system 115 in electronic device 110 and the various systems, units, modules, components and / or parts in the other figures described below can be implemented in various ways, including software, hardware, firmware or any combination thereof.

[0043] It should also be understood that Figure 1 The structure of the electronic device 110 shown is merely illustrative and not limiting. Depending on the specific implementation, the electronic device 110 may also include any other suitable systems, units, modules, components, and / or parts for implementing the corresponding functions. For example, in some embodiments, the electronic device 110 may perform TTS processing, further converting the converted text 125 generated by the text conversion system 115 into corresponding audio output. In these embodiments, the electronic device 110 may include components or parts for performing the TTS process.

[0044] In some embodiments, the text conversion system 115 can be applied to a TTS front-end specifically for Chinese dialects. The following is in conjunction with... Figure 2A and Figure 2B Some embodiments in this regard are discussed.

[0045] Figure 2A The following illustrates a processing procedure 200 for a Chinese dialect TTS front-end according to some embodiments of the present disclosure. Procedure 200 can be implemented at electronic device 110. For ease of discussion, it will be combined with... Figure 1 The environment 100 is used to describe the process 200.

[0046] In process 200, the original text 120 is processed by the text conversion system 115 to obtain the converted text 125. For example, the original text 120 could be the Mandarin text "What are you doing there? Your knees are all scraped!" and the converted text could be the Northeastern Mandarin text "What are you doing there? Your knees are all scraped!" Figure 2A As shown, the text conversion system 115 can use the conversion model 130 in box 202 to perform text conversion, converting the above-mentioned Mandarin text into the corresponding Northeastern dialect text.

[0047] In some embodiments, to protect irregular characters or strings input by the user from being output as is, some untranslatable characters or strings can be replaced with special markers, and then these special markers can be replaced back with the original strings after the text conversion is complete. For example, such as Figure 2A As shown, before text conversion in box 202, character replacement is performed in box 204. For example, some predetermined characters or strings (such as emojis, abbreviations, URLs, etc.) are replaced with predetermined tags (e.g., ...). <rep>)。After the inference of the conversion model 130 ends, at box 206, character reverse replacement is performed to restore or re-replace the predetermined markers with the original characters.

[0048] In process 200, at box 208, text preprocessing is performed. For example, operations such as removing illegal characters, semi-angularizing text, and converting traditional and simplified Chinese can be executed to unify the text format. At box 210, text regularization processing is performed. For example, unreadable text (such as non-literal forms like numbers, symbols, etc.) can be rewritten into readable text (such as pure literal forms). Thus, pronunciation errors that may occur in the TTS system can be eliminated, and the naturalness of the generated speech can be improved. For example, the original text 120 is the Mandarin text "I spent 666 yuan to check into a room with room number 666". In this sentence, it is difficult for the TTS system to distinguish whether the two "666"s before and after should be pronounced as "six six six" or "six hundred and sixty-six". Through the text regularization processing at box 210, the first "666" can be rewritten as "six hundred and sixty-six", and the second "666" can be rewritten as "six six six".

[0049] At box 212, word segmentation,词性, and prosody prediction are performed. For example, models or rules can be used to predict the word segmentation, prosody boundaries, and词性 of the converted text (such as dialect text). Whether the word segmentation is correct has a great impact on the understanding of the sentence meaning. For example, the correct word segmentation of "Welcome to Chimelong Circus in Guangzhou City" is "Guangzhou City", "Chimelong", "Circus", "Welcome to you". If it is wrongly segmented as "Guangzhou", "Mayor", "Long Circus", "Welcome to you", it will lead to a completely incorrect sentence meaning. Prosody prediction can be divided into prosody word boundaries, prosody phrase boundaries, and intonation phrase boundaries according to, for example, the pause duration and the pitch characteristics before and after. By predicting different boundaries, the effect of cadence in the sentence can be achieved, enhancing the realism of the synthesized audio.

[0050] At box 214, graphemes to phonemes (G2P) conversion is performed to sequentially convert text into phonemes. Due to the existence of polyphonic characters in Chinese, the front-end system needs to make correct decisions on the pronunciation of each character based on the auxiliary information (such as word segmentation and词性) obtained from the previous operations and some algorithms. Using G2P conversion, the text can be converted into a sequence of phonemes. The following will be combined with Figure 2B Discuss an example process of G2P conversion.

[0051] Figure 2B Shows an example process 216 of G2P conversion in the front-end processing of Chinese dialect TTS according to some embodiments of the present disclosure.

[0052] It should be noted that there is an unclear "词性" in the original text which might be a misspelling or an incomplete expression. If this is a specific technical term, it needs to be further clarified for an accurate translation.As shown in 2B, at block 218, each single character is phonetically annotated using a dialect-specific dictionary. At block 220, the phonetic sounds predicted by the polyphonic word model overwrite the phonetic annotations in the dictionary. At block 222, the phonetic annotations at the entry level based on the dictionary overwrite the single-character phonetic annotations. At block 224, the tone sandhi rules of different dialects are used to overwrite the phonetic annotations. For example, in the embodiment where the text 125 is a Northeast dialect text, in Northeast dialect, "一", "七", "八" are pronounced as "yi2", "qi2", "ba2" respectively before the fourth tone, while "还", "没", "别" are pronounced as "hai4", "mei4", "bie4" respectively before the first, second, and third tones. At block 226, post-correction rules are proposed based on the audio with poor synthesis effect to correct the phonetic annotations.

[0053] Next, continuing to refer to FIG. 2, after the G2P conversion is performed, at block 228, the TTS front-end result is output. Through process 200, a Mandarin text can be converted into a dialect text with correct orthography and grammar, thereby improving the intelligibility and naturalness of the synthesized speech.

[0054] It should be understood that, merely for illustrative purposes and without any intention of imposing any limitations, in FIG. 2, the text conversion system 115 is shown applied to the front-end processing of Chinese dialect TTS and uses the conversion model 130 to perform text conversion. In some embodiments, the text conversion system 115 can be used in the backend of automatic speech recognition processing. For example, after an English speech is recognized into an English text, the text conversion system 115 can be used to convert the English text into a Chinese text. Moreover, the text conversion system 115 can adopt other appropriate ways to perform text conversion.

[0055] To improve the performance and effect of text conversion, some embodiments of the present disclosure propose a text conversion mechanism for use by the text conversion system 115. According to this mechanism, in the process of converting the original text in the first language into the converted text in the second language, not only the semantic features of each text unit (e.g., character) in the original text are considered, but also the feature of whether each text unit is a boundary of a text unit group (e.g., word) is considered. Thus, in the conversion process, the semantic information at the text unit and text unit group levels can be considered together, improving the accuracy of the converted text.

[0056] The following will be combined with Figure 3 and Figure 4 to discuss some embodiments in this regard.

[0057] Figure 3 FIG. shows a text processing process 300 according to some embodiments of the present disclosure. The process 300 can be implemented at the text conversion system 115 in Figure 1 For ease of discussion, the process 300 will be described in conjunction with the environment 100 of Figure 1

[0058] In box 305, multiple feature representations (referred to as "first feature representations") are generated from multiple text units in the original text 120 based on the first language. These first feature representations indicate the semantics of the corresponding text units. The first feature representations can be generated at any stage of processing the original text. For example, in an embodiment that uses the transformation model 130 to perform text transformation, the first feature representation can be an intermediate embedding representation of the original text computed through self-attention.

[0059] In box 310, multiple first feature representations are combined with another feature representation (referred to as "second feature representations") to form a feature representation sequence. The second feature representation indicates whether each text unit is a boundary of a group of text units. For example, the number "1" can be used to indicate that the corresponding text unit is a word segmentation boundary, and the number "0" can be used to indicate that the corresponding text unit is not a boundary. Thus, the length of the resulting second feature representation is equal to the number of first feature representations.

[0060] The second feature representation can be implemented in any suitable form. For example, in an embodiment where the first feature representation is an intermediate embedded representation of the original text computed through self-attention, the second feature representation can also be implemented as an embedded representation so that it can be concatenated with the first feature representation, for example, concatenated after the first feature representation to form a sequence of feature representations.

[0061] Considering the example where text units are characters and text unit groups are words, during text conversion, a second feature representation indicating whether each text unit is a boundary of a text unit group (e.g., a word segmentation boundary) can be additionally appended after the first feature representation extracted from the original text that indicates the semantics of each text unit (e.g., character level). This allows for a combination of semantic considerations at both the text unit level (e.g., character level) and the text unit group level (e.g., word level) during text conversion, improving the accuracy of the conversion.

[0062] In box 315, a second language converted text 125 is generated based on the feature representation sequence formed by combining the first feature representation and the second feature representation.

[0063] To further improve the efficiency of text conversion, in some embodiments, during the generation of the first feature representation of each text unit, the text units can be grouped and weighted using attention weights. For example, the vector representations of each text unit can be divided into multiple vector representation groups. For a vector representation group, the attention weights of each vector representation are determined, and the first feature representation of each vector representation is determined by weighting each vector representation using these attention weights. The vector representation can be a representation of the corresponding text unit generated at any stage of the original text processing. For example, the vector representation can be an embedded representation in vector form generated during the initialization of the original text.

[0064] Using a multi-branch strategy to compute self-attention can reduce the computational cost of calculating attention. For example, if the number of branches is n (where n is any suitable integer), computing attention for each branch separately reduces the total computational cost to 1 / n.

[0065] In some embodiments, before performing text conversion, predetermined characters or strings can be replaced with predetermined markers so that some irregular characters or strings that cannot be converted can be output as is. In these embodiments, both the original text 120 and the converted text 125 may include predetermined markers corresponding to at least one predetermined character or string. In this case, the predetermined markers in the converted text 125 can be restored to the predetermined characters or strings. By replacing the predetermined markers with the original characters or strings again, the original content expected by the user can be preserved.

[0066] As an example, the original user input is the Mandarin text "Why aren't you logging into QQ?", which can be expressed using a predefined tag. <rep>The string "QQ" was replaced and preprocessed to "Why don't I log in?" <rep>What's going on?”, as the original text 120. After text conversion, the Cantonese text "Dim gaai m唔login <rep>"Huh?", as the converted text 125. Finally, " <rep>Replacing it with "QQ" will produce the final output "Why don't you log in to QQ?".

[0067] To further improve text conversion efficiency, in some embodiments, the text conversion system 115 can use a non-autoregressive machine learning model to perform text conversion. For example, Figure 1 The transformation model 130 shown can be based on a lightweight non-autoregressive model such as GLAT (Glancing Transformer) to further improve inference speed, thereby better adapting to text conversion tasks in dialect TTS front-ends or other text conversion tasks with high timeliness requirements. The following will combine... Figure 4 This paper discusses some examples of using non-autoregressive machine learning models for text transformation.

[0068] Figure 4 An example architecture 400 for a conversion model 130 for text conversion according to some embodiments of the present disclosure is shown.

[0069] In architecture 400, the transformation model 130 generally includes the following four parts: a multi-branch Transformer encoder 402, a multi-branch non-autoregressive Transformer decoder 404, a length predictor 406 that determines when to stop predicting, and a translation predictor 408 for decoding intermediate representations into text translation results.

[0070] like Figure 4 As shown, the conversion model 130 first receives the original text 120 as input. As an example, the original text 120 is the Mandarin text "Then I'll call you now." The initialization vector of the original text 120 is represented as a word-level embedding representation 410, which is then fed into the encoder 402 after adding position encoding 412.

[0071] Encoder 402 computes multi-head self-attention 414 through multiple branches and concatenates the resulting attention through feedforward network 416 to compute intermediate embedding representation 418. Employing a multi-branch strategy to compute self-attention reduces the computational cost of attention computation in feedforward network 416 for model 130. For example, assuming the number of branches is n, encoder 402 divides the input word-level embedding representation 410 into n branches, computes attention for each branch separately, and then concatenates them together through feedforward network 416. Thus, the overall computational cost of feedforward network 416 is reduced to 1 / n.

[0072] An embedded representation 420 that adds word segmentation boundary information is embedded in the middle of the representation 418. Each character has a corresponding word segmentation boundary indication, and thus a word segmentation embedding sequence of the same length as the sequence of the character-level embedded representation 410 of the input is obtained. For example, the word segmentation result of "那我现在给你打电话了。” is "那|我|现在|给|你|打电话|了|。|”。 If 1 is used to represent the word segmentation boundary and 0 is used to represent the non-boundary, the word segmentation embedding sequence is "11101110011”. As Figure 4 shown, the word segmentation embedded representation 420 is concatenated as a separate layer of embedded representation after the intermediate embedded representation 418 generated by the encoder 402. Subsequently, the original dimension can be restored through a fully connected layer as the final embedded representation 422.

[0073] Since the model 130 embeds the original text at the single character level, adding additional word segmentation boundary information to the intermediate embedded representation 418 generated by the encoder 402 will help the model 130 understand the semantic information at the word level, and thus the accuracy of text conversion can be further improved.

[0074] The length predictor 406 receives the attention 424 between the encoder and the decoder, calculates the length information of the target statement, and inputs it to the decoder 404. The decoder 404 receives the intermediate embedded representation 422 and the length information 426, and uses the same multi-branch calculation attention 428. Finally, the translation predictor 408 receives the output of the decoder 404 and maps it to the converted text 125 in combination with the dictionary. For example, the Cantonese text "咁我而家打电话俾你啦。”

[0075] As Figure 4 As shown, the architecture 400 also utilizes a random sampling strategy 430. According to this strategy, during the process of training the model 130, the decoder 404 performs two-step decoding, while performing one-step decoding during the prediction or conversion process. In the first step of the two-step decoding, the decoder 404 directly decodes, estimates the number of target words that the model 130 needs to see based on the quality of the output statement, and then samples the corresponding number of target words and replaces them into the input of the decoder 404. In the second step, during the decoding process of the decoder 404, the input replaced with the target words is used to enable the model 130 to learn the output of the remaining words. For example, after inputting "Then I'll call you now.", the conversion result of the first step is "So I'm calling you", and after comparing this result with the target statement "So I'm calling you now.", it is found that the current result is poor. Therefore, the words "now", "you", "call", and "you" are randomly sampled, and their corresponding word-level intermediate embedding representations in vector form are replaced into the input of the decoder 404 at the corresponding positions. During the second-step decoding process, the decoder 404 needs to predict the remaining words and compare them with "So", "I", "call", "phone", "you", "now", ".", so as to further train the model 130. Thus, a high-performance conversion model 130 based on a non-autoregressive neural network is realized.

[0076] To enhance the inference performance of the conversion model 130, some embodiments of the present disclosure propose a model training scheme. The following will be combined with Figure 5 、 Figure 6A and Figure 6B Discuss some embodiments in this regard.

[0077] Figure 5 Illustrates a model training process 500 according to some embodiments of the present disclosure. The process �00 can be implemented at the electronic device 110 shown in Figure 1 or at other devices physically separated from the electronic device 110, or partly at the electronic device 110 and partly at other devices. For ease of discussion, the process 500 will be described in conjunction with the Figure 1 environment 100.

[0078] As Figure 5 shown, at block 505, training text pairs are obtained, including the original text in the first language and the target text in the second language.

[0079] In box 510, the original text is converted into translated text in the second language using conversion model 130. According to some embodiments of this disclosure, during the text conversion process, multiple first feature representations are generated based on multiple text units in the original text, each indicating the semantics of its corresponding text unit. These first feature representations are combined with second feature representations indicating whether each text unit is a boundary of a text unit group to form a feature representation sequence. The translated text is generated based on this feature representation sequence.

[0080] In this way, the trained model 130 can learn semantic information at the text unit level (e.g., word level) and the text unit group level (e.g., word level), which helps it understand the semantics of the original text at the text unit and text unit group levels, thereby improving the prediction performance of model 130.

[0081] To further improve model training efficiency, in some embodiments, during the generation of the first feature representation of each text unit, attention weights can be applied to group the text units. For example, the vector representations of each text unit can be divided into multiple vector representation groups. For a vector representation group, attention weights are determined for each vector representation, and these attention weights are used to weight each vector representation to determine the first feature representation of each vector representation.

[0082] In box 515, the transformation model 130 is trained based at least on the difference between the transformed text and the target text.

[0083] In some embodiments, model 130 can be further trained based on alignment information between the transformed text and the original text. For example, target alignment information associated with the original text and the target text can be determined, and the transformation model 130 can be trained further based on the alignment loss between the predicted alignment information between the transformed text and the original text and the target alignment information. Alignment information can represent the correspondence between words in the text before and after transformation, and learning reasonable alignment information can improve the prediction performance of model 130.

[0084] The target alignment information can be obtained in any appropriate manner. For example, the GIZA++v2 tool can be used to generate the target alignment information. The alignment information obtained by the GIZA++v2 tool is typically word-dimensional. In the embodiment where the conversion model 130 uses single-character segmentation, the alignment information predicted by model 130 is character-dimensional. In this case, the generated word-dimensional target alignment information can be converted to character-dimensional alignment to calculate the alignment loss with the character-dimensional alignment information of model 130.

[0085] During the dimensional transformation process, the characters of the target word can be aligned with the first character of the original word. The following will combine... Figure 6A and Figure 6B Discuss the changes in the target alignment matrix before and after dimension conversion.

[0086] Figure 6A shows the target alignment matrix for the word dimension according to some embodiments of the present disclosure, while Figure 6B shows the target alignment matrix for the character dimension according to some embodiments of the present disclosure.

[0087] As shown, the Mandarin text "What should I eat early in the morning?" as the original text is converted into the Cantonese text "What should I eat after waking up early in the morning?" as the target text. In the alignment of the word dimension, "waking up" corresponds to the original word "getting up", while in the alignment of the character dimension, the two characters "waking" and "up" both correspond to the first character "waking" of "getting up". Thus, it is convenient to calculate the alignment loss between the target alignment information and the predicted alignment information generated by the model 130, enabling the model 130 to further understand the meaning of the original text at the word level.

[0088] The alignment loss can be calculated using any suitable algorithm. In some embodiments, the mean squared error alignment loss between the target alignment information and the predicted alignment information can be calculated to reduce the gap between the predicted alignment information and the target alignment information generated by the model 130.

[0089] In some embodiments, to enhance the training dataset, an autoregressive machine learning model can be used to obtain more training data. For example, the autoregressive model can be used to convert multiple original texts in a first language into corresponding multiple target texts in a second language, and then an enhanced training dataset can be generated based on the multiple original texts and target texts. During the training of the model 130, training text pairs are obtained from the enhanced training dataset.

[0090] During the process of generating the training dataset, a monolingual corpus of the first language (e.g., Mandarin Chinese) with different domains and sentence lengths can be collected, and the autoregressive model can be used to translate all the original texts in the first language into target texts in the second language (e.g., target dialect texts) as the enhanced dataset, thereby further avoiding the prediction of repeated characters. The autoregressive model can also be pre-trained to further improve its performance.

[0091] To enable model 130 to output untranslatable irregular characters or strings (such as emojis, abbreviations, URLs, etc.) as is, in some embodiments, the training dataset can be preprocessed by replacing predetermined characters or strings (e.g., irregular characters or strings) in the original and target texts with corresponding predetermined markers to generate an updated training dataset. During the training of model 130, training text pairs are obtained from the updated training dataset. For example, regular expressions can be used to match irregular characters or strings, and each character or string can be replaced with a predetermined marker in turn. In this way, model 130 can learn the correspondence between predetermined markers between the original and target texts.

[0092] To further improve model training efficiency, in some embodiments, during the generation of the first feature representation of each text unit, attention weights can be applied to group the text units. For example, the vector representations of each text unit can be divided into multiple vector representation groups. For a vector representation group, the attention weights of each vector representation are determined, and the first feature representation of each vector representation is obtained by weighting accordingly. The vector representation can be an embedded representation in vector form obtained during the initialization of the original text or a representation generated in other stages of the original text processing to represent the corresponding text unit.

[0093] Figure 7 A schematic structural block diagram of a text processing apparatus 700 according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in an electronic device 110. Various modules / components in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0094] As shown, the apparatus 700 includes a feature generation module 710, a feature combining module 720, and a conversion module 730. The feature generation module 710 is configured to generate multiple first feature representations based on multiple text units in the original text of a first language, wherein the multiple first feature representations indicate the semantics of the corresponding text units. The feature combining module 720 is configured to combine the multiple first feature representations with second feature representations to form a feature representation sequence, wherein the second feature representations indicate whether each text unit in the multiple text units is a boundary of a text unit group. The conversion module 730 is configured to generate converted text in a second language based on the feature representation sequence.

[0095] In some embodiments, the feature generation module 710 may be further configured to: obtain vector representations of multiple text units; divide the vector representations into multiple vector representation groups; and for one of the multiple vector representation groups, determine the attention weights of each vector representation in the vector representation group, and use the determined attention weights to weight each vector representation to determine the first feature representation of each vector representation.

[0096] In some embodiments, both the original text and the converted text may include at least one predetermined marker corresponding to at least one predetermined character or string. The apparatus 700 may also include a character recovery module configured to recover at least one predetermined marker in the converted text into at least one predetermined character or string.

[0097] In some embodiments, the device 700 may operate based on a non-autoregressive machine learning model.

[0098] Figure 8 A schematic structural block diagram of a text processing apparatus 800 according to some embodiments of the present disclosure is shown. The apparatus 800 may be implemented as or included in electronic device 110, or may be implemented as or included in other devices physically separate from electronic device 110, or may be partially implemented as or included in electronic device 110 and partially implemented as or included in other devices. Various modules / components in the apparatus 800 may be implemented by hardware, software, firmware, or any combination thereof.

[0099] As shown, the device 800 includes a text acquisition module 810, a text conversion module 820, and a training module 830. The text acquisition module 810 is configured to acquire training text pairs, which include source text in a first language and target text in a second language. The text conversion module 820 is configured to convert the source text into converted text in the second language using a conversion model. The text conversion module 820 is further configured to: generate multiple first feature representations based on multiple text units in the source text, where the multiple first feature representations indicate the semantics of the corresponding text units; combine the multiple first feature representations with second feature representations to form a feature representation sequence, where the second feature representation indicates whether each text unit in the multiple text units is a boundary of a text unit group; and generate converted text based on the feature representation sequence. The training module 830 is configured to train the conversion model based at least on the differences between the converted text and the target text.

[0100] In some embodiments, the training module 830 may be further configured to: determine target alignment information associated with the original text and the target text; and further train the conversion model based on the alignment loss between the predicted alignment information between the converted text and the original text and the target alignment information.

[0101] In some embodiments, the text acquisition module 810 may be further configured to: use an autoregressive model to convert multiple source texts in a first language into multiple target texts in a corresponding second language; generate a training dataset based on the multiple source texts and multiple target texts; and acquire training text pairs from the training dataset.

[0102] In some embodiments, the text acquisition module 810 may be further configured to: replace multiple predetermined characters or strings in the original text and target text in the training dataset with multiple corresponding predetermined markers to generate an updated training dataset; and acquire training text pairs from the updated training dataset.

[0103] In some embodiments, the text conversion module 820 may be further configured to acquire multiple vector representations corresponding to multiple text units; divide the multiple vector representations into multiple vector representation groups; and for one of the multiple vector representation groups, determine the attention weight of each vector representation in the vector representation group, and use the determined attention weight to weight each vector representation to determine the first feature representation of each vector representation.

[0104] In some embodiments, the transformation model can be a non-autoregressive machine learning model.

[0105] It should be understood that the above references Figures 1 to 6B The features and effects discussed regarding processes 300 and 500 also apply to devices 700 and 800, and will not be repeated here. Furthermore, the modules included in device 700 and / or device 800 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 700 and / or device 800 can be implemented at least partially by one or more hardware logic components. Exemplary types of hardware logic components that can be used, by way of example and not limitation, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0106] Figure 9 A block diagram of an electronic device 900 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 9 The electronic device 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0107] like Figure 9 As shown, electronic device 900 is in the form of a general-purpose electronic device. Components of electronic device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage device 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 900.

[0108] Electronic device 900 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 900.

[0109] Electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0110] The communication unit 940 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 900 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0111] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 900 can also communicate with one or more external devices (not shown) via communication unit 940 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 900, or with any device that enables electronic device 900 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0112] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0113] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0114] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0115] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0117] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.< / rep> < / rep> < / rep> < / rep> < / rep>

Claims

1. A text processing method, comprising: Multiple first feature representations are generated from multiple text units in the original text of the first language, and the multiple first feature representations indicate the semantics of the corresponding text units; The plurality of first feature representations are combined with second feature representations to form a feature representation sequence, wherein the second feature representation indicates whether each text unit in the plurality of text units is a boundary of a text unit group, and the text unit group indicates a set of the plurality of text units; and Based on the aforementioned feature representation sequence, converted text in the second language is generated. The original text and the converted text both include at least one predetermined marker corresponding to at least one predetermined character or string, and the method further includes: restoring the at least one predetermined marker in the converted text to the at least one predetermined character or string, wherein the at least one predetermined character or string is a string or character that cannot be converted.

2. The method according to claim 1, wherein generating the plurality of first feature representations based on the plurality of text units comprises: Obtain the vector representation of each of the multiple text units; The vector representation is divided into multiple vector representation groups; as well as For one of the plurality of vector representation groups... Determine the attention weights of each vector representation in the vector representation group, and The determined attention weights are used to weight each vector representation to determine the first feature representation of each vector representation.

3. The method according to claim 1, wherein the method is implemented using a non-autoregressive machine learning model.

4. A text processing method, comprising: Obtain training text pairs, which include source text in the first language and target text in the second language; Using a conversion model, the original text is converted into converted text in the second language, including: Multiple first feature representations are generated based on multiple text units in the original text, and the multiple first feature representations indicate the semantics of the corresponding text units. The plurality of first feature representations are combined with second feature representations to form a feature representation sequence. The second feature representation indicates whether each text unit in the plurality of text units is the boundary of a text unit group, where the text unit group indicates the set of the plurality of text units. Based on the feature representation sequence, the converted text is generated; and The conversion model is trained based at least on the differences between the converted text and the target text. Obtaining the training text pairs includes: The training dataset is updated by replacing multiple predetermined characters or strings in the original and target texts with corresponding predetermined markers, wherein at least one predetermined character or string is an unconvertible string or character; and Obtain the training text pairs from the updated training dataset.

5. The method according to claim 4, wherein training the conversion model further comprises: Determine the target alignment information associated with the original text and the target text; as well as The conversion model is further trained based on the alignment loss between the predicted alignment information between the converted text and the original text and the target alignment information.

6. The method of claim 4, wherein obtaining the training text pair comprises: Using an autoregressive model, multiple source texts in the first language are converted into multiple target texts in the second language. A training dataset is generated based on the multiple source texts and the multiple target texts; as well as Obtain the training text pairs from the training dataset.

7. The method of claim 4, wherein generating the plurality of first feature representations based on the plurality of text units comprises: Obtain the corresponding vector representations of the multiple text units; Divide the multiple vector representations into multiple vector representation groups; as well as For one of the plurality of vector representation groups... Determine the attention weights of each vector representation in the vector representation group, and The determined attention weights are used to weight each vector representation to determine the first feature representation of each vector representation.

8. The method according to claim 4, wherein the transformation model is a non-autoregressive machine learning model.

9. A text processing apparatus, comprising: The feature generation module is configured to generate multiple first feature representations based on multiple text units in the original text of the first language, wherein the multiple first feature representations indicate the semantics of the corresponding text units; The feature combining module is configured to combine the plurality of first feature representations with second feature representations to form a feature representation sequence, wherein the second feature representation indicates whether each text unit in the plurality of text units is the boundary of a text unit group, and the text unit group indicates the set of the plurality of text units; as well as The conversion module is configured to generate converted text in the second language based on the feature representation sequence. The original text and the converted text both include at least one predetermined marker corresponding to at least one predetermined character or string, and the apparatus further includes: restoring the at least one predetermined marker in the converted text to the at least one predetermined character or string, wherein the at least one predetermined character or string is a string or character that cannot be converted.

10. A text processing apparatus, comprising: The text acquisition module is configured to acquire training text pairs, which include source text in a first language and target text in a second language. A text conversion module is configured to convert the original text into converted text in the second language using a conversion model, and the text conversion module is further configured to: Multiple first feature representations are generated based on multiple text units in the original text, and the multiple first feature representations indicate the semantics of the corresponding text units. The plurality of first feature representations are combined with second feature representations to form a feature representation sequence, wherein the second feature representation indicates whether each text unit in the plurality of text units is the boundary of a text unit group, and the text unit group indicates the set of the plurality of text units; as well as The converted text is generated based on the feature representation sequence; as well as The training module is configured to train the conversion model based at least on the difference between the converted text and the target text. Obtaining the training text pairs includes: The original text and target text in the training dataset are replaced with multiple predetermined characters or strings to generate an updated training dataset. The at least one predetermined character or string is a string or character that cannot be converted. as well as Obtain the training text pairs from the updated training dataset.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 3 or claims 4 to 8 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 3 or 4 to 8.

Images