Method, apparatus, device and storage medium for speech processing

Abstract

According to an embodiment of the present disclosure, a method, an apparatus, a device and a storage medium for speech processing are provided. In the method, for each original text unit of a plurality of original text units in an original language, a first timestamp of a corresponding speech unit is determined, the plurality of original text units being generated based on an original speech in the original language; alignment information of the plurality of original text units and a plurality of target text units in a target language is determined, the plurality of target text units being translated from the plurality of original text units; based on the first timestamp and the alignment information, a second timestamp of a corresponding speech unit is determined for each target text unit of the plurality of target text units; and a target speech in the target language is generated based on the plurality of target text units and the second timestamp. In this way, the efficiency and effect of speech conversion are improved, and the user experience is improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatuses, devices, and computer-readable storage media for speech processing. Background Technology

[0002] Speech synthesis technology can be used to synthesize speech from user-provided text. The synthesized speech can then be used for various activities such as dubbing, reading aloud, and interactive activities. Dubbing and simultaneous interpretation are two common application scenarios for speech synthesis. In the production of dubbed films, the original audio in one language (e.g., English) is usually translated and recorded in another language (e.g., Chinese), and the original audio is then overlaid. Simultaneous interpretation is similar to dubbing; it involves translating the original audio, and then a translator reads the audio in the target language. Both scenarios require the participation of translators and have high real-time requirements. Summary of the Invention

[0003] In a first aspect of this disclosure, a speech processing method is provided. The method includes determining a first timestamp for each of a plurality of original text units in an original language, the plurality of original text units being generated based on original speech in the original language; determining alignment information between the plurality of original text units and a plurality of target text units in a target language, the plurality of target text units being translated from the plurality of original text units; determining a second timestamp for each of the target text units in the plurality of target text units based on the first timestamp and the alignment information; and generating target speech in the target language based on the plurality of target text units and the second timestamp.

[0004] In a second aspect of this disclosure, a speech processing apparatus is provided. The apparatus includes a first timing module configured to determine a first timestamp for each of a plurality of original text units in an original language, the plurality of original text units being generated based on original speech in the original language; an alignment module configured to determine alignment information between the plurality of original text units and a plurality of target text units in a target language, the plurality of target text units being translated from the plurality of original text units; a second timing module configured to determine a second timestamp for each of the plurality of target text units based on the first timestamp and the alignment information; and a text-to-speech module configured to generate target speech in the target language based on the plurality of target text units and the second timestamp.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. A computer program is stored on the medium, which, when executed by a processor, implements the method of the first aspect.

[0007] It should be understood that the content described in this section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0010] Figure 2 A flowchart of a speech processing procedure according to some embodiments of the present disclosure is shown;

[0011] Figure 3A An example user interface for presenting a corresponding first timestamp of the original text unit is shown according to some embodiments of the present disclosure;

[0012] Figure 3B An example user interface for presenting alignment information between a target text cell and the original text cell, according to some embodiments of the present disclosure, is shown;

[0013] Figure 4 A flowchart illustrating an example process for creating a video according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram of a text processing apparatus according to some embodiments of the present disclosure is shown; and

[0015] Figure 6 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] The term "text unit" as used herein refers to the basic unit of text processing, the granularity of which can be changed and set according to the application. For example, a text unit can include words, phrases, characters, combinations thereof, or any other unit that appears in natural language expressions. In some examples, a text unit is also referred to as a token.

[0019] The term "text unit group" or "a set of text units" as used herein refers to a collection of multiple text units. For example, in the case where the text units are Chinese characters, a text unit group may include words, phrases, or sentences, etc.

[0020] As used herein, the term "speech unit" refers to the basic unit of speech processing, the granularity of which can be changed and set according to the application. For example, a speech unit can include phonemes, syllables, and so on. Phonemes include, for example, initials and finals.

[0021] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0022] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0023] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0024] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0025] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0026] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0027] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0028] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0029] As mentioned above, the production of traditional dubbed films and the simultaneous interpretation process usually require the involvement of voice actors or translators. Furthermore, some scenes in dubbed films and simultaneous interpretation require the translator to maintain a consistent voice and style with the original speaker; for example, selecting a speaker with a similar voice, performing style matching or transfer, etc. This further increases the difficulty of manual operation.

[0030] Embodiments of this disclosure provide a cross-language speech conversion scheme. Using this scheme, after recognizing the original speech as text (referred to as "original text"), a timestamp (referred to as "first timestamp") is determined for each original text unit and its corresponding speech unit. As an example, in a scenario where the original speech is Mandarin Chinese, the original text can be Mandarin text, the original text unit can be a character in the Mandarin text, and its corresponding speech unit can be the phoneme of each character. Furthermore, according to this speech conversion scheme, after translating the original text into text in a target language (e.g., a Chinese dialect) (referred to as "target text"), alignment information between the original text units and the target text units (e.g., characters in the Chinese dialect text) is determined. Based on this alignment information and the first timestamp of the corresponding speech unit of the original text unit, a timestamp (referred to as "second timestamp") of the corresponding speech unit (e.g., a phoneme) of each target text unit is determined. Then, based on the second timestamp of the corresponding speech unit of each target text unit, the target text is converted into target speech.

[0031] Based on the speech conversion scheme according to embodiments of this disclosure, the converted target speech closely resembles the style of the original speech, improving the speech conversion effect and thus enhancing the user's viewing experience. Furthermore, this scheme can automatically convert between different languages ​​without the need for voice-over or translator involvement, thereby improving speech conversion efficiency. Additionally, since it eliminates the need for voice-over or translator involvement, it allows for the flexible use of various timbres and styles, increasing entertainment value and improving the user experience.

[0032] The above scheme can be applied to any scenario requiring the conversion between different languages. Consider the example of using a Chinese dialect to dub a video with Mandarin Chinese voiceover. After obtaining the original text from the original audio (e.g., the original voice audio) through text recognition, the original text unit sequence can be converted into a speech unit sequence (e.g., a phoneme sequence), and the timestamp of each speech unit can be determined. After translating the original text into the target text, the alignment information between the original and target texts (e.g., the corresponding word segmentation boundaries) is determined. Then, based on the timestamps of the speech units in the original text and the alignment information after translation, the timestamps of the speech unit sequence (e.g., the phoneme sequence) of the target text are determined. This yields more accurate timestamp information for the target text, resulting in a more natural and closer-to-the-original speech. The resulting video can better match the style of the original video, facilitating alignment with the speed and lip movements of the original video.

[0033] Figure 1 An example environment 100 in which embodiments of the present disclosure can be implemented is shown.

[0034] like Figure 1 As shown, example environment 100 includes electronic device 110 configured to process source speech 112 of a source language to generate target speech 114 of a target language, thereby achieving speech conversion between different languages. The source and target languages ​​can be any suitable languages. As an example, the source language can be Mandarin Chinese, and the target language can be a Chinese dialect. As another example, the source language can be English, and the target language can be Chinese.

[0035] Electronic device 110 can be a terminal device or a server device. Terminal devices can be various portable or fixed terminals, such as tablets, desktop computers, laptops, mobile phones, and general electronic devices. Server devices can be centralized servers, distributed servers, mainframes, edge electronic devices, or cloud computing.

[0036] In example environment 100, electronic device 110 includes a speech recognition unit 116 configured to convert source speech 112 into source text 118. Any speech recognition technology can be used for speech-to-text conversion. In some embodiments, a machine learning model can be used to implement the speech recognition task, and this machine learning model can employ any suitable architecture. During the process of text recognition of source speech 112, speech recognition unit 116 can perform text segmentation, such as segmenting the source text into sentences, phrases, and other source text unit groups, and can determine the first timestamp of each source text unit, such as a phoneme.

[0037] The electronic device 110 also includes a text translation unit 120 configured to translate the identified source text 118 into target text 122 in the target language. In some embodiments, a machine learning-based translation model can be used to transcribe the text content. Other text translation methods are also possible. The text translation unit 120 can also provide alignment information between the target text 122 and the source text 118. This alignment information, together with the corresponding first timestamp of each source text unit identified by the speech recognition unit 116, can be used to determine the second timestamp of the corresponding speech unit of each target text unit for generating target speech 114.

[0038] like Figure 1 As shown, the electronic device 110 also includes a text-to-speech unit 124, which is configured to convert target text 122 into target speech 114. The text-to-speech unit 124 can convert target text 122 into target speech 114 based on the corresponding second timestamp of each target text unit. Any suitable speech synthesis technology can be used to achieve the text-to-speech conversion.

[0039] It should be understood that Figure 1 The various units in the electronic device 110 shown herein, as well as the units, modules, components, and / or parts in the other figures described below, can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, units 116, 120, and 124 in the electronic device 110 can utilize machine learning models to implement corresponding operations. It should also be understood that... Figure 1 The structure of the electronic device 110 shown is merely illustrative and not limiting. Depending on the specific implementation, the electronic device 110 may also include any other suitable units, modules, components, and / or parts to achieve the corresponding functions.

[0040] Figure 2 A speech processing procedure 200 according to some embodiments of the present disclosure is illustrated. Procedure 200 can be implemented at electronic device 110. For ease of discussion, it will be combined with... Figure 1 The environment 100 is used to describe the process 200.

[0041] In box 210, for each original text unit among multiple original text units of the original language (e.g., Mandarin Chinese), the first timestamp of the corresponding speech unit is determined. The original text unit is based on the original speech of the original language (e.g., Mandarin Chinese). Figure 1The original speech (112) is generated. In some embodiments, a machine learning-based speech recognition model can be used to perform text recognition and sentence segmentation on the original speech to obtain sentence-level original text unit groups. Moreover, temporal information (e.g., timestamps) of the sentence-level text unit groups can be provided for subsequent processing. The speech recognition model can be built based on any model architecture suitable for implementing speech recognition. As examples, the speech recognition model can be based on a Connectionist Temporal Classification (CTC) model architecture, an attention-based model architecture, a Recurrent Neural Network (RNN) model architecture, and so on.

[0042] After segmenting the original text 112 into sentence-level original text unit groups, further word segmentation, character segmentation, or character-level original text unit segmentation can be performed, and the first timestamp of the corresponding speech unit (e.g., syllable or phoneme) of each original text unit can be determined. For example, in an embodiment where the original language is Mandarin Chinese, the identified character-level text units can be analyzed using pinyin based on the audio, and the first timestamp of the phoneme-level speech unit can be determined. Character-level text units can be sequentially converted into phoneme-level speech units through graphemes to phonemes (G2P) conversion. The pronunciation of each character can be determined based on a dictionary and some algorithms. In this way, the final target speech can be aligned with the original speech at the speech unit level, which helps to match the style of the original speech and the speed of alignment.

[0043] In box 220, alignment information is determined between multiple source text units and multiple target text units in the target language. The target text units are translated from the source text units. In some embodiments, a machine learning-based translation model can be used for text conversion. The translation model can be built based on any model architecture suitable for implementing text conversion, such as the Transformer model.

[0044] In some embodiments, alignment information between target and source text units can be determined during text conversion. This alignment information can be determined by segmenting both the source and target texts. For example, word segmentation information can be added as input and prediction to the translation model. In this way, the translation model predicts word segmentation boundaries while predicting the translated text. Correct word segmentation significantly impacts sentence comprehension; incorrect segmentation can sometimes lead to completely incorrect meaning. By using word segmentation prediction, semantic information at the word segmentation level can be considered during text translation, thereby further improving the accuracy of the translated text and enhancing the performance and effectiveness of text translation.

[0045] For example, in an embodiment where the source text is in Mandarin Chinese and the target text is in Northeast Chinese dialect, the input to the translation model can be the Mandarin text and the word segmentation "你 / 在 / 那里 / 干什么 / 呀 / ? / ", and its output can be the Northeast Chinese dialect text and the word segmentation "你 / 搁 / 那儿 / 干哈 / 呢 / ? / ". By predicting the word segmentation boundaries, the translation model can output alignment information while outputting the target text, that is, "你" → "你", "在" → "搁", "那里" → "那儿", "干什么" → "干哈", "呀" → "呢", "?" → "?". In this way, the alignment information of the text before and after conversion can be provided for subsequent operations.

[0046] At block 230, based on the first timestamp of the corresponding speech unit of the source text unit and the alignment information between the source text unit and the target text unit, for each target text unit in the multiple target text units, determine the second timestamp of the corresponding speech unit. For example, the corresponding speech units of the source text unit and the target text unit can be phonemes. Using this second timestamp, a more natural and source-speech-style-close target speech can be synthesized.

[0047] For example, in an embodiment where the source text is the Mandarin Chinese text "你 / 在 / 那里 / 干什么 / 呀 / ? / ", the translated Northeast Chinese dialect text "你 / 搁 / 那儿 / 干哈 / 呢 / ? / " can calculate the corresponding second timestamp of the target text unit based on the first timestamp of the corresponding phoneme-level speech unit of each character-level source text unit and the obtained alignment information. For example, "你" in the target text can obtain the timestamp of "你" in the source text, and "搁" in the target text can obtain the timestamp of "在" in the source text.

[0048] In an embodiment where the alignment information between the target text unit and the source text unit is determined based on word segmentation, there may be word segmentations where the number of words in the source text and the target text do not match. In some embodiments, the translated target text can be converted into a reference speech in the target language, and based on the reference timestamps of each reference speech unit, generate the corresponding second timestamp of the target text unit to further improve the accuracy of the determined second timestamp.

[0049] In some embodiments, the reference timestamps of the corresponding reference speech unit group can be adjusted based on the comparison of the number of text units in the source text unit group and the target text unit group that are mutually aligned determined by word segmentation. For example, based on the number of text units in the source text unit group and the target text unit group, and the corresponding first timestamp of each source text unit in the source text unit group, adjust the corresponding reference timestamps of each target text unit in the target text unit group. Thereby, the accuracy of the timestamp of the target text word segmentation that does not match the number of words in the corresponding word segmentation of the source text can be improved.

[0050] For example, in an embodiment where the original text is a Mandarin Chinese text "What are you doing there?" and the target text is a Northeast dialect text "What are you doing there?", for a pair of aligned original text units "What are you doing" and a set of target text units "What are you doing", the Northeast dialect voice "What are you doing there?" can be synthesized through text-to-speech processing first. In this way, the timestamp of "What are you doing" can be obtained. Subsequently, based on the number of character-level text units included in the original text unit group "What are you doing", the overall duration of "What are you doing" can be proportionally segmented to obtain the corresponding duration of each text unit in the target text unit group "What are you doing", and then the obtained timestamp of "What are you doing" can be adjusted. In this way, the corresponding voice units of the translated target text and the original text can ensure that the timestamps are as close as possible, so as to be closer to the style of the original voice further.

[0051] If the number of text units (e.g., the number of characters) in the corresponding original text unit group and the target text unit group differ significantly, the corresponding duration of the target text unit obtained by proportionally segmenting the overall duration of the original text unit group according to the number of target text units may be very short, for example, below a certain threshold. In this case, the duration of the target text unit group can borrow a small amount of duration from the adjacent target text units or target text unit groups (e.g., words or phrases). If the corresponding duration of the target text unit is very long, for example, above a specific threshold, a part of the duration of the target text unit group can be allocated to the adjacent target text units or target text unit groups. In this way, the final target voice can be made as natural as possible and as close to the original voice as possible.

[0052] In some embodiments, during the voice conversion process between different languages, an interface that can be intervened by the user can be provided, so that the user can intervene in the processing process of each stage in real time and modify the corresponding processing results. In some embodiments, the user can intervene in the process of determining the corresponding first timestamp of the original text unit. For example, the corresponding first timestamp of the original text unit can be presented on the user interface. The first timestamp can be automatically presented after being determined, or can be presented in response to the user's request. After receiving the modification of the first timestamp by the user (referred to as "the first modification"), the first timestamp can be adjusted accordingly. The following will combine Figure 3A to discuss an example process of modifying the first timestamp.

[0053] Figure 3A FIG. 300 shows an example user interface for presenting the corresponding first timestamp of the original text unit according to an embodiment of the present disclosure.

[0054] In this example, the recognized original text is the Chinese Mandarin sentence "What are you doing there?", and the original text units are the characters therein.

[0055] As Figure 3A shown, in the user interface 300, the identifiers 305-1 to 305-7 (individually or collectively referred to as "identifiers 305") of the corresponding first timestamps of each character-level text unit are presented. The user can adjust the positions of the identifiers 305 by performing predetermined operations such as clicking and dragging on the respective identifiers 305, thereby modifying the first timestamp. For example, the user can drag the identifier 305-5 corresponding to the original text unit "gan" left or right. If dragged to the right, the duration of "gan" is correspondingly increased; if dragged to the left, its duration is decreased. In addition to performing predetermined operations on the identifiers 305, the user can also modify the first timestamp of the original text unit in other ways. For example, a control panel (not shown) can be arranged in the user interface 300 for the user to directly set the values of the corresponding first timestamps or the corresponding durations of each original text unit.

[0056] By modifying the first timestamp of the original text unit, the corresponding second timestamp of the target text unit subsequently generated through translation can be correspondingly changed. Alternatively or additionally, in some embodiments, the user can directly modify the second timestamp of the target text unit. For example, the second timestamp can be presented on the user interface. After receiving the modification of the second timestamp by the user (referred to as the "second modification"), the second timestamp is adjusted based on the second modification. Similar to the modification of the first timestamp, the user can modify the second timestamp by performing a predetermined operation on the corresponding identifier of the second timestamp, or can directly set the value of the second timestamp or the corresponding duration of the target text unit.

[0057] Alternatively or additionally, in some embodiments, the user can intervene in the alignment information between the target text unit and the original text unit. For example, the alignment information can be presented on the user interface. After receiving the modification of the alignment information by the user (referred to as the "third modification"), the alignment information is adjusted based on the third modification. The following will Figure 3B discuss an example process of modifying the alignment information.

[0058] Figure 3B FIG. shows an example user interface 310 for presenting the alignment information between the target text unit and the original text unit according to an embodiment of the present disclosure.

[0059] In this example, a set of original text units 315 recognized is the Mandarin sentence "What are you doing there?", and a set of translated target text units 320 is the Northeast dialect sentence "What are you doing there?".

[0060] In the user interface 310, alignment information 325-1 to 325-4 at each word level of the original text unit and the target text unit is presented (individually or collectively referred to as "alignment information 325"). As Figure 3B shown, the original text "干什么" is misaligned with the target text "干". In this case, the user can manually drag the word segmenter 330 to align the original text "干什么" with the target text "干哈". In this way, more accurate alignment information can be obtained, and based on such alignment information, the second timestamp of the target text unit can be determined and the target voice can be generated subsequently, which can further improve the naturalness of speech conversion between different languages.

[0061] The speech conversion scheme between different languages according to the embodiments of the present disclosure can be applied to any suitable scenario. In some embodiments, this scheme can be used in the video creation process of users.

[0062] Figure 4 FIG. 400 shows an example process for creating a video according to some embodiments of the present disclosure.

[0063] In this example, the input of process 400 is the original video, and after a series of processes, its output is the produced video. Additionally, the original voice is Mandarin Chinese, and the target voice is a dialect. The original text unit is a Chinese character in Mandarin, the original text unit group is a Mandarin sentence, the target text unit is a Chinese character in the dialect, and the corresponding speech units of the original text unit and the target text unit are phonemes.

[0064] As Figure 4 shown, at block 402, the original video is processed. At block 404, the original video image is generated. At block 406, the original video audio is generated. At block 408, the background music (abbreviated as BGM) is separated. At block 410, the original audio BGM is generated. At block 412, the original audio human voice is generated. A BGM separation model based on a neural network architecture and implemented through machine learning can be used to separate the BGM of the original video audio, obtaining the separated BGM audio and human voice audio. By separating the background music, the original video's music style can be maximally retained, and the accuracy of speech recognition can be improved.

[0065] At block 414, speech recognition is performed. A speech recognition model based on machine learning can be used. At block 416, the original text (e.g., Mandarin Chinese text) is recognized. At block 418, the timestamp information of the text clauses (as an example of the original text unit group) is generated. For example, the timestamp information of the original text unit group at the Mandarin sentence level can be obtained by sentence segmentation. In this way, text and time information can be provided for subsequent processing.

[0066] At block 420, perform voice forced alignment. For example, a machine learning-based forced alignment model can be used to perform phonetic parsing of the recognized text according to the audio, and output the first timestamp of the phoneme-level speech unit corresponding to the text unit at each word level. At block 422, generate timestamp information (as an example of the first timestamp) of the original text phonemes (examples corresponding to each original text unit). Thereby, timestamp information can be provided for subsequent target speech synthesis, so as to maximize the alignment of the speed and lip movements of the original video.

[0067] After the original text is recognized (block 416), at block 424, perform text translation. At block 426, generate the target text. At block 428, generate alignment information for the target text. A machine learning-based translation model can be used to transcribe the text content and provide alignment information for the translated text.

[0068] In an embodiment where the original text is the Mandarin text "What are you doing there?" and the target text is the Northeast dialect "What are you doing there?", the translation model can use a translation model with a sequence-to-sequence (seq2seq) architecture. In this architecture, a multi-task module for inputting and predicting word segmentation information can also be added to determine the alignment information of the translated text. For example, during the process of predicting the translated text, the translation model can also predict the word segmentation boundary. If the input is the Mandarin text and word segmentation "You / are / there / doing / what / ? / ", the output can be the Northeast dialect text and word segmentation "You / are / there / doing / what / ? / ". Finally, the translation model can output alignment information while outputting the translated text, that is, "You" → "You", "are" → "are", "there" → "there", "doing" → "doing", "what" → "what", "?" → "?". By providing the multi-task ability of aligning the translated text by the translation model, alignment information is provided for subsequent operations.

[0069] At block 430, perform phoneme duration adjustment of the target text. At block 432, determine the second timestamp of the target text unit. For example, according to the phoneme timestamp information of the original text and the translated alignment information, the timestamp information of the phoneme sequence of the target text can be calculated (as an example of the second timestamp). In this way, it is possible to ensure that the timestamps of the corresponding words of the target text and the original text are as close as possible, making the produced video closer to the style of the original video and helping with lip movement alignment.

[0070] For example, after word segmentation, for the original text "你 / 在 / 那里 / 干什么 / 呀 / ? / ", after forced alignment, each character-level original text unit can obtain the corresponding timestamp. The translated text "你 / 搁 / 那儿 / 干哈 / 呢 / ? / " can calculate the timestamp based on the alignment information obtained from word segmentation and the timestamps of the original text units. For example, the target text unit "你" can obtain the timestamp of the original text unit "你", and the target text unit "搁" can obtain the timestamp of the original text unit "在".

[0071] For the groups of original text units and target text units at the word segmentation level with different numbers of words, such as "干什么" and "干哈", the reference voice "你搁那儿干哈呢?" can be synthesized through text-to-speech processing to obtain the timestamp of "干哈". Then, according to the number of words in the target original text unit group "干哈", the overall duration of "干什么" can be divided proportionally to obtain the duration of each character-level target text unit. Through text-to-speech processing, more accurate timestamp information of phoneme-level speech units can be obtained, making the synthesized voice more natural and closer to the voice of the original video.

[0072] For the durations of phoneme-level speech units after segmentation that are lower or higher than a certain threshold due to a large difference in the number of words, the durations of a small number of adjacent word-level text units or text unit groups can be borrowed, or a part of the duration can be allocated to adjacent word-level text units or text unit groups, thereby further making the synthesized voice as natural as possible and as close as possible to the voice of the original video.

[0073] In block 434, text-to-speech conversion is performed. In block 436, the target voice is synthesized. In an embodiment where the original voice is Mandarin and the target voice is Northeast dialect, a dialect text-to-speech (TTS) system can be used for text-to-speech conversion. Compared with the Mandarin TTS system, the dialect TTS system has an independent front end, including a dictionary, a lexicon, a word segmentation and part-of-speech model, a prosody model, a polyphonic character model, and tone sandhi rules. After inputting the text, it can output a dialect phoneme sequence according to the characteristics of the dialect. For example, "我媳(xi3)妇儿一餐能吃七(qi2)八(ba2)个包子", which has different phonetic notations and phoneme sequences compared with Mandarin. The dialect TTS system can support controllable duration and can synthesize a voice with the corresponding duration according to the input phoneme timestamp information.

[0074] In block 438, the synthesized voice is combined with the original video image and the original audio background music to produce a video.

[0075] In some embodiments, users can intervene in or modify the results of multiple processing stages in process 400. For example, users can manually intervene in processes such as identifying the original text (box 416), generating the target text (box 426), generating the alignment information of the target text (box 428), generating the timestamp information of the phonemes in the original text (box 422), and / or determining the timestamp of the phonemes in the target text (box 432). This can further improve the accuracy of the processing results at each stage, making the produced video more in line with user needs, adding flexibility to the speech-to-speech process, and further improving the user experience.

[0076] Figure 5 A schematic structural block diagram of a text processing apparatus 500 according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in an electronic device 110. Various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] As shown, the device 500 includes a first timing module 510, an alignment module 520, a second timing module 530, and a text-to-speech module 540. The first timing module 510 is configured to determine a first timestamp for each of a plurality of original text units in the source language, corresponding to a speech unit. The plurality of original text units are generated based on the original speech of the source language. The alignment module 520 is configured to determine alignment information between the plurality of original text units and a plurality of target text units in the target language. The plurality of target text units are translated from the plurality of original text units. The second timing module 530 is configured to determine a second timestamp for each of the plurality of target text units, corresponding to a speech unit, based on the first timestamp and the alignment information. The text-to-speech module 540 is configured to generate target speech in the target language based on the plurality of target text units and the second timestamp.

[0078] In some embodiments, the second timing module 530 may be further configured to: convert a plurality of target text units into reference speech of the target language; and determine a second timestamp based on the reference timestamp of each reference speech unit in the reference speech, according to the first timestamp and alignment information.

[0079] In some embodiments, the apparatus 500 may further include a reference adjustment module configured to: determine a set of original text units aligned with a set of target text units based on alignment information for a set of reference speech units corresponding to a set of target text units among a plurality of reference speech units; and adjust the reference timestamps of each reference speech unit in a set of reference speech units based on the number of target text units in a set of target text units and the first timestamps of the corresponding speech units of each original text unit in a set of original text units.

[0080] In some embodiments, the first timing module 510 may be further configured to: present a first timestamp on a user interface; receive a first modification of the first timestamp by the user; and adjust the first timestamp based on the first modification.

[0081] In some embodiments, the second timing module 530 may be further configured to: present a second timestamp on a user interface; receive a second modification of the second timestamp by the user; and adjust the second timestamp based on the second modification.

[0082] In some embodiments, the alignment module 520 may be further configured to: present alignment information on a user interface; receive a third modification of the alignment information by the user; and adjust the alignment information based on the third modification.

[0083] It should be understood that the above references Figures 1 to 4 The features and effects discussed regarding processes 200 and 400 also apply to device 500 and will not be repeated here. Furthermore, the modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 500 can be implemented at least partially by one or more hardware logic components. Exemplary types of hardware logic components that can be used, by way of example and not limitation, include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0084] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0085] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0086] Electronic device 600 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0087] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0088] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0089] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0090] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0091] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0092] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0093] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0094] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0095] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A speech processing method, comprising: For each original text unit in a plurality of original text units of the original language, a first timestamp of the corresponding speech unit is determined, wherein the plurality of original text units are generated based on the original speech of the original language; The alignment information of the plurality of source text units and the plurality of target text units of the target language is determined, wherein the plurality of target text units are translated from the plurality of source text units; Based on the first timestamp and the alignment information, for each target text unit among the plurality of target text units, a second timestamp of the corresponding speech unit is determined; as well as Based on the plurality of target text units and the second timestamp, the target speech of the target language is generated; Determining the second timestamp includes: The plurality of target text units are converted into reference speech of the target language; as well as Based on the first timestamp and the alignment information, the second timestamp is determined according to the reference timestamps of each reference speech unit in the reference speech.

2. The method according to claim 1, further comprising: For the set of reference speech units that correspond to a set of target text units among the plurality of reference speech units, Based on the alignment information, a set of original text units aligned with the set of target text units is determined; as well as Based on the number of target text units in the set of target text units and the first timestamp of the corresponding speech unit in each original text unit in the set of original text units, the reference timestamp of each reference speech unit in the set of reference speech units is adjusted.

3. The method according to claim 1, wherein determining the first timestamp comprises: The first timestamp is displayed on the user interface; Receive the user's first modification to the first timestamp; as well as Based on the first modification, the first timestamp is adjusted.

4. The method of claim 1, wherein determining the second timestamp comprises: The second timestamp is displayed on the user interface; Receive the user's second modification to the second timestamp; as well as Based on the second modification, the second timestamp is adjusted.

5. The method of claim 1, wherein determining the alignment information comprises: The alignment information is displayed on the user interface; Receive a third modification from the user to the alignment information; as well as Based on the third modification, the alignment information is adjusted.

6. A speech processing apparatus, comprising: The first timing module is configured to determine the first timestamp of the corresponding speech unit for each of the multiple original text units in the original language, wherein the multiple original text units are generated based on the original speech of the original language. An alignment module is configured to determine alignment information between the plurality of source text units and a plurality of target text units of a target language, the plurality of target text units being translated from the plurality of source text units; The second timing module is configured to determine the second timestamp of the corresponding speech unit for each of the plurality of target text units based on the first timestamp and the alignment information; as well as The text-to-speech module is configured to generate target speech in the target language based on the plurality of target text units and the second timestamp; The second timing module is further configured as follows: Convert the plurality of target text units into reference speech of the target language; and Based on the first timestamp and the alignment information, the second timestamp is determined according to the reference timestamps of each reference speech unit in the reference speech.

7. The apparatus according to claim 6, further comprising: The reference adjustment module is configured to target a set of reference speech units that correspond to a set of target text units among the plurality of reference speech units. Based on the alignment information, a set of original text units aligned with the set of target text units is determined; as well as Based on the number of target text units in the set of target text units and the first timestamp of the corresponding speech unit in each original text unit in the set of original text units, the reference timestamp of each reference speech unit in the set of reference speech units is adjusted.

8. The apparatus of claim 6, wherein the first timing module is further configured to: The first timestamp is displayed on the user interface; Receive the user's first modification to the first timestamp; and Based on the first modification, the first timestamp is adjusted.

9. The apparatus of claim 6, wherein the second timing module is further configured to: The second timestamp is displayed on the user interface; Receive the user's second modification to the second timestamp; and Based on the second modification, the second timestamp is adjusted.

10. The apparatus of claim 6, wherein the alignment module is further configured to: The alignment information is displayed on the user interface; Receive a third modification of the alignment information by the user; and Based on the third modification, the alignment information is adjusted.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the device to perform the method according to any one of claims 1 to 5 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images