Speech translation method and apparatus, electronic device, storage medium

Abstract

The present disclosure can provide a speech translation method and device, an electronic device, and a storage medium, wherein the speech translation method comprises: receiving first speech, the first speech being source language speech to be translated; extracting first acoustic feature information from the first speech, the first acoustic feature information being used to represent the semantics of the source language speech; converting the first acoustic feature information into frequency spectrum information of a target language, and encoding the frequency spectrum information to generate second speech; the second speech being target language speech obtained after the source language speech is translated. The speech translation scheme provided by the present disclosure can be applied to more application scenarios, the speech translation function is optimized, and the user experience is significantly improved.

Description

Technical Field

[0001] This disclosure relates to the field of speech processing technology, specifically to speech translation methods and apparatus, electronic devices, and storage media. Background Technology

[0002] With the rapid development of speech processing technology, the accuracy and intelligence level of speech translation functions have been effectively improved. In related technologies, the process generally involves first identifying the source language text corresponding to the source language speech to be translated, then translating the source language text into the corresponding target language text, and finally generating the target language speech corresponding to the target language text. The target language differs from the source language; for example, the source language is Chinese and the target language is English, or the source language is Mandarin and the target language is a dialect.

[0003] The solutions adopted by the aforementioned technologies can be applied to many scenarios, but for languages ​​that do not have a correct written form or even no written form at all, such as dialects without a correct written form, resulting in the inability to obtain language text, the relevant technologies are difficult to realize the voice translation function, which urgently needs to be solved. Summary of the Invention

[0004] To address the problem that speech translation is difficult to achieve due to the absence of a correct textual form or even no textual form in related technologies, this disclosure provides a speech translation method and apparatus, electronic device, and storage medium to solve at least one of the problems existing in related technologies.

[0005] To achieve the aforementioned technical objectives, this disclosure provides a speech translation method, the method comprising: receiving a first speech, wherein the first speech is a source language speech to be translated; extracting first acoustic feature information from the first speech, wherein the first acoustic feature information is used to represent the semantics of the source language speech; converting the first acoustic feature information into spectral information of a target language; encoding the spectral information to generate a second speech; wherein the second speech is the target language speech obtained after translating the source language speech.

[0006] To achieve the aforementioned technical objectives, this disclosure also provides a speech translation device, comprising: a speech receiving module for receiving first speech, wherein the first speech is source language speech to be translated; a feature extraction module for extracting first acoustic feature information from the first speech, wherein the first acoustic feature information is used to represent the semantics of the source language speech; an information conversion module for converting the first acoustic feature information into spectral information of a target language; and a speech encoding module for encoding the spectral information to generate second speech; wherein the second speech is target language speech obtained after translating the source language speech.

[0007] To achieve the above-mentioned technical objectives, this disclosure may also provide an electronic device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor causes the processor to perform the speech translation method described in any embodiment of this disclosure.

[0008] To achieve the above-mentioned technical objectives, this disclosure may also provide a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the speech translation method described in any embodiment of this disclosure.

[0009] The beneficial effects of this disclosure include: Compared with related technologies, this disclosure determines the corresponding target language's spectral information based on the first acoustic feature information extracted from the source language's speech information, and then encodes the spectral information into the target language's speech; it is evident that the entire speech translation process of this disclosure does not require the use of text information, and can still achieve speech translation functionality well even when the source language and / or target language lacks a correct written form or even has no written form at all. This disclosure is particularly suitable for dialect translation scenarios without a correct written form, so the speech translation solution provided by this disclosure can be applied to more application scenarios, achieving optimization of speech translation functionality and significantly improving the user experience. In addition, this disclosure can also help to significantly reduce the cost of implementing speech translation functionality. Attached Figure Description

[0010] Figure 1 A schematic flowchart of a speech translation method according to one or more embodiments of the present disclosure is shown.

[0011] Figure 2 A schematic diagram of the process for converting first acoustic feature information into spectral information of a target language is shown in one or more embodiments of this disclosure.

[0012] Figure 3 A schematic diagram of the process for translating first acoustic feature information is shown in one or more embodiments of this disclosure.

[0013] Figure 4 A schematic diagram illustrating the process of converting second acoustic feature information into spectral information of a target language in one or more embodiments of this disclosure is shown.

[0014] Figure 5 A schematic diagram of the process for generating spectral information of a target language using second semantic information is shown in one or more embodiments of this disclosure.

[0015] Figure 6 A schematic diagram of the process for extracting first acoustic feature information from a first speech in one or more embodiments of this disclosure is shown.

[0016] Figure 7 FIG. 1 is a schematic diagram illustrating the principle of implementing the speech translation solution of the present disclosure based on a trained neural network model in one or more embodiments of the present disclosure.

[0017] Figure 8 FIG. 2 is a schematic diagram illustrating the working principle of a translator sub-module and a synthesizer sub-module in one or more embodiments of the present disclosure.

[0018] Figure 9 FIG. 3 is a schematic diagram illustrating the structure of a speech translation apparatus in one or more embodiments of the present disclosure.

[0019] Figure 10 FIG. 4 is a schematic diagram illustrating the internal structural composition of an electronic device in one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] In the related art, for a speech translation task, an ASR (Automatic Speech Recognition) system first recognizes source language text corresponding to source language speech, then a language translation system translates the source language text into corresponding target language text, and finally a TTS (Text To Speech) system can output target language speech based on the target language text.

[0021] Taking dialects as an example, in communication between different dialects, since dialects are mainly spread by speech, the written form of a dialect is generally only proficiently mastered by linguists, and most users have difficulty inputting accurate dialect text as TTS input; while for non-locals, it is often even more difficult to input accurate dialect text. Taking Cantonese as an example of dialect, most non-locals and some locals have difficulty writing the accurate written form of Cantonese. For example, to express the meaning of "I didn't do it on purpose" in Mandarin, the accurate Cantonese expression is "我唔系特登嘅", but non-locals can often only write the Mandarin text form, and some locals will only write it in a homophonic form, such as "我唔海得登嘅". Taking Northeastern dialect as an example of dialect, although the difference between Northeastern dialect and Mandarin is small, to express "It is very cold in our place" intended by non-locals, locals have a more accurate expression, such as "俺们那旮瘩老冷了". Therefore, translation between Mandarin and dialects can help non-locals and locals communicate accurately, but a great deal of professional knowledge about dialects is required; however, translation data between dialects is generally very scarce, and many dialects are so niche that they do not even have a correct written form, making it difficult for the related art to implement the speech translation function for dialects.

[0022] Therefore, the related art cannot solve the problem that automatic speech translation cannot be performed for some special uncommon dialects due to the absence of a written form. In addition, each of the ASR system, language translation system and TTS system requires a large amount of data for training, resulting in high implementation costs.

[0023] In view of this, the present disclosure can provide a speech translation method and apparatus, an electronic device, and a storage medium, so as to effectively solve at least one problem existing in the related art.

[0024] As shown Figure 1 in the drawings, one or more embodiments of the present disclosure can provide a speech translation method, and the speech translation method includes but is not limited to step S100 to step S400.

[0025] Step S100: receiving a first speech, where the first speech is a source language speech to be translated.

[0026] In specific applications, the first speech may for example be a Mandarin speech uttered by a user, such as the sentence "I did not do it on purpose". For this source language speech, the embodiments of the present disclosure can be used to translate the source language speech into a corresponding target language speech, for example, translating the Mandarin speech into a Cantonese speech.

[0027] Step S200: extracting first acoustic feature information from the first speech, where the first acoustic feature information is used to represent the semantics of the source language speech.

[0028] In this embodiment, a speech recognition model can be used to implement the acoustic feature extraction function, so as to extract the first acoustic feature information from the first speech, and certainly the implementation is not limited thereto.

[0029] Step S300: converting the first acoustic feature information into spectrum information of a target language.

[0030] The embodiments of the present disclosure directly convert the acoustic features of the source language into the spectrum of the target language, thereby skipping the text translation process that requires high costs, so as to overcome at least one problem in the related art.

[0031] Step S400: encoding the spectrum information to generate a second speech; where the second speech is the target language speech obtained after translating the source language speech.

[0032] The embodiments of the present disclosure can use a vocoder to encode the spectrum information, so as to encode the spectrum information into the second speech, and certainly the implementation is not limited thereto.

[0033] In specific applications, the second speech may for example be the Cantonese speech "I didn't do it on purpose" obtained after translation by the solution of the embodiments of the present disclosure.

[0034] In one or more embodiments of this disclosure, the spectral information of the corresponding target language is determined based on the first acoustic feature information extracted from the source language speech information, and then the spectral information is encoded into the target language speech. It is evident that the entire speech translation process of this disclosure does not require the use of text information. Even when the source language and / or target language lacks a correct written form or even has no written form, it can still achieve a relatively good speech translation function. This disclosure is particularly suitable for dialect translation scenarios without a correct written form. Therefore, the speech translation solution provided by this disclosure can be applied to more application scenarios, optimizing the speech translation function and significantly improving the user experience. Furthermore, this disclosure can also help to significantly reduce the cost of implementing speech translation functions.

[0035] like Figure 2 As shown, in one or more embodiments of this disclosure, the first acoustic feature information is converted into the spectral information of the target language, including but not limited to steps S310 to S320.

[0036] Step S310: Translate the first acoustic feature information to obtain the second acoustic feature information; the second acoustic feature information is used to represent the semantics of the target language speech.

[0037] The embodiments disclosed herein can realize the translation of the first acoustic feature information into the second acoustic feature information through the translator submodule in the end-to-end speech translation model, but are not limited thereto.

[0038] Step S320: Convert the second acoustic feature information into the spectral information of the target language.

[0039] The embodiments disclosed herein, for example, utilize a synthesizer submodule in a trained end-to-end speech translation model to convert second acoustic feature information into spectral information of the target language, but are not limited thereto.

[0040] Based on the translation of the first acoustic feature information, this embodiment can convert the first acoustic feature information used to represent the meaning of the source language speech into the second acoustic feature information used to represent the semantics of the target language speech. This embodiment can significantly improve the accuracy of the obtained target language spectral information, thereby improving the accuracy of speech translation and enhancing the performance of the speech translation function of this disclosure.

[0041] like Figure 3 As shown, in one or more embodiments of this disclosure, the first acoustic feature information is translated, including but not limited to steps S311 to S313.

[0042] Step S311: Obtain context information from the first acoustic feature information, which includes context information.

[0043] Among them, contextual information refers to the information in the first acoustic feature information that affects the first semantic information, including information on various acoustic feature changes brought about by different semantics of adjacent sentences, such as one or more of intonation, rhythm, and stress, but not limited to these.

[0044] Step S312: Extract first semantic information from first acoustic feature information based on context information. The first semantic information is used to represent the meaning expressed through the source language.

[0045] For example, contextual information can be used as the basis for distinguishing between different semantics of adjacent sentences, thereby accurately identifying the first semantic information.

[0046] This embodiment can extract the first semantic information from the first acoustic feature information based on the context information through the deep convolutional layer contained in the translator submodule of the trained end-to-end speech translation model.

[0047] Step S313: Convert the first semantic information into the second acoustic feature information.

[0048] This embodiment can convert the first semantic information into the second acoustic feature information by using the nonlinear activation function (ReLU), random deactivation layer (Dropout), and fully connected layer (Dense) contained in the translator submodule of the trained end-to-end speech translation model.

[0049] This embodiment can accurately extract the first semantic information through the obtained context information, so as to achieve a more comprehensive and accurate determination of the semantics of the source language speech, thereby ensuring that the meaning expressed by the source language can be more comprehensively and accurately represented in the second acoustic feature information.

[0050] like Figure 4 As shown, in one or more embodiments of this disclosure, the second acoustic feature information is converted into the spectral information of the target language, including but not limited to steps S321 to S322.

[0051] Step S321: Identify second semantic information from the second acoustic feature information. The second semantic information is used to represent the meaning expressed through the target language.

[0052] This embodiment uses a trained neural network model to understand the second acoustic feature information. For example, a trained long short-term memory artificial neural network unit can be used to understand the semantics contained in the second acoustic feature information, so as to identify the second semantic information from the second acoustic feature information.

[0053] Step S322: Generate spectral information of the target language using the second semantic information.

[0054] In this embodiment, the trained attention mechanism module can predict the spectral information of the target language based on the semantics contained in the second acoustic feature information.

[0055] Based on the identification of second semantic information from second acoustic feature information, an accurate understanding of the meaning expressed in the second acoustic feature information is achieved, and then a more accurate spectrum of the target language is generated on the basis of effectively understanding the meaning expressed by the second acoustic feature information.

[0056] like Figure 5 As shown, in one or more embodiments of this disclosure, the second semantic information is used to generate the spectral information of the target language, including but not limited to steps S3220 to S3221.

[0057] Step S3220: Utilize the second semantic information to predict the first prediction information used to characterize the coarse-grained spectrum.

[0058] In this embodiment, the trained attention mechanism module can predict the coarse-grained spectral information of the target language based on the semantics contained in the second acoustic feature information.

[0059] Step S3221: Utilize the first prediction information to predict the second prediction information used to characterize the fine-grained spectrum, and use the second prediction information as the spectral information of the target language.

[0060] In this embodiment, fine-grained spectral information can be predicted based on N (the specific value is set according to actual needs) one-dimensional convolutional and fully connected layers that have been trained.

[0061] This embodiment provides a two-step prediction method for the spectral information of a target language. This method includes the process of first predicting the coarse-grained spectrum and then predicting the fine-grained spectrum. Based on the above embodiment, this disclosure can effectively improve the speed of spectrum prediction while taking into account the quality of spectrum prediction, and its overall performance is better.

[0062] like Figure 6 As shown, in one or more embodiments of this disclosure, first acoustic feature information is extracted from the first speech, including but not limited to steps S210 and S220.

[0063] Step S210: Extract the original acoustic feature information from the first speech. The original acoustic feature information includes the first acoustic feature information and the third acoustic feature information. The third acoustic feature information includes timbre information and pronunciation emotion information.

[0064] This embodiment extracts the original acoustic feature information from the first speech using an ASR (Automatic Speech Recognition) model, but it is not limited to this.

[0065] Step S220: Filter the original acoustic feature information to remove the third acoustic feature information and obtain the first acoustic feature information.

[0066] The first acoustic feature information in this embodiment includes first semantic information and contextual information. The first semantic information includes, for example, speech meaning information, and the contextual information includes, for example, prosodic information.

[0067] For example, the first voice recording is made by a cheerful young man in a lively tone. After processing by this embodiment, the timbre and emotional information of the voice will be filtered out, while the meaning and rhythm information of the voice will be retained, so as to understand the meaning of the voice without being disturbed by extraneous information.

[0068] Based on the above embodiments, this disclosure can also filter out third acoustic feature information that is unrelated to the speech translation function, thereby providing more accurate first acoustic feature information for the subsequent first acoustic feature information conversion and spectrum information encoding process, improving the reliability of the subsequent processing, avoiding interference from information unrelated to the speech translation function to the subsequent processing, and significantly reducing the amount of data to be processed in the subsequent processing, thereby significantly improving the efficiency of speech translation of this disclosure.

[0069] Optionally, the source language speech is Mandarin and the target language speech is a dialect, or the source language speech is a dialect and the target language speech is Mandarin.

[0070] Of course, in the optional embodiments of this disclosure, the source language is Chinese and the target language is a foreign language, or the source language is a foreign language and the target language is Chinese. The foreign language can be one of the non-Chinese languages ​​such as English, Japanese, or French.

[0071] The embodiments disclosed herein are well applicable to speech translation between different languages, and are particularly suitable for speech translation between dialects and Mandarin, where a certain language has no correct written form or even no written form at all, and have a wide range of application scenarios.

[0072] Optionally, the spectral information of the target language is the Mel spectral information of the target language. Here, Mel spectral information represents information composed of Mel spectra.

[0073] This embodiment can encode Mel spectral information into a second speech by using a melgAN (mel Generative Adversarial Network) vocoder, thereby achieving the encoding of Mel spectral information into audio.

[0074] Based on the spectral information of the target language composed of Mel spectra, the embodiments of this disclosure can accurately represent the distribution of the semantically corresponding signal of the source language speech at different frequencies, thereby improving the accuracy of speech translation.

[0075] like Figure 7 As shown, this embodiment employs a feature extraction module to extract first acoustic feature information from the input source language speech, thereby achieving the output function of acoustic features of the corresponding source language speech. Specifically, the feature extraction module in this embodiment can be a sound feature extractor. The sound feature extractor in this embodiment is implemented using a neural network within an ASR (Automatic Speech Recognition) model based on a Conformer structure, using the intermediate result BNF (BottleNeck Features) of the ASR model as the first acoustic feature information. The ASR model may include a 72-layer neural network; for example, the first 32 layers can be used to extract the first acoustic feature information, and non-content-related information such as timbre and emotion can be filtered out.

[0076] Next, this embodiment can translate acoustic features into the spectrum of the target language using an end-to-end speech translation model (S2ST). This end-to-end speech translation model includes a translator submodule and a synthesizer submodule. The translator submodule takes the acoustic features of the source language speech as input and outputs the acoustic features (BNF) of the target language speech. The synthesizer submodule takes the acoustic features (BNF) of the target language speech as input and outputs the spectrum of the target language speech.

[0077] like Figure 8 As shown, the translator submodule may include, in sequence, a non-linear activation function (GLU), a deep convolutional layer (DLU), a non-linear activation function (ReLU), a dropout layer, an adder, a layer normalization (Layer Norm), a fully connected layer (Dense), a dropout layer, and a fully connected layer (Dense). The GLU and ReLU non-linear activation functions are used to eliminate the vanishing gradient problem. The deep convolutional layer extracts semantic information from the current input based on contextual information. The dropout layer introduces randomness to the model, improving its performance. The layer normalizes the model parameters, ensuring model stability. Both fully connected layers (Dense) are used for dimensionality adjustment. The N in the translator submodule can represent the stacking of N graphical submodules.

[0078] like Figure 8 As shown, the synthesizer submodule includes a nonlinear transform network (Pre-net), a long short-term memory artificial neural network unit (LSTM cell), an attention mechanism module, N one-dimensional convolutional layers (Conv1D×N), and a fully connected layer (Dense). This synthesizer submodule can be used to output coarse-grained spectrum, fine-grained spectrum, and stop symbols. The nonlinear transform network (Pre-net) is used for feature input processing, the long short-term memory artificial neural network unit (LSTM cell) is used to understand semantics from the acoustic features of the target language, and the attention mechanism module is used to align the input and output, that is, to align the speech information between different languages. For example, time steps 1-5 of the source language correspond to time steps 1-8 of the target language, and all represent the same meaning (e.g., all mean "today"). Of course, it is not limited to this. Additionally, StopProj indicates the prediction stop symbol, i.e., whether the sentence has reached the last time step; if so, prediction stops. FrameProj is used to predict the coarse-grained spectrum. Conv1D is used to refine the predicted coarse-grained spectrum to enrich more details. The fully connected layer (Dense) is used to adjust the dimensions and finally output the predicted fine-grained spectrum.

[0079] It should be understood that the translator submodule involved in this disclosure is specifically a trained translator submodule, and the synthesizer submodule involved is specifically a trained synthesizer submodule.

[0080] like Figure 7 As shown, this disclosure takes the training process of an end-to-end speech translation model (S2ST) for translating Cantonese from Chinese as an example to specifically illustrate the training process of the translator submodule and the synthesizer submodule included in the end-to-end speech translation model.

[0081] The training data used in the training process includes Mandarin audio corpora and Cantonese audio corpora of preset durations, for example, 1200 hours. The content expressed by the Mandarin audio corpora is the same as that expressed by the Cantonese audio corpora.

[0082] During model training, a sound feature extractor is used to extract acoustic feature information from Mandarin and Cantonese audio corpora, respectively, i.e., the process of extracting acoustic feature information from source and target language speech. The acoustic feature information from the Mandarin audio corpus is input into the translator submodule to obtain predicted acoustic feature information. This predicted acoustic feature information and the acoustic feature information from the Cantonese audio corpus are then used to determine the loss (L) of the translator submodule. 翻译器This loss can be L2 loss (mean absolute error), and it is used to determine whether the translator submodule has been trained successfully. For example, the loss (L... 翻译器 If the value is less than a first preset value, training is considered complete. This embodiment can employ a joint training method, inputting the acoustic feature information output by the trained translator submodule into the synthesizer submodule to obtain the predicted target language spectral information. Then, the loss (L) of the synthesizer submodule is determined using the predicted target language spectral information and the spectral information of the actual target speech corresponding to the Cantonese audio corpus. 合成器 This loss, for example, is L2 loss (mean absolute error), which is used to determine whether the synthesizer submodule has been trained successfully. For example, the loss (L... 合成器 If the value is less than the second preset value, the training is considered complete.

[0083] In a preferred embodiment, the L2 loss of the coarse-grained spectrum and the L2 loss of the fine-grained spectrum output by the synthesizer submodule are trained respectively, and the softmax (normalization function) cross-entropy between the stop characters predicted by the attention mechanism module and the actual stop characters corresponding to the Cantonese audio corpus is determined. Then, the synthesizer submodule training is determined to be complete after the L2 loss of the coarse-grained spectrum, the L2 loss of the fine-grained spectrum, and the softmax cross-entropy all meet the specified conditions.

[0084] Compared to conventional techniques, this embodiment uses an end-to-end speech translation model (S2ST) to translate the first acoustic feature information into the predicted target language's spectral (mel) information. Compared to existing technologies that rely on a multi-system concatenation scheme involving ASR, language translation, and TTS systems, this embodiment achieves end-to-end dialect translation, effectively avoiding the cumulative error problem caused by multiple systems. Furthermore, it only requires training the end-to-end speech translation model, significantly reducing data volume requirements, data annotation requirements, and the time and space consumption for model training. The speech translation method provided in this disclosure requires no text information, effectively solving the problem of languages ​​without written form (no text) such as uncommon dialects, thus greatly reducing implementation costs. In addition, the amount of training data used in this disclosure is relatively small, further reducing implementation costs.

[0085] like Figure 9 As shown, the speech translation method provided in at least one embodiment of this disclosure is based on the same inventive concept, and at least one embodiment of this disclosure can also provide a speech translation device.

[0086] The speech translation device in one or more embodiments of this disclosure includes, but is not limited to, a speech receiving module 901, a feature extraction module 902, an information conversion module 903, and a speech encoding module 904.

[0087] The voice receiving module 901 is used to receive the first voice, which is the source language voice to be translated.

[0088] The feature extraction module 902 is used to extract first acoustic feature information from the first speech, and the first acoustic feature information is used to represent the semantics of the source language speech.

[0089] The information conversion module 903 is used to convert the first acoustic feature information into the spectral information of the target language.

[0090] The speech encoding module 904 is used to encode the spectral information to generate a second speech; the second speech is the target language speech obtained by translating the source language speech.

[0091] Optionally, the information conversion module 903 includes a translator submodule and a synthesizer submodule.

[0092] The translator submodule is used to translate the first acoustic feature information to obtain the second acoustic feature information; the second acoustic feature information is used to represent the semantics of the target language speech.

[0093] The synthesizer submodule is used to convert the second acoustic feature information into spectral information of the target language.

[0094] Optionally, the translator submodule includes a context acquisition module, a semantic extraction module, and a semantic conversion module.

[0095] The context acquisition module is used to obtain context information from the first acoustic feature information, which includes the context information.

[0096] The semantic extraction module is used to extract first semantic information from the first acoustic feature information based on context information. The first semantic information is used to represent the meaning expressed by the source language.

[0097] The semantic conversion module is used to convert the first semantic information into the second acoustic feature information.

[0098] Optionally, the synthesizer submodule includes a semantic recognition module and a spectrum generation module.

[0099] The semantic recognition module is used to identify second semantic information from the second acoustic feature information. The second semantic information is used to represent the meaning expressed in the target language.

[0100] The spectrum generation module is used to generate spectrum information for the target language using second semantic information.

[0101] Optionally, the spectrum generation module includes a first prediction module and a second prediction module.

[0102] The first prediction module is used to predict first prediction information for characterizing the coarse-grained spectrum using the second semantic information.

[0103] The second prediction module is used to predict second prediction information for characterizing fine-grained spectrum using the first prediction information, and to use the second prediction information as the spectrum information of the target language.

[0104] Optionally, the feature extraction module 902 includes an extraction submodule and a filtering submodule.

[0105] The extraction submodule is used to extract the original acoustic feature information from the first speech. The original acoustic feature information includes the first acoustic feature information and the third acoustic feature information. The third acoustic feature information includes timbre information and pronunciation emotion information.

[0106] The filtering submodule is used to filter the original acoustic feature information to remove the third acoustic feature information and obtain the first acoustic feature information.

[0107] Optionally, the source language speech is Mandarin and the target language speech is a dialect, or the source language speech is a dialect and the target language speech is Mandarin.

[0108] Optionally, the spectral information of the target language is the Mel spectral information of the target language.

[0109] like Figure 10 As shown, based on the same inventive concept, one or more embodiments of this disclosure also provide an electronic device, including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to perform the speech translation method in one or more embodiments of this disclosure. The detailed implementation process of the speech translation method has been described in detail in this specification and will not be repeated here.

[0110] In this disclosure, the electronic device can serve as the execution subject of the speech translation method. The electronic device may include, but is not limited to, a computer, a mobile terminal, a portable translator, or other device capable of implementing the speech translation method of this disclosure.

[0111] like Figure 10 As shown, the speech translation method provided in one or more embodiments of this disclosure is based on the same inventive concept. One or more embodiments of this disclosure can also provide a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors cause the one or more processors to execute the speech translation method in one or more embodiments of this disclosure. The detailed implementation process of the speech translation method has been described in detail in this specification and will not be repeated here.

[0112] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM, or flash memory), fiber optic devices, and compact disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0113] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0114] In the description of this specification, the references to terms such as "this embodiment," "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0115] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0116] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, and simple improvements made to the substantive content of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A speech translation method, characterized in that, The method includes: Receive the first speech, which is the source language speech to be translated; First acoustic feature information is extracted from the first speech, and the first acoustic feature information is used to represent the semantics of the source language speech; Converting the first acoustic feature information into spectral information of the target language includes: obtaining context information from the first acoustic feature information, wherein the context information includes information on acoustic feature changes caused by different semantics of adjacent sentences and serves as the basis for dividing the different semantics of adjacent sentences; extracting first semantic information from the first acoustic feature information based on the context information; converting the first semantic information into second acoustic feature information; identifying second semantic information from the second acoustic feature information; using the second semantic information to predict first prediction information for characterizing coarse-grained spectrum; using the first prediction information to predict second prediction information for characterizing fine-grained spectrum; and using the second prediction information as the spectral information of the target language. The spectral information is encoded to generate a second speech; the second speech is the target language speech obtained by translating the source language speech.

2. The speech translation method according to claim 1, characterized in that, The second acoustic feature information is used to represent the semantics of the target language speech.

3. The speech translation method according to claim 2, characterized in that, The first semantic information is used to represent the meaning expressed through the source language.

4. The speech translation method according to any one of claims 1 to 3, characterized in that, The second semantic information is used to represent the meaning expressed through the target language.

5. The speech translation method according to claim 1, characterized in that, The step of extracting the first acoustic feature information from the first speech includes: The original acoustic feature information is extracted from the first speech, the original acoustic feature information including the first acoustic feature information and the third acoustic feature information, the third acoustic feature information including timbre information and pronunciation emotion information; The original acoustic feature information is filtered to remove the third acoustic feature information, thus obtaining the first acoustic feature information.

6. The speech translation method according to claim 1, characterized in that, The source language speech is Mandarin and the target language speech is a dialect, or the source language speech is a dialect and the target language speech is Mandarin.

7. The speech translation method according to claim 1, characterized in that, The spectral information of the target language is the Mel spectral information of the target language.

8. A voice translation device, characterized in that, The device includes: A voice receiving module is used to receive first voice, which is the source language voice to be translated; The feature extraction module is used to extract first acoustic feature information from the first speech, the first acoustic feature information being used to represent the semantics of the source language speech; An information conversion module is used to convert the first acoustic feature information into spectral information of the target language; A speech encoding module is used to encode the spectral information to generate a second speech; the second speech is the target language speech obtained by translating the source language speech. The information conversion module includes a translator submodule and a synthesizer submodule. The translator submodule includes a context acquisition module, a semantic extraction module, and a semantic conversion module. The context acquisition module is used to acquire context information from the first acoustic feature information. The context information includes information on acoustic feature changes caused by different semantics of adjacent sentences and serves as the basis for dividing between different semantics of adjacent sentences. The semantic extraction module is used to extract first semantic information from the first acoustic feature information based on the context information; The semantic conversion module is used to convert the first semantic information into the second acoustic feature information; The synthesizer submodule is used to convert the second acoustic feature information into spectral information of the target language; The synthesizer submodule includes a semantic recognition module and a spectrum generation module; The semantic recognition module is used to identify second semantic information from the second acoustic feature information, the second semantic information being used to represent the meaning expressed through the target language; The spectrum generation module is used to generate spectrum information of the target language using the second semantic information; The spectrum generation module includes a first prediction module and a second prediction module; The first prediction module is used to predict first prediction information for characterizing the coarse-grained spectrum using the second semantic information; The second prediction module is used to predict second prediction information for characterizing fine-grained spectrum using the first prediction information, and to use the second prediction information as the spectrum information of the target language.

9. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores computer-readable instructions that, when executed by the processor, cause the processor to perform the speech translation method as described in any one of claims 1 to 7.

10. A storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by one or more processors, the one or more processors cause the speech translation method as described in any one of claims 1 to 7.

Images