Method for generating speech translation model, translation method and device
By using only the decoder architecture and semantic feature extractor in the speech translation model, the problem of cascaded task error accumulation and large parameters is solved, and efficient speech translation effect is achieved.
Patent Information
- Application Number
- CN202310659465.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2024-12-06
AI Technical Summary
The existing speech translation model has poor results due to the accumulation of errors in cascading tasks, and the encoder-decoder architecture has large parameters and low training efficiency, which has high engineering implementation requirements.
Using a model architecture that uses only the decoder, speech translation is processed separately through a semantic feature extractor and multiple decoders to avoid audio conversion to text, process unwritten language, and add an acoustic unit to preserve source audio characteristics.
Reduces model training parameters, improves training efficiency, reduces engineering implementation requirements, achieves better speech translation effects, and can handle ordinary and unwritten languages.
Smart Images

Figure CN119091870A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to the field of computers, and more particularly, to a method, apparatus, electronic device, and medium for generating a speech translation model. Background Art
[0002] Speech-to-speech Translation (S2ST) refers to a type of technology that converts speech in one language into speech in another language, also known as speech translation, which is very helpful in breaking down communication barriers between people who do not speak the same language. Unlike traditional text-to-text machine translation, the input and output of speech translation tasks are both speech. This technology is becoming increasingly important under the trend of globalization, especially in providing more direct convenience in cross-border communication, tourism, business and other fields.
[0003] A typical speech translation model usually consists of three tasks: automatic speech recognition (ASR), machine translation (MT), and text-to-speech conversion (TTS). The encoder-decoder structure is often used for sequence-to-sequence learning, which can be used to learn a direct end-to-end speech translation model. In the end-to-end speech translation model, the research paths can be divided into two: continuous speech spectra and discrete speech units, according to the prediction form of the target end. Summary of the invention
[0004] Embodiments of the present disclosure provide a method, an apparatus, an electronic device, and a computer-readable storage medium for generating a speech translation model.
[0005] According to a first aspect of the present disclosure, a method for generating a speech translation model is provided, the speech translation model comprising a semantic feature extractor and a plurality of decoders. The method comprises extracting, by the semantic feature extractor, a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, the source language audio corresponding to the target language audio. The method further comprises adjusting a first decoder among the plurality of decoders based on the source semantic unit sequence and the target semantic unit sequence. The method further comprises adjusting a second decoder among the plurality of decoders based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
[0006] According to a second aspect of the present disclosure, a method for speech translation is provided, which is performed by a speech translation model generated according to the first aspect, wherein the speech translation model includes a semantic feature extractor and a plurality of decoders. The method includes generating a predicted target semantic unit sequence based on a given source semantic unit sequence of a given source language audio. The method also includes generating a predicted acoustic unit sequence based on a given source semantic unit sequence of a given source language audio, a predicted target semantic unit sequence, and a given source acoustic unit sequence.
[0007] In a third aspect of the present disclosure, a device for generating a speech translation model includes a semantic feature extractor and multiple decoders. The device includes a semantic feature extraction module configured to extract a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, the source language audio corresponding to the target language audio. The device also includes a first decoding adjustment module configured to adjust a first decoder among multiple decoders based on the source semantic unit sequence and the target semantic unit sequence. The device also includes a second decoding adjustment module configured to adjust a second decoder among multiple decoders based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
[0008] According to a fourth aspect of the present disclosure, there is provided a device for speech translation, which is configured to use the speech translation model generated according to the third aspect. The device includes a first predicted semantic determination module, which is configured to generate a predicted target semantic unit sequence given a given source semantic unit sequence of a given source language audio. The device also includes a first predicted acoustic determination module, which is configured to generate a predicted acoustic unit sequence based on a given source semantic unit sequence of the given source language audio, the predicted target semantic unit sequence, and a given source acoustic unit sequence.
[0009] According to a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.
[0010] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein one or more computer instructions are stored on the computer-readable storage medium, wherein the one or more computer instructions are executed by a processor to implement the method according to the first aspect.
[0011] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0013] Figure 1 A schematic diagram showing an example environment in which a speech translation method according to some embodiments of the present disclosure may be implemented;
[0014] Figure 2 A flowchart of a method for generating a speech translation model according to some embodiments of the present disclosure is shown;
[0015] Figure 3 A schematic diagram showing a process of training a translation decoder according to some embodiments of the present disclosure;
[0016] Figure 4 A schematic diagram showing a process of training a synthesis decoder according to some embodiments of the present disclosure;
[0017] Figure 5 A flowchart showing a process of training a temporal decoder according to certain embodiments of the present disclosure is shown;
[0018] Figure 6 A schematic diagram illustrating a process of training a translation decoder using a multi-task learning scheme according to certain embodiments of the present disclosure;
[0019] Figure 7 A schematic diagram showing a process for speech translation from source audio to target audio according to certain embodiments of the present disclosure;
[0020] Figure 8 A block diagram of an apparatus for generating a speech translation model according to some embodiments of the present disclosure is shown;
[0021] Fig. 9 A flowchart showing a method of speech translation according to some embodiments of the present disclosure is shown;
[0022] Fig.10 A block diagram showing an apparatus for speech translation according to some embodiments of the present disclosure; and
[0023] Fig.11 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.
[0024] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION
[0025] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, usage scenarios, etc. of the personal information (such as voice) involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.
[0026] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information (such as voice) to software or hardware such as electronic devices, applications, servers or storage media that perform operations of the technical solution of the present disclosure based on the prompt message. It can be understood that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation method of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation method of the present disclosure.
[0027] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.
[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0029] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.
[0030] A typical speech translation model usually consists of three tasks: speech recognition, machine translation, and text-to-speech conversion. However, since the task is divided into three cascaded subtasks, the errors of the previous subtasks will accumulate to the subsequent subtasks, affecting the effect of the final speech translation task.
[0031] The encoder-decoder model architecture is mostly used for sequence-to-sequence learning. This approach can also be used to learn a direct end-to-end speech translation model, which can avoid the error accumulation of the cascade solution. However, the use of the encoder-decoder architecture at the same time results in a huge number of model parameters, low training efficiency, and high engineering implementation requirements.
[0032] In order to solve the above problems, the embodiments of the present disclosure provide a speech translation scheme for training a model architecture that uses only a decoder (Decoder-Only). This scheme can only train multiple decoders in the speech translation model when training the speech translation model, so that the number of parameters in the training process is reduced, the training efficiency is improved, and the requirements for engineering implementation are reduced, and better speech translation effects are achieved. This scheme uses a decoder to convert the semantic units of the source audio. Since it avoids converting the audio into text, it can not only process ordinary written languages, but also unwritten languages (for example, dialects, minority languages, etc.), and at the same time, another decoder is used to realize the synthesis from semantic units to audio, and the acoustic units of the source audio are added during processing, so the source audio characteristics and speech patterns can be used.
[0033] Figure 1 FIG. 1 is a schematic diagram of an example environment 100 in which a speech translation method according to some embodiments of the present disclosure may be implemented. Figure 1 As shown, the example environment 100 may include a computing device 110, which may be a user terminal, a mobile device, a computer, etc., or a computing system, a single server, a distributed server, or a cloud-based server. The computing device 110 may receive source audio 160-1, source audio 160-2, source audio 160-3, and source audio 160-4 (individually or collectively referred to as source audio 160). The source audio 160 may be understood as a speech of a certain length, such as the sound of a user speaking, which contains meaningful content and can usually be recorded in writing. It is understandable that more or less audio may also be included in the environment 100.
[0034] In the computing device 110, a speech translation model 120 may also be included. For example, the speech translation model 120 is deployed in the computing device 110. The speech translation model 120 may be used to generate a translation result of the audio 160 based on the audio 160, i.e., a target audio 170-1, a target audio 170-2, a target audio 170-3, and a target audio 170-4 (individually or collectively referred to as the target audio 170). Each source audio in the source audio 160 corresponds to each target audio in the target audio 170, for example, the source audio 160-1 may be translated into the target audio 170-1, the source audio 160-2 may be translated into the target audio 170-2, and the like by the speech translation model 120. In some embodiments, the speech translation model 120 may be obtained by unsupervised training by constructing input data by the source audio 160 and the target audio 170 based on the architecture using only a decoder.
[0035] refer to Figure 1 , the speech translation model 120 includes a semantic feature extractor 130. Using the semantic feature extractor 130, the original audio can be converted into a semantic unit sequence. It should be noted that the semantic unit sequence here is related to the text content corresponding to the audio, but it is not the actual text, but some discrete digital unit sequences, so that the semantic feature extractor can process unwritten languages. The obtained semantic unit sequence can be used in the subsequent training process of the decoder. In the embodiment of the present disclosure, the semantic feature extractor 130 is a pre-trained model that can directly extract the semantic unit sequence of the audio without further training. In the process of training the decoder 140, the decoder 145 and the decoder 150, the relevant model parameters in the semantic feature extractor 130 remain unchanged.
[0036] The speech translation model 120 also includes a decoder 140. The decoder 140 can convert a source semantic unit sequence of the source language audio into a target semantic unit sequence of the target language audio. When training the decoder 140, the source semantic unit sequence and the target semantic unit sequence in the training data are simultaneously input into the decoder 140 to perform model training through unsupervised training.
[0037] The speech translation model 120 also includes a decoder 145. The decoder 145 can restore the timbre information in the source audio 160 to the target semantic unit sequence generated by the decoder 140, so as to finally retain it in the generated target audio 170. It should be understood that in some embodiments, the speech translation model 120 may not have the decoder 145, and can still implement the function of speech translation.
[0038] The speech translation model 120 also includes a decoder 150. The decoder 150 can convert a target semantic unit sequence of the target language audio into a target acoustic unit sequence of the target language audio. When training the decoder 150, the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence, and the target acoustic unit sequence in the training data are simultaneously input into the decoder 150 to adjust the model through unsupervised training, wherein the source acoustic unit sequence and the target acoustic unit sequence can be extracted by the audio stream encoder.
[0039] Both decoder 140 and decoder 150 can be language models that only use decoder structures, so they can use prompt sequences related to source audio information and target audio information for unsupervised training. During the unsupervised training process, the parameters of decoder 140 and decoder 150 are adjusted by maximizing the probability of the next unit predicted from left to right by the prediction sequence. In the inference stage, the prompt sequence related to the source audio information can be input into decoder 140 and decoder 150 to generate a target acoustic sequence associated with the target audio.
[0040] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.
[0041] The following will combine Figures 2 to 9 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.
[0042] Figure 2 A flow chart of a method 200 for generating a speech translation model according to some embodiments of the present disclosure is shown, the speech translation model comprising a semantic feature extractor and a plurality of decoders. At block 202, the semantic feature extractor extracts a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, the source language audio corresponding to the target language audio. For example, Figure 1 As shown, the speech translation model 120 obtains the source audio 160 and the target audio 170 of the corresponding semantics in different languages. It should be understood that since it is the model training stage, both the source audio 160 and the target audio 170 can be obtained, and there is no need to generate the target audio 170 from the source audio 160. It can be understood that in the inference stage, the speech translation model 120 can generate the target audio 170 based on the source audio 160.
[0043] At block 204, a first decoder of a plurality of decoders is adjusted based on the source semantic unit sequence and the target semantic unit sequence. Figure 1 , the model parameters of the decoder 140 are adjusted by the source semantic unit sequence and the target semantic unit sequence. It can be understood that the semantic unit sequence here is related to the text corresponding to the audio, for example, the semantic unit sequences of similar audio contents are also similar, and have nothing to do with the acoustic features of the audio such as duration, frequency and timbre.
[0044] At block 206, a second decoder of the plurality of decoders is adjusted based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, wherein the semantic feature extractor remains unchanged during the adjustment of the first decoder and the second decoder. Figure 1 , based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, the decoder 150 is adjusted. It can be understood that the model parameters of the decoder 150 can be adjusted through the semantic unit sequence and the acoustic unit sequence. It can be understood that when adjusting the decoder 150, the acoustic unit sequence is added, so the decoder 150 learns audio-related features, such as speech patterns, frequencies, and timbre.
[0045] Therefore, according to the method 200 of the embodiment of the present disclosure, when generating a speech translation model, only multiple decoders in the semantic translation model are trained, which can reduce model training parameters, improve training efficiency, and reduce engineering implementation requirements, such as hardware configuration, while improving the effect of speech translation.
[0046] The following will combine Figures 3 to 5 The training process of multiple decoders is specifically described, where Figure 3 The training process of the translation decoder is shown. The translation decoder can be Figure 1 The decoder 140 in is used to translate the source semantic unit sequence into the target semantic unit sequence. Figure 4 shows the training process of the synthesis decoder, which can be Figure 1 The decoder 150 in is used to synthesize the target semantic unit sequence into the target acoustic unit sequence. Figure 5 The training process of the temporal decoder is shown. The temporal decoder can be Figure 1 The decoder 145 in is used to restore the compressed target semantic unit sequence.
[0047] Figure 3A schematic diagram of a process 300 for training a translation decoder according to some embodiments of the present disclosure is shown. First, a source audio 302 and a target audio 304 are obtained. For example, the source audio 302 can be an English audio, and the target audio 304 can be a Chinese audio corresponding to the English audio. It should be understood that although the source audio is in English and the target audio is in Chinese, the speech conversion between other languages can also be used in combination with the embodiments of the present disclosure. The semantic features of the source audio 302 and the semantic features of the target audio 304 are extracted by a semantic feature extractor 306. The semantic feature extractor 306 may include a self-supervised speech representation learning model and a clustering model to convert the source audio 302 and the target audio 304 into a source semantic unit sequence 308 and a target semantic unit sequence 310. During the training of the translation encoder, the parameters of the semantic feature extractor 306 will not change. It can be understood that the source semantic unit sequence 308 and the target semantic unit sequence 310 are both discrete unit sequences, in which each unit corresponds to a numerical value, so in the whole process, it is not involved to convert the audio into text.
[0048] The source semantic unit sequence 308 and the target semantic unit sequence 310 are processed by the compression module 312 to obtain a compressed source semantic unit sequence 316 and a compressed target semantic unit sequence 318. For example, since there are some repetitions, pauses, etc. in the audio, the effect of the translation decoder 322 can be improved by removing this part of redundant information through the compression module 312, but it can be understood that the compression module 312 is optional. That is, in some embodiments, the source semantic unit sequence 308 and the target semantic unit sequence 310 may not be compressed, but directly input into the translation decoder 322 for training.
[0049] The compressed source semantic unit sequence 314, the compressed target semantic unit sequence 316 and the task information 318 can be combined to obtain a prompt sequence 320 (also called a first prompt sequence). For example, when the source audio 302 is English audio, an example of the prompt sequence 320 can be "Translate English unit{src_unit}to Chinese unit{tat_unit}", where the task information 318 is "Translate English unit…to Chinese unit…", which prompts the translation decoder 322 to process the task of converting English units into Chinese units. It can be understood that the language type here can be changed as needed, "{src_unit}" represents the compressed source semantic unit sequence 314, and {tat_unit} represents the compressed target semantic unit sequence 316. It should be understood that in Figure 3The task information 318 is placed on the leftmost side of the prompt sequence 320 for the convenience of drawing. In fact, as can be seen from the above example, the task information 318 is interspersed in the compressed source semantic unit sequence 314 and the compressed target semantic unit sequence 316.
[0050] The parameters of the translation decoder 322 are trained and optimized by inputting the prompt sequence 320 into the translation decoder 322 for unsupervised training. For example, the translation decoder 322 can perform unsupervised training based on the prompt sequence 320, predict the probability of the next prompt unit appearing from left to right on the prompt sequence 320, and maximize this probability to train the translation decoder 322.
[0051] Figure 4 A schematic diagram of a process 400 for training a synthesis decoder according to some embodiments of the present disclosure is shown, where the translation decoder may be Figure 1 The decoder 150 shown can realize the synthesis of acoustic feature sequences, wherein the acoustic feature sequences can be converted into target audio. Figure 4 In the embodiment, the source audio 402 and the target audio 404 are processed by the semantic feature extractor 406 to obtain a source semantic unit sequence 408 and a target semantic unit sequence 410, and the source semantic unit sequence 408 and the target semantic unit sequence 410 are compressed by the compression module 412 to obtain a compressed source semantic unit sequence 414 and a compressed target semantic unit sequence 416.
[0052] The source audio 402 is processed by the acoustic unit extractor 418 to obtain a source acoustic unit sequence 420, wherein the source acoustic unit sequence 420 represents acoustic characteristics related to the source audio 402, such as frequency, timbre, speaking style, etc. Similarly, the target audio 404 is processed by the acoustic unit extractor 420 to obtain a target acoustic unit sequence 422.
[0053] The compressed source semantic unit sequence 414, the compressed target semantic unit sequence 416, the source acoustic unit sequence 420, and the target acoustic unit sequence 422 are combined to obtain a prompt sequence 424 (also referred to as a second prompt sequence), and the prompt sequence 424 is input into a synthesis decoder 426 for unsupervised training to train and optimize the parameters of the synthesis decoder 426. For example, the synthesis decoder 426 can perform unsupervised training based on the prompt sequence 424, predict the probability of the next prompt unit appearing from left to right on the prompt sequence 424, and maximize this probability to train the synthesis decoder 426.
[0054] Figure 5 A flowchart of a process 500 for training a temporal decoder according to certain embodiments of the present disclosure is shown. The temporal decoder may be Figure 1The decoder 145 shown in the figure can restore the compressed speech unit to retain the timbre of the source audio to the target audio. As described in the process 300 of training the translation decoder, in order to better model the semantic unit sequence, the source semantic unit sequence 308 and the target semantic unit sequence 310 are compressed to remove some repetitions, pauses, etc. in the audio. However, when generating the target audio, it is hoped that these repetitions and pauses of the source audio can be retained in the target audio, and a timing decoder is required to restore the compressed speech unit.
[0055] like Figure 5 As shown, the source audio 502 and the target audio 504 are processed by the semantic feature extractor 506 to obtain a source semantic unit sequence 508 and a target semantic unit sequence 510, and the source semantic unit sequence 508 and the target semantic unit sequence 510 are compressed by the compression module 512 to obtain a compressed source semantic unit sequence 514 and a compressed target semantic unit sequence 516, and a source time series value sequence 518 and a target time series value sequence 520. For example, in some embodiments, after the source semantic unit sequence 508 is obtained by the semantic feature extractor 506, and the length of the source semantic unit sequence 508 is 4, and the length of the target semantic unit sequence 510 is 6, after compression by the compression module 512, the length of the compressed source semantic unit sequence 514 is 3, and the length of the compressed target semantic unit sequence 516 is 4. Therefore, the corresponding source timing value sequence 518 is [1, 2, 1], which is associated with the compressed mode of the compressed source semantic unit sequence 314, and the target timing value sequence 520 is [1, 2, 2, 1], which is associated with the compressed mode of the compressed target semantic unit sequence 516.
[0056] The prompt sequence 522 is obtained by combining the compressed source semantic unit sequence 514, the compressed target semantic unit sequence 516, the source time sequence value sequence 518 and the target time sequence value sequence 520. The prompt sequence 522 is input into the time sequence decoder 524 for unsupervised training to train and optimize the parameters of the time sequence decoder 524. It can be understood that the time sequence decoder is similar to the translation decoder and the synthesis decoder, and is trained based on the language model. Specifically, the training is performed by predicting the next unit in the sequence through the constructed input sequence.
[0057] Figure 6A schematic diagram of a process 600 of training a translation decoder using a multi-task learning scheme according to certain embodiments of the present disclosure is illustrated. At 610, speech recognition data is obtained. It should be understood that the speech recognition task only involves a specific language type, converting the audio and text of the language type, and the data format of the speech recognition data is:<unit,text> , that is, converting the semantic unit unit into the corresponding text text, where the semantic unit unit can be obtained by Figure 1 The semantic feature extractor 130 extracts the semantic feature. In some embodiments, data of existing language recognition tasks can be obtained. At 615, a speech recognition prompt sequence is generated. For example, the speech recognition prompt sequence can be "Translate [lang] unit {unit} to [lang] text {text}", and the prompt sequence is used to prompt the translation decoder 640 (i.e., the first decoder) to convert the semantic unit unit into the corresponding text text, where "lang" is used to specify the current language type. In addition, the speech recognition prompt can also be "Translate [lang] text {text} to [lang] unit {unit}", and the prompt sequence is used to prompt the translation decoder 640 to convert the text text into the corresponding semantic unit unit. After the speech recognition prompt sequence is generated, the speech recognition prompt sequence is used to adjust the translation decoder 640 to optimize the parameters of the translation decoder 640.
[0058] At 620, text translation data is obtained. It should be understood that the text translation task involves text conversion between two types of languages, and does not involve audio conversion. The data format of the text translation is<src_text,tgt_text> , that is, converting the source language text into the target language text. At 625, a text translation prompt sequence is generated. For example, the text translation prompt sequence can be "Translate[src lang]text{text}to[tgt lang]text{text}", which prompt sequence is used to prompt the translation decoder 640 to translate the source language text into the text of the target language, where "src lang" is the language type of the source language and "tgt lang" is the language type of the target language. For example, the source language can be English and the target language is Chinese. After the text translation prompt sequence is generated, the text translation prompt sequence is used to adjust the translation decoder 640 to optimize the parameters of the translation decoder 640.
[0059] At 630, speech-to-speech data is obtained. It should be understood that the speech-to-speech task involves direct speech conversion of two types of speech, and the format of the speech-to-speech data is<src_unit,tgt_unit,src_text,tgt_text> , including source semantic units, target semantic units, source text, and target text. At 635, a speech-to-speech prompt sequence is generated. Generally speaking, the speech-to-speech prompt sequence is "Translate[src lang]unit{src_unit}to[tgt lang]unit{tgt_unit}", which is used to prompt the translation decoder 640 to translate the source semantic unit into the target semantic unit. After generating the speech-to-speech prompt sequence, the speech-to-speech prompt sequence is used to adjust the translation decoder 640 to optimize the parameters of the translation decoder 640. Since the speech-to-speech data includes a total of 4 types of data, namely, source semantic units, target semantic units, source text, and target text, the following prompt sequence can also be constructed:
[0060] “Translate [src lang] unit {src_unit} to [src lang] text {src_text}”, this prompt sequence is used to prompt the translation decoder 640 to translate the source semantic unit into the source text;
[0061] “Translate [src lang] unit {src_unit} to [tgt lang] text {tgt_text}”, this prompt sequence is used to prompt the translation decoder 640 to translate the source semantic unit into the target text;
[0062] “Translate [src lang] text {src_text} to [tgt lang] unit {tgt_unit}”, this prompt sequence is used to prompt the translation decoder 640 to translate the source text into the target semantic unit;
[0063] “Translate [tgt lang] text {tgt_text} to [tgt lang] unit {tgt_unit}”, this prompt sequence is used to prompt the translation decoder 640 to translate the target text into the target semantic unit.
[0064] It can be seen that by utilizing a multi-task learning scheme, data of various task types, such as any one or more of speech recognition data 610, text translation data 620, and speech-to-speech data 630, can be used to adjust the parameters of the translation decoder 640, thereby improving the performance and effect of the translation decoder 640.
[0065] Figure 7A schematic diagram of a process 700 for speech translation from source audio to target audio is shown, in accordance with certain embodiments of the present disclosure. Figures 3 to 5 Main description Figure 1 An example training process for multiple decoders in the speech translation model 120 in FIG. Figure 7 How to use the speech translation model 120 to convert the source audio into the target audio will be discussed. Figure 7 As shown, first obtain source audio 710. For example, the source audio can be an audio of a language type such as Chinese, English, German, etc. In some embodiments, the source audio 710 can be an audio of an unwritten language, and an unwritten language refers to a language without text, such as some dialects, or a minority language, etc. It is processed by a speech to unit module 720, for example, first extracting a semantic unit sequence of the source audio by a semantic feature extractor 722, wherein the semantic feature extractor 722 may include a self-supervised speech representation learning model and a clustering model to extract a discrete source semantic unit sequence from the source audio 710. Next, the source semantic unit sequence is compressed by a compression module 724 to obtain a compressed source semantic unit sequence. Since there are some repetitions, pauses, etc. in the source audio, removing this part of redundant information by the compression module 724 can improve the effect of the translation decoder 726. By combining the compressed source semantic unit sequence with the task information, a prompt sequence is obtained. For example, the prompt sequence may be “Translate[src lang]unit{src_unit}to[tgt lang]unit”, that is, the prompt sequence includes the language type “src lang” of the source audio 710, the compressed source semantic unit sequence “{src_unit}”, the target audio language type “tgt lang”, and the task type is unit to unit “unit to unit”. Therefore, the translation decoder 726 generates a target semantic unit sequence according to the prompt sequence. Since the input is a compressed source semantic unit sequence, a compressed target semantic unit sequence is actually obtained.
[0066] Next, the unit to language module 730 is used to continue processing. For example, the compressed target semantic unit sequence is processed by the implementation decoder 732 to obtain the target semantic unit sequence, so as to restore the timbre information contained in the source audio 710 to the target semantic unit sequence. Then, the source acoustic unit sequence of the source audio is extracted by the acoustic feature extractor 740, and the source semantic unit sequence, the target semantic unit sequence, and the source acoustic unit sequence are combined into another prompt sequence, and the other prompt sequence is input into the synthesis decoder 734 to generate the target acoustic unit sequence. The target acoustic unit sequence is processed by the acoustic unit converter 750 to obtain the predicted target audio 740.
[0067] Figure 8 FIG. 8 is a block diagram of an apparatus 800 for generating a speech translation model according to some embodiments of the present disclosure. Figure 8 As shown, the device 800 includes a semantic feature extraction module 802, which is configured to extract a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, the source language audio corresponding to the target language audio. The device 800 also includes a first decoding adjustment module 804, which is configured to adjust a first decoder among multiple decoders based on the source semantic unit sequence and the target semantic unit sequence. In addition, the device 800 also includes a second decoding adjustment module 806, which is configured to adjust a second decoder among multiple decoders based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
[0068] Therefore, through the device 800 of the present disclosure, when generating a speech translation model, only the decoder is trained, which can reduce model training parameters, improve training efficiency, reduce engineering implementation requirements, such as hardware configuration, etc., and at the same time improve the effect of speech translation.
[0069] Fig. 9 A flowchart of a method 900 for speech translation according to some embodiments of the present disclosure is shown. The method 900 may be performed by Figure 2 The speech translation model generated by the method 200 is executed, and the speech translation model includes a semantic feature extractor and a plurality of decoders. In box 902, a predicted target semantic unit sequence is generated based on a given source semantic unit sequence of a given source language audio. In box 904, a predicted acoustic unit sequence is generated based on a given source semantic unit sequence of a given source language audio, a predicted target semantic unit sequence, and a given source acoustic unit sequence.
[0070] Fig.10 FIG. 1 is a block diagram of an apparatus 1000 for speech translation according to some embodiments of the present disclosure. Fig.10 As shown, the apparatus 1000 includes a first predicted semantic determination module 1002, which is configured to generate a predicted target semantic unit sequence given a given source semantic unit sequence of a given source language audio. The apparatus 1000 also includes a first predicted acoustic determination module 1004, which is configured to generate a predicted acoustic unit sequence based on a given source semantic unit sequence of a given source language audio, a predicted target semantic unit sequence, and a given source acoustic unit sequence.
[0071] Fig.11 1 shows a block diagram of an electronic device 1100 according to some embodiments of the present disclosure. The device 1100 may be a device or apparatus described in an embodiment of the present disclosure. Fig.11As shown, the device 1100 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 1102 or loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The CPU / GPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104. Although not shown in FIG. Fig.11 As shown in FIG. 1 , device 1100 may further include a coprocessor.
[0072] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0073] The various methods or processes described above may be executed by the CPU / GPU 1101. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the CPU / GPU 1101, one or more steps or actions in the methods or processes described above may be performed.
[0074] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.
[0075] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.
[0076] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0077] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.
[0078] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0080] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0081] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
[0082] Some example implementations of the present disclosure are listed below.
[0083] Example 1. A method for generating a speech translation model, the speech translation model comprising a semantic feature extractor and a plurality of decoders, the method comprising:
[0084] The semantic feature extractor extracts a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, wherein the source language audio corresponds to the target language audio;
[0085] Adjusting a first decoder of the plurality of decoders based on the source semantic unit sequence and the target semantic unit sequence; and
[0086] Based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, adjust the second decoder among the multiple decoders, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
[0087] Example 2. The method of Example 1, wherein adjusting the first decoder of the plurality of decoders comprises:
[0088] Obtaining a first prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, and task information, wherein the task information at least specifies a language type of the source language audio and a language type of the target language audio; and
[0089] Based on the first hint sequence, the first decoder is adjusted.
[0090] Example 3. The method of Examples 1-2, wherein adjusting the second decoder of the plurality of decoders comprises:
[0091] Obtaining a second prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence, and the target acoustic unit sequence; and
[0092] Based on the second hint sequence, the second decoder is adjusted.
[0093] Example 4. The method according to examples 1-3 further includes:
[0094] By compressing the source semantic unit sequence and the target semantic unit sequence, a compressed source semantic unit sequence and a compressed target semantic unit sequence are obtained; and
[0095] The first decoder is adjusted using the compressed source semantic unit sequence, the compressed target semantic unit sequence, and the task information.
[0096] Example 5. The method according to examples 1-4 further includes:
[0097] By compressing the source semantic unit sequence and the target semantic unit sequence, a source time series value sequence and a target time series value sequence are obtained, wherein the source time series value sequence and the target time series value sequence are related to the compression mode; and
[0098] A third decoder of the plurality of decoders is adjusted using the compressed source semantic unit sequence, the compressed target semantic unit sequence, the source timing value sequence, and the target timing value sequence.
[0099] Example 6. The method according to Examples 1-5, wherein the semantic feature extractor includes any one of an unsupervised model and a clustering model.
[0100] Example 7. The method of Examples 1-6, further comprising adjusting the first decoder using multi-task learning, wherein the multi-task learning comprises at least one of the following:
[0101] Speech recognition tasks;
[0102] Text translation tasks; or
[0103] Speech-to-speech conversion task.
[0104] Example 8. The method of Examples 1-7, wherein at least one of the source language audio and the target language audio comprises an unwritten language, wherein the unwritten language does not have a handwritten script.
[0105] Example 9. A method for speech translation, the method being performed by a speech translation model generated according to any one of claims 1 to 8, wherein the speech translation model comprises a semantic feature extractor and a plurality of decoders, the method comprising
[0106] generating a predicted target semantic unit sequence based on a given source semantic unit sequence of a given source language audio; and
[0107] A predicted acoustic unit sequence is generated based on the given source semantic unit sequence of the given source language audio, the predicted target semantic unit sequence, and the given source acoustic unit sequence.
[0108] Example 10. The method according to Example 9, wherein generating the predicted target semantic unit sequence comprises:
[0109] Obtaining a first prediction prompt sequence by combining the given source semantic unit sequence and task information; and
[0110] The predicted target semantic unit sequence is generated by inputting the first prediction hint sequence into the first decoder among the plurality of decoders.
[0111] Example 11. The method of Examples 9-10, wherein generating the predicted acoustic unit sequence comprises:
[0112] Obtaining a second prediction prompt sequence by combining the given source semantic unit sequence, the predicted target semantic unit sequence, and the given source acoustic unit sequence; and
[0113] The predicted acoustic unit sequence is generated by inputting the second prediction cue sequence into the second decoder of the plurality of decoders.
[0114] Example 12. The method according to examples 9-11, further comprising:
[0115] Based on the predicted acoustic unit sequence, a predicted target language audio is generated.
[0116] Example 13. A device for generating a speech translation model, the speech translation model comprising a semantic feature extractor and a plurality of decoders, the device comprising:
[0117] A semantic feature extraction module configured to extract a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, the source language audio corresponding to the target language audio;
[0118] A first decoding adjustment module is configured to adjust a first decoder among the plurality of decoders based on the source semantic unit sequence and the target semantic unit sequence; and
[0119] A second decoding adjustment module is configured to adjust a second decoder among the multiple decoders based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
[0120] Example 14. The apparatus of Example 13, wherein the first decoding adjustment module comprises:
[0121] A first prompt sequence determination module is configured to obtain a first prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, and task information, wherein the task information at least specifies a language type of the source language audio and a language type of the target language audio; and
[0122] The third decoding adjustment module is configured to adjust the first decoder based on the first prompt sequence.
[0123] Example 15. The apparatus of Examples 13-14, wherein the second decoding adjustment module comprises:
[0124] a second prompt sequence determination module configured to obtain a second prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence, and the target acoustic unit sequence; and
[0125] The fourth decoding adjustment module is configured to adjust the second decoder based on the second prompt sequence.
[0126] Example 16. The apparatus of Examples 13-15, further comprising:
[0127] a compression unit determination module configured to obtain a compressed source semantic unit sequence and a compressed target semantic unit sequence by compressing the source semantic unit sequence and the target semantic unit sequence; and
[0128] The fifth decoding adjustment module is configured to adjust the first decoder using the compressed source semantic unit sequence, the compressed target semantic unit sequence, and the task information.
[0129] Example 17. The apparatus of Examples 13-16, further comprising:
[0130] a timing value sequence determination module, configured to obtain a source timing value sequence and a target timing value sequence by compressing the source semantic unit sequence and the target semantic unit sequence, wherein the source timing value sequence and the target timing value sequence are related to the compression mode; and
[0131] The sixth decoding adjustment module adjusts a third decoder among the plurality of decoders using the compressed source semantic unit sequence, the compressed target semantic unit sequence, the source timing value sequence, and the target timing value sequence.
[0132] Example 18. The apparatus of Examples 13-17, wherein the semantic feature extraction module comprises one of an unsupervised model and a clustering model.
[0133] Example 19. The apparatus of Examples 13-18, further comprising:
[0134] A seventh decoding adjustment module is configured to adjust the first decoder using multi-task learning, wherein the multi-task learning includes at least one of the following:
[0135] Speech recognition tasks;
[0136] Text translation tasks; or
[0137] Speech-to-speech conversion task.
[0138] Example 20. The apparatus of Examples 13-19, wherein at least one of the source language audio and the target language audio comprises an unwritten language, wherein the unwritten language does not have a handwritten script.
[0139] Example 21. A device for translation, the device being configured to use a language translation model generated according to any one of Examples 13 to 20, wherein the speech translation model comprises a semantic feature extractor and a plurality of decoders, the device comprising:
[0140] A first predicted semantic determination module, configured to generate a predicted target semantic unit sequence based on a given source semantic unit sequence of a given source language audio; and
[0141] The first predicted acoustic determination module is configured to generate a predicted acoustic unit sequence based on a given source semantic unit sequence of the given source language audio, the predicted target semantic unit sequence, and a given source acoustic unit sequence.
[0142] Example 22. The apparatus of Example 21, wherein the first target semantic determination module comprises:
[0143] A first prediction prompt sequence determination module is configured to obtain a first prediction prompt sequence by combining the given source semantic unit sequence and task information; and
[0144] The second predicted semantic determination module is configured to generate the predicted target semantic unit sequence by inputting the first prediction hint sequence into the first decoder among the multiple decoders.
[0145] Example 23. The apparatus of Examples 21-22, wherein the first target acoustic determination module comprises:
[0146] a first prediction prompt sequence determination module configured to obtain a second prediction prompt sequence by combining the given source semantic unit sequence, the predicted target semantic unit sequence, and the given source acoustic unit sequence; and
[0147] The second predicted acoustic determination module is configured to generate the predicted acoustic unit sequence by inputting the second predicted cue sequence into the second decoder among the plurality of decoders.
[0148] Example 24. The apparatus of Examples 21-23, further comprising:
[0149] The predicted target audio determination module is configured to generate predicted target language audio based on the predicted acoustic unit sequence.
[0150] Example 25. An electronic device comprising:
[0151] Processor; and
[0152] A memory coupled to the processor, the memory having instructions stored therein, wherein the instructions, when executed by the processor, cause the electronic device to perform actions, the actions being used to generate a speech translation model, the speech translation model comprising a semantic feature extractor and a plurality of decoders, the actions comprising:
[0153] The semantic feature extractor extracts a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, wherein the source language audio corresponds to the target language audio;
[0154] Adjusting a first decoder of the plurality of decoders based on the source semantic unit sequence and the target semantic unit sequence; and
[0155] Based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, adjust the second decoder among the multiple decoders, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
[0156] Example 26. The electronic device of Example 25, wherein adjusting the first of the plurality of decoders comprises:
[0157] Obtaining a first prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, and task information, wherein the task information at least specifies a language type of the source language audio and a language type of the target language audio; and
[0158] Based on the first hint sequence, the first decoder is adjusted.
[0159] Example 27. The electronic device of Examples 25-26, wherein adjusting the second decoder of the plurality of decoders comprises:
[0160] Obtaining a second prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence, and the target acoustic unit sequence; and
[0161] Based on the second hint sequence, the second decoder is adjusted.
[0162] Example 28. The electronic device of Examples 25-27, wherein the actions further include:
[0163] By compressing the source semantic unit sequence and the target semantic unit sequence, a compressed source semantic unit sequence and a compressed target semantic unit sequence are obtained; and
[0164] The first decoder is adjusted using the compressed source semantic unit sequence, the compressed target semantic unit sequence, and the task information.
[0165] Example 29. The electronic device of Examples 25-28, wherein the actions further include:
[0166] By compressing the source semantic unit sequence and the target semantic unit sequence, a source time series value sequence and a target time series value sequence are obtained, wherein the source time series value sequence and the target time series value sequence are related to the compression mode; and
[0167] A third decoder of the plurality of decoders is adjusted using the compressed source semantic unit sequence, the compressed target semantic unit sequence, the source timing value sequence, and the target timing value sequence.
[0168] Example 30. An electronic device according to Examples 25-29, wherein the semantic feature extractor includes one of an unsupervised model and a clustering model.
[0169] Example 31. An electronic device according to Examples 25-30, further comprising adjusting the first decoder using multi-task learning, wherein the multi-task learning includes at least one of the following:
[0170] Speech recognition tasks;
[0171] Text translation tasks; or
[0172] Speech-to-speech conversion task.
[0173] Example 32. An electronic device as described in Examples 25-31, wherein at least one of the source language audio and the target language audio comprises an unwritten language, wherein the unwritten language does not have handwriting.
[0174] Example 33. An electronic device comprising:
[0175] Processor; and
[0176] A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the electronic device to perform actions when executed by the processor, the actions being performed by a speech translation model generated according to any one of Examples 25 to 32, wherein the speech translation model comprises a semantic feature extractor and a plurality of decoders, the actions comprising:
[0177] generating a predicted target semantic unit sequence based on a given source semantic unit sequence of a given source language audio; and
[0178] A predicted acoustic unit sequence is generated based on the given source semantic unit sequence of the given source language audio, the predicted target semantic unit sequence, and the given source acoustic unit sequence.
[0179] Example 34. The electronic device according to Example 33, wherein the predicted target semantic unit sequence is generated:
[0180] Obtaining a first prediction prompt sequence by combining the given semantic unit sequence and task information; and
[0181] The predicted target semantic unit sequence is generated by inputting the first prediction hint sequence into a first decoder among the plurality of decoders.
[0182] Example 35. An electronic device according to Examples 33-34, wherein generating the predicted acoustic unit sequence comprises:
[0183] Obtaining a second prediction prompt sequence by combining the given source semantic unit sequence, the predicted target semantic unit sequence, and the given source acoustic unit sequence; and
[0184] The predicted acoustic unit sequence is generated by inputting the second prediction cue sequence into a second decoder of the plurality of decoders.
[0185] Example 36. The electronic device of Examples 33-35, wherein the actions further include:
[0186] Based on the predicted acoustic unit sequence, a predicted target language audio is generated.
[0187] Example 37. A computer-readable storage medium having one or more computer instructions stored thereon, wherein the one or more computer instructions are executed by a processor to implement a method according to any one of Examples 1 to 12.
[0188] Example 38. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform a method according to any one of Examples 1 to 12.
[0189] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A method for generating a speech translation model, the speech translation model comprising a semantic feature extractor and a plurality of decoders, the method include: The semantic feature extractor extracts a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, wherein the source language audio corresponds to the target language audio; Adjusting a first decoder among the plurality of decoders based on the source semantic unit sequence and the target semantic unit sequence; as well as Based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, adjust the second decoder among the multiple decoders, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
2. The method of claim 1, wherein adjusting the first decoder of the plurality of decoders include: Obtaining a first prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, and task information, wherein the task information at least specifies a language type of the source language audio and a language type of the target language audio; as well as Based on the first hint sequence, the first decoder is adjusted.
3. The method according to claim 2, wherein adjusting the second decoder among the plurality of decoders include: Obtaining a second prompt sequence by combining the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence, and the target acoustic unit sequence; as well as Based on the second hint sequence, the second decoder is adjusted.
4. The method according to claim 2, further comprising: include: By compressing the source semantic unit sequence and the target semantic unit sequence, a compressed source semantic unit sequence and a compressed target semantic unit sequence are obtained; as well as The first decoder is adjusted using the compressed source semantic unit sequence, the compressed target semantic unit sequence, and the task information.
5. The method according to claim 4, further comprising: include: By compressing the source semantic unit sequence and the target semantic unit sequence, a source time series value sequence and a target time series value sequence are obtained, wherein the source time series value sequence and the target time series value sequence are related to the compression mode; as well as A third decoder of the plurality of decoders is adjusted using the compressed source semantic unit sequence, the compressed target semantic unit sequence, the source timing value sequence, and the target timing value sequence.
6. The method according to claim 1, wherein the semantic feature extractor comprises any one of an unsupervised model and a clustering model.
7. The method of claim 2, further comprising tuning the first decoder using multi-task learning, wherein the multi-task learning comprises at least one of: Speech recognition tasks; Text translation tasks; or Speech-to-speech conversion task.
8. The method of any one of claims 1-7, wherein at least one of the source language audio and the target language audio comprises an unwritten language, wherein the unwritten language does not have a handwritten script.
9. A method for speech translation, the method being performed by a speech translation model generated according to any one of claims 1 to 8, wherein the speech translation model comprises a semantic feature extractor and a plurality of decoders, the method include: Based on a given source semantic unit sequence of a given source language audio, generating a predicted target semantic unit sequence; as well as A predicted acoustic unit sequence is generated based on the given source semantic unit sequence of the given source language audio, the predicted target semantic unit sequence, and the given source acoustic unit sequence.
10. The method according to claim 9, wherein the predicted target semantic unit sequence is generated include: Obtaining a first prediction prompt sequence by combining the given source semantic unit sequence and task information; as well as The predicted target semantic unit sequence is generated by inputting the first prediction hint sequence into a first decoder among the plurality of decoders.
11. The method according to claim 10, wherein the predicted acoustic unit sequence is generated include: Obtaining a second prediction prompt sequence by combining the given source semantic unit sequence, the predicted target semantic unit sequence, and the given source acoustic unit sequence; as well as The predicted acoustic unit sequence is generated by inputting the second prediction cue sequence into a second decoder of the plurality of decoders.
12. The method according to claim 9, further comprising: include: Based on the predicted acoustic unit sequence, a predicted target language audio is generated.
13. A device for generating a speech translation model, the speech translation model comprising a semantic feature extractor and a plurality of decoders, the device include: A semantic feature extraction module configured to extract a source semantic unit sequence of a source language audio and a target semantic unit sequence of a target language audio, the source language audio corresponding to the target language audio; A first decoding adjustment module is configured to adjust a first decoder among the multiple decoders based on the source semantic unit sequence and the target semantic unit sequence; as well as A second decoding adjustment module is configured to adjust a second decoder among the multiple decoders based on the source semantic unit sequence, the target semantic unit sequence, the source acoustic unit sequence of the source language audio, and the target acoustic unit sequence of the target language audio, wherein the semantic feature extractor remains unchanged during the adjustment process of the first decoder and the second decoder.
14. A device for speech translation, the device being configured to use the speech translation model generated according to claim 13, the device include: A first predicted semantic determination module is configured to generate a predicted target semantic unit sequence given a source semantic unit sequence of a given source language audio; as well as The first predicted acoustic determination module is configured to generate a predicted acoustic unit sequence based on a given source semantic unit sequence of the given source language audio, the predicted target semantic unit sequence, and a given source acoustic unit sequence.
15. An electronic device, include: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 12. 16 . A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to claim 1 .
