Model training method, speech-to-speech translation method, device and medium

By generating pseudo-noted speech-to-speech translation samples and combining real-sample training, the problem of lack of data in real scenarios of the speech-to-speech translation model is solved, and the training accuracy and translation performance of the model are improved.

CN114822499BActive Publication Date: 2024-11-01BEIJING YOUZHUJU NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210448585.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-26
Publication Date
2024-11-01
Estimated Expiration
2042-04-26

AI Technical Summary

Technical Problem

Existing speech-to-speech translation models are difficult to collect paired translation samples in real-world scenarios, resulting in lower training accuracy.

Method used

By obtaining speech recognition samples, a pseudo-noted speech-to-speech translation samples are generated, and the training model is trained in combination with real samples, the training data is expanded to improve the model accuracy.

Benefits of technology

It effectively improves the training accuracy of the speech-to-speech translation model, overcomes the problem of data scarcity, and enhances the translation performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114822499B_ABST
    Figure CN114822499B_ABST
Patent Text Reader

Abstract

The present application provides a model training method, a speech-to-speech translation method, a device and a medium, the method comprising: obtaining speech recognition samples and real speech-to-speech translation samples; generating pseudo-annotated speech-to-speech translation samples according to the speech recognition samples; and training a speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples. Thus, the model training accuracy can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of machine learning technology, and in particular to a model training method, a speech-to-speech translation method, a device, and a medium. Background Art

[0002] The Speech-to-Speech Translation (S2ST) model aims to translate the source language speech into the target language speech. It is widely used in various scenarios such as video translation, multinational conference speeches, and translation walkie-talkies. Usually, the speech-to-speech translation model needs to be trained with a large amount of data, but it is currently difficult to collect paired speech-to-speech translation samples in real scenarios. This lack of data leads to the problem of low model training accuracy. Summary of the invention

[0003] The present application provides a model training method, a speech-to-speech translation method, a device and a medium, thereby improving the accuracy of model training.

[0004] In a first aspect, a model training method is provided, comprising: obtaining speech recognition samples and real speech-to-speech translation samples; generating pseudo-annotated speech-to-speech translation samples based on the speech recognition samples; and training a speech-to-speech translation model based on the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples.

[0005] In a second aspect, a speech-to-speech translation method is provided, comprising: obtaining a source language speech feature; inputting the source language speech feature into a speech-to-speech translation model trained according to the first aspect or an optional method of the first aspect, to obtain a target language speech feature corresponding to the source language speech feature.

[0006] In a third aspect, a model training device is provided, including: an acquisition module, a generation module and a training module, the acquisition module is used to acquire speech recognition samples and real speech-to-speech translation samples; the generation module is used to generate pseudo-annotated speech-to-speech translation samples based on the speech recognition samples; the training module is used to train a speech-to-speech translation model based on the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples.

[0007] In a fourth aspect, a speech-to-speech translation device is provided, comprising: an acquisition module and a processing module, wherein the acquisition module is used to acquire source language speech features; the processing module is used to input the source language speech features into a speech-to-speech translation model trained according to the first aspect or the optional method of the first aspect, to obtain target language speech features corresponding to the source language speech features.

[0008] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method described in the first aspect or the second aspect.

[0009] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method described in the first aspect or the second aspect.

[0010] In summary, although it is difficult to collect paired speech-to-speech translation samples in real scenarios, speech recognition samples are relatively easy to collect. Based on these speech recognition samples, pseudo-annotated speech-to-speech translation samples can be generated, thereby expanding the speech-to-speech translation samples and further improving the model training accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 This is the framework diagram of Transformer;

[0013] Figure 2 A schematic diagram of a system architecture involved in an embodiment of the present application;

[0014] Figure 3 A flowchart of a model training method provided in an embodiment of the present application;

[0015] Figure 4 A flowchart of another model training method provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of a speech-to-speech translation model provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of another speech-to-speech translation model provided in an embodiment of the present application;

[0018] Figure 7 A flow chart of a speech-to-speech translation method provided in an embodiment of the present application;

[0019] Figure 8 A schematic diagram of a model training device 800 provided in an embodiment of the present application;

[0020] Fig. 9A schematic diagram of a speech-to-speech translation device 900 provided in an embodiment of the present application;

[0021] Fig.10 It is a schematic block diagram of an electronic device 1000 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0024] Before introducing the technical solution of this application, the following first describes the relevant knowledge of the technical solution of this application:

[0025] 1. Encoder and Decoder:

[0026] The encoder is used to process the source language speech features and compress the source language speech features into a hidden representation of a fixed length. The hidden representation is also called a context vector, semantic coding, semantic vector, etc. It is expected that the hidden representation can better represent the information of the language speech features.

[0027] Decoder: Use the hidden representation to initialize the decoder to obtain the target language speech features.

[0028] 2. Transformer:

[0029] Figure 1 This is the framework diagram of Transformer, such as Figure 1As shown in the figure, the encoder can be composed of N=6 identical units. Each unit contains two sub-units. The first is a self-attention network using a multi-head self-attention mechanism, and the second is a fully connected feedforward network with ReLU as the activation function. Both sub-units use residual connection (ADD) and layer normalization (Norm). The decoder is almost the same as the encoder, except that an additional layer of multi-head attention mechanism (encoder-decoder attention) can be added in the middle to process the output of the encoder. At the same time, the first unit of the decoder, that is, the first unit using the multi-head self-attention mechanism, performs a masking operation to ensure that the decoder does not read information after the current position.

[0030] The technical problems and inventive concepts to be solved by this application are described below:

[0031] As mentioned above, speech-to-speech translation models usually need to be trained with a large amount of data. However, it is currently difficult to collect paired speech-to-speech translation samples in real scenarios. This lack of data leads to the problem of low model training accuracy.

[0032] In order to solve the above technical problems, this application proposes to expand the training data to improve the accuracy of model training.

[0033] In some embodiments, the system architecture of the embodiments of the present application is as follows: Figure 2 shown.

[0034] Figure 2 This is a schematic diagram of a system architecture involved in an embodiment of the present application, including a user device 201, a data acquisition device 202, a training device 203, an execution device 204, a database 205 and a content library 206.

[0035] The data acquisition device 202 is used to read the training data from the content library 206 and store the read training data in the database 205. The training data involved in the embodiment of the present application includes pseudo-annotated speech-to-speech translation samples and real speech-to-speech translation samples.

[0036] The training device 203 trains the speech-to-speech translation model based on the training data maintained in the database 205, so that the trained speech-to-speech translation model can effectively realize the translation from the source language speech to the target language speech.

[0037] In addition, refer to Figure 2The execution device 204 is configured with an I / O interface 207 for data exchange with external devices. For example, the source language speech features sent by the user device 201 are received through the I / O interface. The calculation module 208 in the execution device 204 processes the input source language speech features using the trained speech-to-speech translation model, outputs the target language speech features, which can be the target language speech features, and sends the corresponding results to the user device 201 through the I / O interface.

[0038] The user device 201 may include a mobile phone, a tablet computer, a laptop computer, a PDA, a mobile internet device (MID), a desktop computer, or other terminal devices with a browser function installed.

[0039] The execution device 204 may be a server.

[0040] Exemplarily, the server may be a computing device such as a rack server, a blade server, a tower server or a cabinet server, etc. The server may be an independent test server or a test server cluster composed of multiple test servers.

[0041] In this embodiment, the execution device 204 is connected to the user device 201 through a network. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobilecommunication (GSM), Wideband Code Division Multiple Access (WCDMA), the 4th Generation (4G) network, the 5th Generation (5G) network, Bluetooth, wireless fidelity (Wi-Fi), a call network, etc.

[0042] It should be noted that Figure 2 It is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. In some embodiments, the data acquisition device 202 and the user device 201, the training device 203 and the execution device 204 can be the same device. The database 205 can be distributed on one server or on multiple servers, and the content library 206 can be distributed on one server or on multiple servers.

[0043] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0044] Figure 3 A flowchart of a model training method provided in an embodiment of the present application, which can be executed by any electronic device such as a mobile phone, tablet computer, laptop computer, PDA, MID, desktop computer, etc., for example, can be executed by Figure 2 The training device in the application is not limited to this. Figure 3 As shown, the method includes:

[0045] S310: Acquire speech recognition samples and real speech-to-speech translation samples;

[0046] S320: Generate a pseudo-annotated speech-to-speech translation sample according to the speech recognition sample;

[0047] S330: Training a speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples.

[0048] Optionally, the speech-to-speech translation model in the present application can be a speech-to-speech translation model based on multi-task learning (MTL), and of course, it can also be a speech-to-speech translation model based on single-task learning, and the present application does not limit this. Among them, multi-task learning is a very promising field in machine learning, and its goal is to use the useful information contained in multiple learning tasks to help learn a more accurate learner for each task. And in multi-task learning, since the inductive preference is shared between different tasks, tasks can generally improve each other and avoid single tasks from easily falling into local optimality.

[0049] For convenience, the real speech-to-speech translation samples, or pseudo-annotated speech-to-speech translation samples, are collectively referred to as speech-to-speech translation samples below. It should be understood that the number of data elements included in the speech-to-speech translation sample is related to whether the speech-to-speech translation model is based on multi-task learning or single-task learning. For example: if the speech-to-speech translation model is a speech-to-speech translation model based on single-task learning, the speech-to-speech translation sample can be a two-tuple, including: source language speech features and target language speech features. If the speech-to-speech translation model is a speech-to-speech translation model based on multi-task learning, and the multi-task includes: a main task and two auxiliary tasks, the main task is a speech-to-speech translation task; the two auxiliary tasks are speech recognition tasks and speech-to-text translation tasks, the speech recognition task is used to convert the source language speech features into source language text, the speech-to-text translation task is used to convert the source language speech features into source language text, and the source language text is converted into target language text, in this case, the speech-to-speech translation sample can be a four-tuple, including: source language speech features, source language text, target language speech features and target language text.

[0050] Optionally, the real speech-to-speech translation sample includes: a first source language speech feature, a first source language text, a first target language speech feature, and a first target language text.

[0051] It should be understood that the first source language speech feature is a real source language speech feature, the first source language text is a real source language text, the first target language text is also a real target language text, and the first target language speech feature is a target language speech feature obtained by the electronic device synthesizing the first target language text. For example, the electronic device can input the first target language text into a speech synthesis model to obtain the first target language speech feature.

[0052] Optionally, the speech recognition sample includes: a second source language speech feature and a second source language text, wherein the second source language speech feature is a real source language speech feature, and the second source language text is also a real source language text.

[0053] It should be understood that the so-called real source language speech features refer to the source language speech features that can be obtained in real scenes, for example, an electronic device can collect a user's voice through a microphone and extract the features of the voice. The real source language text can be a language text obtained by artificial means, for example, a user can record a voice to form a language text corresponding to the voice. The real target language text can also be a language text obtained by artificial means, for example, a user translates the content in the source language text into the target language text.

[0054] Optionally, the speech recognition sample may be one or more, and the real speech-to-speech translation sample may be one or more.

[0055] Optionally, the electronic device can translate the second source language text to obtain the second target language speech features; synthesize the second target language speech features to obtain the second target language speech features; wherein the pseudo-annotated speech-to-speech translation sample can be a four-tuple, including: the second source language speech features, the second source language text, the second target language text and the second target language speech features. In other words, the first two items in the pseudo-annotated speech-to-speech translation sample, namely the second source language speech features and the second source language text, are both true.

[0056] Optionally, the electronic device may input the second source language text into a machine translation (MT) model to obtain the second target language text. The electronic device may input the second target language text into a text-to-speech (TTS) model to obtain the second target language speech features.

[0057] It should be understood that according to the construction process of the pseudo-annotated speech-to-speech translation sample, the difference between the real speech-to-speech translation sample and the pseudo-annotated speech-to-speech translation sample is mainly in the target language text. For example, the real speech-to-speech translation sample is a four-tuple, including: {s src ,t src ,t tgt ,s tgt}, where s src Represents the true source language speech features, t src Indicates the true source language text, t tgt represents the real target language text, s tgt Indicates t tgt The target language speech features obtained after synthesis. The pseudo-annotated speech-to-speech translation sample is a four-tuple, including: {s src ,t src ,t' tgt ,s tgt}, where s src Represents the true source language speech features, t src Represents the true source language text, t' tgt It means the target language text obtained after inputting the real source language text into MT. tgt Indicates t' tgt The target language speech features obtained after synthesis.

[0058] In the present application, the speech-to-speech translation based on such pseudo-annotated speech-to-speech translation samples may be referred to as Pseudo Translation Label Adaptation (PTLA).

[0059] It should be understood that the above-mentioned pseudo-annotated speech-to-speech translation samples are obtained on the basis of speech recognition samples. In fact, pseudo-annotated speech-to-speech translation samples can also be constructed based on source language speech features. For example, an electronic device can obtain real source language speech features, input the source language speech features into an Automatic Speech Recognition (ASR) model, obtain a source language text corresponding to the source language speech features, and then input the source language text into an MT model to obtain a target language text. Finally, the target language text can be input into a TTS model to obtain target language speech features. Based on this, these source language speech features, source language text, target language text and target language speech features constitute pseudo-annotated speech-to-speech translation samples.

[0060] Optionally, in the present application, the source language speech feature may be a log-mel spectrogram of the source language speech, and the log-mel spectrogram may be an 80-channel log-mel spectrogram, but is not limited thereto.

[0061] Optionally, in the present application, the source language speech feature may be a linear frequency spectrogram of the target language speech, but is not limited thereto.

[0062] It should be understood that the training process of the speech-to-speech translation model by the electronic device includes: a pre-training stage (pre-training) and a fine tuning stage (fine tuning).

[0063] Pre-training refers to the process of pre-training a model or pre-training a model. Fine-tuning refers to the process of applying a pre-trained model to a dataset for a task and adapting the parameters to the dataset for that task.

[0064] For example: when a network model needs to be built to complete a specific image classification task, first, the electronic device needs to randomly initialize the parameters, and then start training the network model, and continuously adjust the parameters of the model so that the loss of the network model becomes smaller and smaller until the training stop condition is met. This process is the pre-training process. When an image classification task similar to the above-mentioned specific image classification task is obtained, the electronic device can directly use the previously trained network model, use the parameters of the network model as the initialization parameters of this task, and then train the network model, and continuously adjust the parameters of the model so that the loss of the network model becomes smaller and smaller until the training stop condition is met. This process is the fine-tuning process.

[0065] It should be understood that the above-mentioned real speech-to-speech translation sample can also be referred to as the original speech-to-speech translation sample. The pseudo-annotated speech-to-speech translation sample can also be referred to as a derived speech-to-speech translation sample. Wherein, the real speech-to-speech translation sample can act on the pre-training stage of the speech-to-speech translation model, and can also act on the fine-tuning stage of the model. The pseudo-annotated speech-to-speech translation sample can act on the pre-training stage of the speech-to-speech translation model, and can also act on the fine-tuning stage of the model, and this application does not limit this.

[0066] In summary, although it is difficult to collect paired speech-to-speech translation samples in real scenarios, speech recognition samples are relatively easy to collect. Based on these speech recognition samples, pseudo-annotated speech-to-speech translation samples can be generated, thereby expanding the speech-to-speech translation samples and further improving the model training accuracy.

[0067] The following will illustrate several model training methods:

[0068] like Figure 4 As shown, the above S330 includes:

[0069] S410: Pre-training a speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation samples;

[0070] S420: Fine-tune the pre-trained speech-to-speech translation model according to real speech-to-speech translation samples.

[0071] It should be understood that since the present application introduces pseudo-annotated speech-to-speech translation samples, in order to improve the model training accuracy, the pseudo-annotated speech-to-speech translation samples can be applied to the pre-training stage of the model, while the real speech-to-speech translation samples can be applied to the fine-tuning stage of the model.

[0072] Optionally, the electronic device may fine-tune the pre-trained speech-to-speech translation model directly through real speech-to-speech translation samples, that is, the electronic device fine-tunes the pre-trained speech-to-speech translation model only through real speech-to-speech translation samples.

[0073] Optionally, the electronic device may also fine-tune the pre-trained speech-to-speech translation model according to the real speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples. The model training method based on this approach may be referred to as a hybrid training method.

[0074] It should be understood that this hybrid training algorithm can maximize the retention of pseudo-annotated speech-to-speech translation sample benefits. Since the scale of speech recognition samples is much larger than that of real speech-to-speech translation samples, based on this, the scale of pseudo-annotated speech-to-speech translation samples is much larger than that of real speech-to-speech translation samples. In order to prevent pseudo-annotated speech-to-speech translation samples from misleading model optimization results, in this application, real speech-to-speech translation samples can be upsampled to expand the scale of real speech-to-speech translation samples, and then the pre-trained speech-to-speech translation model can be fine-tuned by the upsampled speech-to-speech translation samples and pseudo-annotated speech-to-speech translation samples.

[0075] Optionally, before fine-tuning the pre-trained speech-to-speech translation model according to the real speech-to-speech translation sample and the pseudo-annotated speech-to-speech translation sample, the electronic device can also annotate the real speech-to-speech translation sample with a first label, the first label is used to identify the real speech-to-speech translation sample as a real sample, which can be represented by real; and annotate the pseudo-annotated speech-to-speech translation sample with a second label, the second label is used to identify the pseudo-annotated speech-to-speech translation sample as a pseudo-annotated sample, which can be represented by pseudo. The model training method based on this method can be called a prompt training method. Based on this prompt training method, the model can better distinguish between real speech-to-speech translation samples and pseudo-annotated speech-to-speech translation samples.

[0076] In an embodiment of the present application, the electronic device can pre-train a speech-to-speech translation model based on pseudo-annotated speech-to-speech translation samples; and fine-tune the pre-trained speech-to-speech translation model based on real speech-to-speech translation samples, that is, the pseudo-annotated data is mainly used in the pre-training process. In this way, the pseudo-annotated speech-to-speech translation samples can be prevented from misleading the model optimization results.

[0077] Furthermore, due to the scarcity of real speech-to-speech translation samples, pseudo-annotated data can also be involved in the fine-tuning process. However, in order to prevent the pseudo-annotated speech-to-speech translation samples from misleading the model optimization results, the real speech-to-speech translation samples can be upsampled, and then the pre-trained speech-to-speech translation model can be fine-tuned using the upsampled speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples. This method can solve the problem of low model training accuracy caused by the scarcity of real speech-to-speech translation samples on the one hand, and prevent the pseudo-annotated speech-to-speech translation samples from misleading the model optimization results on the other hand.

[0078] Furthermore, before fine-tuning the pre-trained speech-to-speech translation model based on the real speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples, the electronic device may also annotate the real speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples with corresponding labels, so that the model can better distinguish between the real speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples.

[0079] Optionally, in the present application, the speech-to-speech translation model may be an existing Translatotron model or may be Figure 5 The speech-to-speech translation model shown is not limited in this application.

[0080] Figure 5 A schematic diagram of a speech-to-speech translation model provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the model includes: an encoder module 510, a first attention module 520, a first decoder module 530, N second attention modules 540 and N second decoder modules 550, where N is a positive integer, and the N second attention modules correspond to the N second decoder modules one by one. Figure 5 Taking N=2 as an example, of course N can also be equal to 1, 3, and so on.

[0081] Optionally, the model can be a speech-to-speech translation model based on multi-task learning, and the multi-task includes: a main task and N auxiliary tasks, the main task is a speech-to-speech translation task, wherein the number of the second attention modules and the second decoder modules is consistent with the number of auxiliary tasks, for example: N = 2, that is, there are two auxiliary tasks, and the two auxiliary tasks can be speech recognition tasks and speech-to-text translation tasks, but not limited to this. The speech recognition task is used to convert the source language speech features into the source language text, and the speech-to-text translation task is used to convert the source language speech features into the source language text, and convert the source language text into the target language text. For another example: N = 1, that is, there is an auxiliary task, and the auxiliary task can be a speech recognition task or a speech-to-text translation task, but not limited to this. The first attention module 520 and the first decoder module 530 correspond to the main task, and each group of the second attention module 540 and the second decoder module 550 below corresponds to an auxiliary task. The first decoder module 530 is mainly used to predict the synthesis of the target language speech features. During training, the two auxiliary tasks accept the input of the encoder module 510 and add the predicted loss value to the main task in the form of a weighted sum. During testing, the second decoder module 550 is not used.

[0082] The encoder module 510 is used to obtain source language speech features and process the source language speech features to obtain multiple groups of first hidden state representations corresponding to the source language speech features.

[0083] Optionally, Figure 6 A schematic diagram of another speech-to-speech translation model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the encoder module 510 includes: a convolutional neural network submodule 5101 and a first converter module 5102; the convolutional neural network submodule 5101 is used to obtain source language speech features, and process the source language speech features to obtain a second hidden state representation corresponding to the source language speech features; the first converter module 5102 is used to obtain the second hidden state representation, and process the second hidden state representation to obtain multiple groups of first hidden state representations.

[0084] Optionally, the convolutional neural network submodule 5101 may include two layers of convolutional neural network layers, but is not limited thereto. For example, after the 80-channel logarithmic Mel spectrogram is input into the convolutional neural network submodule 5101, the two layers of convolutional neural network layers can map the length of the 80-channel logarithmic Mel spectrogram to one-fourth of the original length, that is, assuming that the 80-channel logarithmic Mel spectrogram is represented by 100 vectors, each of which is 80-dimensional, then after being processed by two layers of convolutional neural network layers, 25 vectors are obtained. In order to be consistent with the number of hidden units in the first converter module 5102, if the number of hidden units is 512, then the dimension of the 25 vectors after being processed by two layers of convolutional neural network layers is also 512, where the 25 512-dimensional vectors can be understood as the above-mentioned second hidden state representation.

[0085] Optionally, the first converter module 5102 may be similar to Figure 1 The encoder structure shown, that is, the first converter module 5102 may include 6 converter layers, or 12 converter layers, which is not limited in the present application. Each converter layer may have 512 hidden units, that is, the hidden representation output by the converter layer may be 512-dimensional. Each converter layer may include two sub-units. The first is a self-attention network using a multi-head self-attention mechanism, where the self-attention network of the multi-head self-attention mechanism may be an 8-head self-attention mechanism self-attention network, which is not limited in the present application. The second is a fully connected feedforward network, where the feedforward network may use a 2048-dimensional internal state. Both sub-units use residual connections and layer normalization. After the first converter module 5102 processes the second hidden state representation, N groups of first hidden state representations can be obtained, where N represents the number of converter layers included in the first converter module 5102.

[0086] In combination with the above example, assuming that the convolutional neural network submodule 5101 outputs 25 512-dimensional vectors, then after processing by the first converter module 5102, N groups of first hidden state representations can be obtained, and each group of first hidden state representations is also 25 512-dimensional vectors, among which the 25 512-dimensional vectors obtained by the last layer of the first converter module 5102 can be output to the first attention module 520, and the 25 512-dimensional vectors obtained by the middle layer can be output to the second attention module 540.

[0087] The first attention module 520 is used to obtain a group of first hidden state representations from multiple groups of first hidden state representations and a first vector corresponding to each time step output by the first decoder, and process the group of first hidden state representations and the first vector corresponding to each time step to obtain a first attention representation corresponding to each time step.

[0088] The first decoder module 530 is used to obtain the second vector corresponding to each time step, and process the second vector corresponding to each time step to obtain the first vector corresponding to each time step, and output the first vector corresponding to each time step to the first attention module 520 to obtain the first attention representation corresponding to each time step, and process the first attention representation corresponding to each time step to obtain the target language speech feature corresponding to the source language speech feature.

[0089] Alternatively, if Figure 6 As shown, the first decoder module 530 includes: a pre-processing network (prenet) 5301, a second converter module 5302 and a post-processing network (postnet) 5303; the pre-processing network 5301 is used to obtain the second vector corresponding to each time step, and process the second vector corresponding to each time step to obtain the first vector corresponding to each time step, and output the first vector corresponding to each time step to the first attention module; the second converter module 5302 is used to obtain the first attention representation corresponding to each time step, and process the first attention representation corresponding to each time step to obtain the target language speech feature at each time step; the post-processing network 5303 is used to process the target language speech feature at each time step to obtain the target language speech feature corresponding to the source language speech feature.

[0090] Optionally, the bottleneck dimension of the pre-processing network 5301 can be 32.

[0091] In combination with the above example, in the training stage, the pre-processing network 5301 can obtain an 80-dimensional all-0 vector, which is the second vector corresponding to the first time step. The pre-processing network 5301 can process the all-0 vector to obtain a 512-dimensional all-0 vector, which is the first vector corresponding to the first time step. Further, the pre-processing network 5301 can input the all-0 vector into the first attention module 520, and the first attention module 520 can process the all-0 vector and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the first attention representation corresponding to the first time step. Further, the first attention module 520 inputs the first attention representation corresponding to the first time step into the second converter module 5302, and the second converter module 5302 can process the first attention representation corresponding to the first time step to obtain the target language speech feature at the first time step, which is the predicted target language speech feature at the first time step. In addition, the pre-processing network 5301 can also obtain the actual target language speech features at the first time step, which can be understood as the second vector corresponding to the second time step. The pre-processing network 5301 can process the second vector corresponding to the second time step to obtain a 512-dimensional vector, which is the first vector corresponding to the second time step. Further, the pre-processing network 5301 can input the first vector corresponding to the second time step into the first attention module 520, and the first attention module 520 can process the first vector corresponding to the second time step and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the first attention representation corresponding to the second time step. Further, the first attention module 520 inputs the first attention representation corresponding to the second time step into the second converter module 5302, and the second converter module 5302 can process the first attention representation corresponding to the second time step to obtain the target language speech features at the second time step, which are the predicted target language speech features at the second time step.

[0092] In summary, the pre-processing network 5301 can obtain the actual target language speech features at the i-th time step, i = 1, 2...M, M is the total number of time steps. The actual target language speech feature at the i-th time step can be understood as the second vector corresponding to the i+1-th time step. The pre-processing network 5301 can process the second vector corresponding to the i+1-th time step to obtain a 512-dimensional vector, which is the first vector corresponding to the i+1-th time step. Further, the pre-processing network 5301 can input the first vector corresponding to the i+1-th time step into the first attention module 520. The first attention module 520 can process the first vector corresponding to the i+1-th time step and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the first attention representation corresponding to the i+1-th time step. Furthermore, the first attention module 520 inputs the first attention representation corresponding to the i+1-th time step into the second converter module 5302. The second converter module 5302 can process the first attention representation corresponding to the i+1-th time step to obtain the target language speech feature at the i+1-th time step.

[0093] During the testing phase, the pre-processing network 5301 can obtain an 80-dimensional all-0 vector, which is the second vector corresponding to the first time step. The pre-processing network 5301 can process the all-0 vector to obtain a 512-dimensional all-0 vector, which is the first vector corresponding to the first time step. Further, the pre-processing network 5301 can input the all-0 vector into the first attention module 520, and the first attention module 520 can process the all-0 vector and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the first attention representation corresponding to the first time step. Further, the first attention module 520 inputs the first attention representation corresponding to the first time step into the second converter module 5302, and the second converter module 5302 can process the first attention representation corresponding to the first time step to obtain the target language speech feature at the first time step, which is the predicted target language speech feature at the first time step. Furthermore, the pre-processing network 5301 can process the predicted target language speech features at the first time step to obtain a 512-dimensional vector, wherein the predicted target language speech features at the first time step can be understood as the second vector corresponding to the above-mentioned second time step, and the 512-dimensional vector is the first vector corresponding to the second time step. Further, the pre-processing network 5301 can input the first vector corresponding to the second time step into the first attention module 520, and the first attention module 520 can process the first vector corresponding to the second time step and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the first attention representation corresponding to the second time step. Further, the first attention module 520 inputs the first attention representation corresponding to the second time step into the second converter module 5302, and the second converter module 5302 can process the first attention representation corresponding to the second time step to obtain the target language speech features at the second time step.

[0094] In summary, the pre-processing network 5301 can obtain the predicted target language speech features at the i-th time step, i = 1, 2...M, M is the total number of time steps. The predicted target language speech feature at the i-th time step can be understood as the second vector corresponding to the i+1-th time step. The pre-processing network 5301 can process the second vector corresponding to the i+1-th time step to obtain a 512-dimensional vector, which is the first vector corresponding to the i+1-th time step. Further, the pre-processing network 5301 can input the first vector corresponding to the i+1-th time step into the first attention module 520. The first attention module 520 can process the first vector corresponding to the i+1-th time step and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the first attention representation corresponding to the i+1-th time step. Further, the first attention module 520 inputs the first attention representation corresponding to the i+1-th time step into the second converter module 5302. The second converter module 5302 can process the first attention representation corresponding to the i+1-th time step to obtain the target language speech feature at the i+1-th time step.

[0095] Optionally, the second converter module 5302 may input the target language speech features at each time step to the post-processing network 5303, and the post-processing network 5303 may perform weighted summation on the target language speech features at each time step to obtain the target language speech features corresponding to the source language speech features.

[0096] Optionally, the second converter module 5302 may be similar to Figure 1 The decoder structure shown, that is, the second converter module 5302 may include 6 converter layers, which is not limited in the present application. The second converter module 5302 may use the same hyperparameters as the first converter module 5102.

[0097] In the training phase of the speech-to-speech translation model, the second attention module 540 is used to obtain a group of first hidden state representations from the multiple groups of first hidden state representations and a third vector corresponding to each time step output by the second decoder corresponding to the second attention module 540, and process the group of first hidden state representations and the third vector corresponding to each time step to obtain a second attention representation corresponding to each time step;

[0098] The second decoder module 550 corresponding to the second attention module 540 is used to obtain the fourth vector corresponding to each time step, and process the fourth vector corresponding to each time step to obtain the third vector corresponding to each time step, output the third vector corresponding to each time step to the second attention module 540, obtain the second attention representation corresponding to each time step, and process the second attention representation corresponding to each time step to obtain the auxiliary representation corresponding to the source language speech feature.

[0099] Optionally, the second decoder module 550 may include: a pre-processing network, a third converter module and a post-processing network; the pre-processing network is used to obtain the fourth vector corresponding to each time step, and process the fourth vector corresponding to each time step to obtain the third vector corresponding to each time step, and output the third vector corresponding to each time step to the second attention module 540; the third converter module is used to obtain the second attention representation corresponding to each time step, and process the second attention representation corresponding to each time step to obtain an auxiliary representation at each time step; the post-processing network is used to process the auxiliary representation at each time step to obtain the auxiliary representation corresponding to the source language speech feature.

[0100] Optionally, the bottleneck dimension of the pre-processing network can be 32.

[0101] Exemplarily, during the training phase, the pre-processing network 5301 can obtain an 80-dimensional embedding vector, which is the fourth vector corresponding to the first time step. The pre-processing network can process the vector to obtain a 512-dimensional vector, which is the third vector corresponding to the first time step. Further, the pre-processing network can input the vector into the second attention module 540, and the second attention module 540 can process the vector and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain a second attention representation corresponding to the first time step. Further, the second attention module 540 inputs the second attention representation corresponding to the first time step into the third converter module, and the third converter module can process the second attention representation corresponding to the first time step to obtain an auxiliary representation at the first time step, which is the predicted auxiliary representation at the first time step. In addition, the pre-processing network can also obtain the actual auxiliary representation at the first time step, which can be understood as the fourth vector corresponding to the second time step. The pre-processing network can process the fourth vector corresponding to the second time step to obtain a 512-dimensional vector, which is the third vector corresponding to the second time step. Further, the pre-processing network can input the third vector corresponding to the second time step into the second attention module 540, and the second attention module 540 can process the third vector corresponding to the second time step and the 25 512-dimensional vectors obtained from the encoder module 510 to obtain the second attention representation corresponding to the second time step. Further, the second attention module 540 inputs the second attention representation corresponding to the second time step into the third converter module, and the third converter module can process the second attention representation corresponding to the second time step to obtain the auxiliary representation at the second time step, which is the predicted auxiliary representation at the second time step.

[0102] In summary, the pre-processing network can obtain the actual auxiliary representation at the i-th time step, i=1,2...M, M is the total number of time steps. The actual auxiliary representation at the i-th time step can be understood as the fourth vector corresponding to the i+1-th time step, the pre-processing network can process the fourth vector corresponding to the i+1-th time step to obtain a 512-dimensional vector, the 512-dimensional vector is the third vector corresponding to the i+1-th time step, further, the pre-processing network can input the third vector corresponding to the i+1-th time step into the second attention module 540, the second attention module 540 can process the third vector corresponding to the i+1-th time step and the 25 512-dimensional vectors obtained from the encoder module 510, and obtain the second attention representation corresponding to the i+1-th time step, further, the second attention module 540 inputs the second attention representation corresponding to the i+1-th time step into the third converter module, the third converter module can process the second attention representation corresponding to the i+1-th time step, and obtain the auxiliary representation at the i+1-th time step.

[0103] Optionally, the third converter module may input the auxiliary representation at each time step to a post-processing network, and the post-processing network may perform weighted summation on the auxiliary representation at each time step to obtain an auxiliary representation corresponding to the source language speech feature.

[0104] Optionally, when the auxiliary task is a speech recognition task, the auxiliary representation can be a speech recognition result, such as a source language text corresponding to the source language speech. When the auxiliary task is a speech-to-text translation task, the auxiliary representation can be a speech translation result, such as a target language text.

[0105] It should be understood that the speech-to-speech translation model provided in the present application is a corresponding improvement on the existing translator (Translatotron) model, specifically replacing the long short-term memory network (Long Short-Term Memory, LSTM) in the translator (Translatotron) model with a converter module. In the present application, the speech-to-speech translation model can be referred to as a translator model based on a converter (Transformer-based Translatotron). In LSTM, the calculation on each time step is a local calculation, while in the converter module, the calculation on each time step is a global calculation, thereby improving the model accuracy.

[0106] It should be understood that the Transformer-based Translatotron provided in this application can be trained based on speech-to-speech translation samples other than pseudo-annotated ones, and the speech-to-speech model using this training method can be referred to as a baseline system. Of course, the Transformer-based Translatotron provided in this application can also be trained based on speech-to-speech translation samples with pseudo-annotated ones, and the speech-to-speech model using this training method can be referred to as a baseline system + PTLA.

[0107] This application provides the TEDEn2Zh dataset (English to Chinese) commonly used in speech translation to test the performance of the baseline system and the baseline system + PTLA, as shown in Table 1:

[0108] Table 1

[0109] method S-PER Tp-BLEU Dev-BLEU test-BLEU Baseline system 13.06 44.57 7.0 11.2 Baseline system + PTLA 11.93 48.64 10.4 18.0

[0110] Among them, S-PER represents the phoneme recognition error rate of the speech recognition task on the test set. Tp-BLEU represents the phoneme-based bilingual evaluation understudy (BLEU) of the speech-to-text translation task on the test set. Dev-BLER represents the phoneme-based BLEU of the main task on the development set. Test-BLEU represents the phoneme-based BLEU of the main task on the test set.

[0111] As can be seen from Table 1, the baseline system can achieve good performance on complex language translation, and the baseline system + PTLA solution can effectively improve the model performance.

[0112] A speech-to-speech translation method is provided below:

[0113] Figure 7 A flowchart of a speech-to-speech translation method provided in an embodiment of the present application, which method can be executed by any electronic device such as a mobile phone, tablet computer, laptop computer, PDA, MID, desktop computer, etc., for example, can be executed by Figure 2 The execution device in the embodiment is executed, and this application does not impose any restrictions on this. Figure 7 As shown, the method includes:

[0114] S710: Acquire source language speech features;

[0115] S720: Input the source language speech features into a speech-to-speech translation model to obtain the target language speech features corresponding to the source language speech features.

[0116] It should be understood that the speech-to-speech translation model can be trained by the above-mentioned model training method. Since the speech-to-speech translation model obtained by the above-mentioned training method has higher accuracy, based on this, speech-to-speech translation can be better achieved.

[0117] Figure 8 A schematic diagram of a model training device 800 provided in an embodiment of the present application is shown as follows: Figure 8 As shown, the device 800 includes: an acquisition module 810, a generation module 820 and a training module 830, wherein the acquisition module 810 is used to acquire speech recognition samples and real speech-to-speech translation samples; the generation module 820 is used to generate pseudo-annotated speech-to-speech translation samples according to the speech recognition samples; and the training module 830 is used to train a speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples.

[0118] Optionally, the training module 830 is specifically used to: pre-train a speech-to-speech translation model according to pseudo-annotated speech-to-speech translation samples, and fine-tune the pre-trained speech-to-speech translation model according to real speech-to-speech translation samples.

[0119] Optionally, the training module 830 is specifically used to: fine-tune the pre-trained speech-to-speech translation model through real speech-to-speech translation samples; or, fine-tune the pre-trained speech-to-speech translation model according to real speech-to-speech translation samples and pseudo-annotated speech-to-speech translation samples.

[0120] Optionally, the device 800 also includes: a labeling module 840, which is used to label the real speech-to-speech translation samples with a first label before fine-tuning the pre-trained speech-to-speech translation model based on the real speech-to-speech translation samples and the pseudo-labeled speech-to-speech translation samples, and the first label is used to identify the real speech-to-speech translation samples as real samples; and label the pseudo-labeled speech-to-speech translation samples with a second label, and the second label is used to identify the pseudo-labeled speech-to-speech translation samples as pseudo-labeled samples.

[0121] Optionally, the training module 830 is specifically used to: upsample real speech-to-speech translation samples to obtain upsampled speech-to-speech translation samples; and fine-tune the pre-trained speech-to-speech translation model using the upsampled speech-to-speech translation samples and pseudo-annotated speech-to-speech translation samples.

[0122] Optionally, the real speech-to-speech translation sample includes: first source language speech features, first source language text, first target language speech features and first target language text; and the speech recognition sample includes: second source language speech features and second source language text.

[0123] Optionally, the generation module 820 is specifically used to: translate the second source language text to obtain the second target language text; synthesize the second target language text to obtain the second target language speech features; wherein the pseudo-annotated speech-to-speech translation sample includes: the second source language speech features, the second source language text, the second target language text and the second target language speech features.

[0124] Optionally, the speech-to-speech translation model includes: an encoder module, a first attention module, a first decoder module, N second attention modules and N second decoder modules, N is a positive integer, and the N second attention modules and the N second decoder modules correspond one to one;

[0125] The encoder module is used to obtain source language speech features and process the source language speech features to obtain multiple groups of first hidden state representations corresponding to the source language speech features;

[0126] The first attention module is used to obtain a group of first hidden state representations from the multiple groups of first hidden state representations and a first vector corresponding to each time step output by the first decoder, and process the group of first hidden state representations and the first vector corresponding to each time step to obtain a first attention representation corresponding to each time step;

[0127] The first decoder module is used to obtain the second vector corresponding to each time step, and process the second vector corresponding to each time step to obtain the first vector corresponding to each time step, output the first vector corresponding to each time step to the first attention module, obtain the first attention representation corresponding to each time step, and process the first attention representation corresponding to each time step to obtain the target language speech feature corresponding to the source language speech feature;

[0128] In the training phase of the speech-to-speech translation model, the second attention module is used to obtain a group of first hidden state representations from the multiple groups of first hidden state representations and a third vector corresponding to each time step output by the second decoder corresponding to the second attention module, and process the group of first hidden state representations and the third vector corresponding to each time step to obtain a second attention representation corresponding to each time step;

[0129] The second decoder module corresponding to the second attention module is used to obtain the fourth vector corresponding to each time step, and process the fourth vector corresponding to each time step to obtain the third vector corresponding to each time step, output the third vector corresponding to each time step to the second attention module, obtain the second attention representation corresponding to each time step, and process the second attention representation corresponding to each time step to obtain the auxiliary representation corresponding to the source language speech feature.

[0130] Optionally, the encoder module includes: a convolutional neural network sub-module and a first converter module; the convolutional neural network sub-module is used to obtain source language speech features, and process the source language speech features to obtain a second hidden state representation corresponding to the source language speech features; the first converter module is used to obtain the second hidden state representation, and process the second hidden state representation to obtain multiple groups of first hidden state representations.

[0131] Optionally, the first decoder module includes: a pre-processing network, a second converter module and a post-processing network; the pre-processing network is used to obtain the second vector corresponding to each time step, and process the second vector corresponding to each time step to obtain the first vector corresponding to each time step, and output the first vector corresponding to each time step to the first attention module; the second converter module is used to obtain the first attention representation corresponding to each time step, and process the first attention representation corresponding to each time step to obtain the target language speech features at each time step; the post-processing network is used to process the target language speech features at each time step to obtain the target language speech features corresponding to the source language speech features.

[0132] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 8 The device 800 shown can execute the above-mentioned model training method embodiment, and the aforementioned and other operations and / or functions of each module in the device 800 are respectively for realizing the corresponding processes in the above-mentioned model training method, which will not be repeated here for the sake of brevity.

[0133] The above describes the device 800 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned model training method embodiment in conjunction with its hardware.

[0134] Fig. 9 A schematic diagram of a speech-to-speech translation device 900 provided in an embodiment of the present application is shown in FIG. Fig. 9As shown, the device 900 includes: an acquisition module 910 and a processing module 920, wherein the acquisition module 910 is used to acquire source language speech features; the processing module 920 is used to input the source language speech features into a speech-to-speech translation model trained by the above-mentioned model training method to obtain target language speech features corresponding to the source language speech features.

[0135] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig. 9 The device 900 shown can execute the above-mentioned speech-to-speech translation method embodiment, and the above-mentioned and other operations and / or functions of each module in the device 900 are respectively for realizing the corresponding processes in the above-mentioned speech-to-speech translation method, which will not be repeated here for the sake of brevity.

[0136] The above describes the device 900 of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned speech to speech translation method embodiment in conjunction with its hardware.

[0137] Fig.10 It is a schematic block diagram of an electronic device 1000 provided in an embodiment of the present application.

[0138] like Fig.10 As shown, the electronic device 1000 may include:

[0139] The memory 1010 and the processor 1020, the memory 1010 is used to store the computer program and transmit the program code to the processor 1020. In other words, the processor 1020 can call and run the computer program from the memory 1010 to implement the method in the embodiment of the present application.

[0140] For example, the processor 1020 may be configured to execute the above method embodiments according to instructions in the computer program.

[0141] In some embodiments of the present application, the processor 1020 may include but is not limited to:

[0142] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0143] In some embodiments of the present application, the memory 1010 includes but is not limited to:

[0144] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0145] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 1010 and executed by the processor 1020 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0146] like Fig.10 As shown, the electronic device may also include:

[0147] The transceiver 1030 may be connected to the processor 1020 or the memory 1010 .

[0148] The processor 1020 may control the transceiver 1030 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 1030 may include a transmitter and a receiver. The transceiver 1030 may further include an antenna, and the number of antennas may be one or more.

[0149] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0150] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0151] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0152] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0154] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0155] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Get speech recognition samples and real speech-to-speech translation samples; generating a pseudo-annotated speech-to-speech translation sample based on the speech recognition sample; Training a speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples; The pseudo-annotated speech-to-speech translation sample and the real speech-to-speech translation sample include: source language speech features and actual target language speech features; The speech-to-speech translation model includes: an encoder module, a first attention module and a first decoder module, the encoder module includes: a convolutional neural network submodule and a first converter module, the first decoder module includes: a pre-processing network, a second converter module and a post-processing network; The convolutional neural network submodule is used to obtain the source language speech feature and process the source language speech feature to obtain a second hidden state representation corresponding to the source language speech feature; The first converter module is used to obtain the second hidden state representation and process the second hidden state representation to obtain multiple groups of first hidden state representations; The first attention module is used to obtain a group of first hidden state representations from the multiple groups of first hidden state representations and a first vector corresponding to each time step output by the pre-processing network, and process the group of first hidden state representations and the first vector corresponding to each time step to obtain a first attention representation corresponding to each time step; The pre-processing network is used to obtain the second vector corresponding to each time step, and process the second vector corresponding to each time step to obtain the first vector corresponding to each time step, and output the first vector corresponding to each time step to the first attention module, wherein the second vector corresponding to the i+1th time step is the actual target language speech feature at the i-th time step, i=1, 2...M, M is the total number of time steps; The second converter module is used to obtain the first attention representation corresponding to each time step, and process the first attention representation corresponding to each time step to obtain the predicted target language speech feature at each time step; The post-processing network is used to process the predicted target language speech features at each time step to obtain the predicted target language speech features corresponding to the source language speech features.

2. The method according to claim 1, characterized in that The training of the speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation sample and the real speech-to-speech translation sample comprises: A speech-to-speech translation model is pre-trained according to the pseudo-annotated speech-to-speech translation samples, and the pre-trained speech-to-speech translation model is fine-tuned according to the real speech-to-speech translation samples.

3. The method according to claim 2, characterized in that The fine-tuning of the pre-trained speech-to-speech translation model according to the real speech-to-speech translation sample comprises: Fine-tune the pre-trained speech-to-speech translation model using the real speech-to-speech translation sample; or, The pre-trained speech-to-speech translation model is fine-tuned according to the real speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples.

4. The method according to claim 3, characterized in that Before fine-tuning the pre-trained speech-to-speech translation model according to the real speech-to-speech translation sample and the pseudo-annotated speech-to-speech translation sample, the method further includes: Annotating the real speech-to-speech translation sample with a first label, wherein the first label is used to identify the real speech-to-speech translation sample as a real sample; A second label is labeled on the pseudo-labeled speech-to-speech translation sample, where the second label is used to identify the pseudo-labeled speech-to-speech translation sample as a pseudo-labeled sample.

5. The method according to claim 3 or 4, characterized in that: The step of fine-tuning the pre-trained speech-to-speech translation model according to the real speech-to-speech translation sample and the pseudo-annotated speech-to-speech translation sample comprises: Upsampling the real speech-to-speech translation sample to obtain an upsampled speech-to-speech translation sample; The pre-trained speech-to-speech translation model is fine-tuned using the upsampled speech-to-speech translation samples and the pseudo-annotated speech-to-speech translation samples.

6. The method according to any one of claims 1 to 4, characterized in that: The real speech-to-speech translation sample includes: a first source language speech feature, a first source language text, a first target language speech feature and a first target language text; the speech recognition sample includes: a second source language speech feature and a second source language text.

7. The method according to claim 6, characterized in that The generating of a pseudo-annotated speech-to-speech translation sample according to the speech recognition sample comprises: Translating the second source language text to obtain a second target language text; synthesizing the second target language text to obtain second target language speech features; The pseudo-annotated speech-to-speech translation sample includes: the second source language speech feature, the second source language text, the second target language text and the second target language speech feature.

8. The method according to any one of claims 1 to 4, characterized in that: The speech-to-speech translation model further includes: N second attention modules and N second decoder modules, where N is a positive integer, and the N second attention modules and the N second decoder modules correspond one to one; In the training phase of the speech-to-speech translation model, the second attention module is used to obtain a group of first hidden state representations from the multiple groups of first hidden state representations and a third vector corresponding to each time step output by the second decoder corresponding to the second attention module, and process the group of first hidden state representations and the third vector corresponding to each time step to obtain a second attention representation corresponding to each time step; The second decoder module corresponding to the second attention module is used to obtain the fourth vector corresponding to each time step, and process the fourth vector corresponding to each time step to obtain the third vector corresponding to each time step, output the third vector corresponding to each time step to the second attention module, obtain the second attention representation corresponding to each time step, and process the second attention representation corresponding to each time step to obtain the auxiliary representation corresponding to the source language speech feature.

9. A speech-to-speech translation method, characterized in that: include: Acquire source language speech features; The source language speech features are input into a speech-to-speech translation model trained according to any one of claims 1 to 8 to obtain target language speech features corresponding to the source language speech features.

10. A model training device, characterized in that: include: An acquisition module, used to acquire speech recognition samples and real speech-to-speech translation samples; A generating module, configured to generate a pseudo-annotated speech-to-speech translation sample according to the speech recognition sample; A training module, used for training a speech-to-speech translation model according to the pseudo-annotated speech-to-speech translation samples and the real speech-to-speech translation samples; The pseudo-annotated speech-to-speech translation sample and the real speech-to-speech translation sample include: source language speech features and actual target language speech features; The speech-to-speech translation model includes: an encoder module, a first attention module and a first decoder module, the encoder module includes: a convolutional neural network submodule and a first converter module, the first decoder module includes: a pre-processing network, a second converter module and a post-processing network; The convolutional neural network submodule is used to obtain the source language speech feature and process the source language speech feature to obtain a second hidden state representation corresponding to the source language speech feature; The first converter module is used to obtain the second hidden state representation and process the second hidden state representation to obtain multiple groups of first hidden state representations; The first attention module is used to obtain a group of first hidden state representations from the multiple groups of first hidden state representations and a first vector corresponding to each time step output by the pre-processing network, and process the group of first hidden state representations and the first vector corresponding to each time step to obtain a first attention representation corresponding to each time step; The pre-processing network is used to obtain the second vector corresponding to each time step, and process the second vector corresponding to each time step to obtain the first vector corresponding to each time step, and output the first vector corresponding to each time step to the first attention module, wherein the second vector corresponding to the i+1th time step is the actual target language speech feature at the i-th time step, i=1, 2...M, M is the total number of time steps; The second converter module is used to obtain the first attention representation corresponding to each time step, and process the first attention representation corresponding to each time step to obtain the predicted target language speech feature at each time step; The post-processing network is used to process the predicted target language speech features at each time step to obtain the predicted target language speech features corresponding to the source language speech features.

11. A speech-to-speech translation device, characterized in that: include: An acquisition module, used for acquiring speech features of a source language; A processing module is used to input the source language speech features into a speech-to-speech translation model trained according to any one of the methods of claims 1 to 8 to obtain target language speech features corresponding to the source language speech features.

12. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 9.