Speech translation method, model, apparatus, device, and storage medium

Abstract

The present disclosure provides a speech translation method, model, device, equipment and storage medium, the method comprising: obtaining original speech data, and inputting the original speech data into a speech translation model, wherein the speech translation model comprises an encoder and a decoder; performing speech recognition on the original speech data through a low-layer calculation layer of the encoder to obtain a low-layer recognition result, wherein the speech recognition accuracy of the low-layer calculation layer is less than an accuracy threshold; performing speech recognition on the low-layer recognition result through a high-layer calculation layer of the encoding layer, and performing prompt word learning on the low-layer recognition result to obtain a learning recognition result, wherein the speech recognition accuracy of the high-layer calculation layer is greater than or equal to the accuracy threshold; and translating the learning recognition result through the decoder to obtain a speech translation result. The present disclosure is used to solve the defects of low calculation efficiency and low accuracy when speech translation is realized by two independent parts of speech recognition and text translation in the prior art.

Description

Technical Field

[0001] This disclosure relates to the field of speech processing technology, and in particular to a speech translation method, model, apparatus, device, and storage medium. Background Technology

[0002] Speech translation is a new task in the fields of artificial intelligence and natural language processing, used to convert speech in one language into text in another. Traditional speech translation methods consist of independent modules, each dedicated to a simpler subtask, which are ultimately integrated into a cascaded architecture. In existing technologies, speech translation mainly comprises two parts: speech recognition and text translation. The former generates text from audio, while the latter translates the text into the target language. However, speech recognition and text translation are two independent parts with no inter-processing involved. This leads to both parts requiring significant computational resources to achieve speech translation functionality, reducing computational efficiency and making it difficult to guarantee high-accuracy speech translation results. Summary of the Invention

[0003] This disclosure provides a speech translation method, model, apparatus, device, and storage medium to address the shortcomings of existing technologies that achieve speech translation through two independent parts—speech recognition and text translation—resulting in low computational efficiency and low accuracy.

[0004] This disclosure provides a speech translation method, comprising: acquiring raw speech data and inputting the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; performing speech recognition on the raw speech data through a low-level computational layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold; performing speech recognition on the low-level recognition result through a high-level computational layer of the encoder, and performing cue word learning on the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold; and translating the learned recognition result through the decoder to obtain a speech translation result.

[0005] According to a speech translation method provided in this disclosure, the encoder includes at least two computational layers connected in sequence; before acquiring raw speech data and inputting the raw speech data into a speech translation model, the method further includes: acquiring a preset probe dataset; based on the probe dataset, performing speech recognition tests on each computational layer in the encoder layer to obtain the speech recognition accuracy corresponding to each computational layer; starting from the computational layer where the data is input, sequentially comparing the speech recognition accuracy corresponding to each computational layer with an accuracy threshold until it is determined that the speech recognition accuracy of any computational layer is greater than or equal to the accuracy threshold; identifying at least one computational layer with a speech recognition accuracy less than the accuracy threshold as the lower-level computational layer; and identifying at least one computational layer with a speech recognition accuracy greater than or equal to the accuracy threshold as the higher-level computational layer.

[0006] According to a speech translation method provided in this disclosure, after determining at least one of the computational layers with a speech recognition accuracy greater than or equal to the accuracy threshold as the higher-level computational layers, the method further includes: embedding prompt word markers for each of the higher-level computational layers, wherein the prompt word markers are used to learn prompt words from the lower-level recognition results.

[0007] According to a speech translation method provided in this disclosure, after performing speech recognition on the low-level recognition result through the high-level computation layer of the encoding layer, and learning prompt words from the low-level recognition result to obtain a learned recognition result, and before translating the learned recognition result through the decoder to obtain a speech translation result, the method further includes: inputting the learned recognition result into a fully connected layer in the speech translation model; performing data dimension processing on the learned recognition result through the fully connected layer to obtain a target dimension learned recognition result; and translating the learned recognition result through the decoder to obtain a speech translation result, including: translating the target dimension learned recognition result through the decoder to obtain the speech translation result.

[0008] According to a speech translation method provided in this disclosure, the data categories of the probe dataset include grammatical information, semantic information, and / or surface information.

[0009] This disclosure also provides a speech translation model, including an encoder and a decoder. The encoder includes a low-level computational layer and a high-level computational layer. The low-level computational layer is used to perform speech recognition on the original speech data to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold. The high-level computational layer is used to perform speech recognition on the low-level recognition result and to learn prompt words from the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold. The decoder is used to translate the learned recognition result to obtain a speech translation result.

[0010] This disclosure also provides a speech translation model in which each of the higher-level computational layers embeds a prompt word marker.

[0011] This disclosure also provides a speech translation model, which further includes a fully connected layer; the fully connected layer is used to perform data dimension processing on the learned recognition results to obtain the learned recognition results in the target dimension.

[0012] This disclosure also provides a speech translation device, comprising: an acquisition module for acquiring raw speech data and inputting the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; a low-level recognition module for performing speech recognition on the raw speech data through a low-level computation layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computation layer is less than an accuracy threshold; a high-level recognition module for performing speech recognition on the low-level recognition result through a high-level computation layer of the encoder, and for learning prompt words from the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computation layer is greater than or equal to the accuracy threshold; and a translation module for translating the learned recognition result through the decoder to obtain a speech translation result.

[0013] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech translation methods.

[0014] This disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the speech translation method as described above.

[0015] The speech translation method, model, apparatus, device, and storage medium disclosed herein acquire raw speech data; input the raw speech data into an encoder in a speech translation model; perform speech recognition on the raw speech data through the low-level computational layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold; perform speech recognition on the low-level recognition result through a high-level computational layer of the encoding layer, and perform cue word learning on the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold; input the learned recognition result into a decoder in the speech translation model, and translate the learned recognition result through the decoder to obtain a speech translation result. In the above process, the high-level computational layer performs cue word learning, and the decoder translates the learned recognition result. That is, a cue word learning process is added to the speech recognition process, linking speech recognition and translation, preparing for the translation function of the decoder, reducing the translation difficulty of the decoder, and thus improving the computational efficiency and accuracy of the speech translation result. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the speech translation method disclosed herein;

[0018] Figure 2 This is a flowchart illustrating the voice detection task provided in this publication;

[0019] Figure 3 This is a schematic diagram of the structure of the publicly provided speech translation model;

[0020] Figure 4 This is a schematic diagram of the structure of the Transformer module provided in this disclosure;

[0021] Figure 5 This is a schematic diagram of the structure of the voice translation device provided in this disclosure;

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided in this disclosure. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this disclosure.

[0024] The following is combined with Figures 1-2 This disclosure describes a speech translation method according to an embodiment of the present disclosure.

[0025] In one embodiment, such as Figure 1 As shown, the speech translation method follows these steps:

[0026] Step 101: Obtain the raw speech data and input the raw speech data into the speech translation model, wherein the speech translation model includes an encoder and a decoder.

[0027] In this embodiment, speech translation (ST) is implemented through a speech translation model. This speech translation model is a machine model implemented using software algorithms. The model consists of two parts: an encoder and a decoder. The encoder is used for speech recognition, i.e., generating text from audio; the decoder is used for text translation, i.e., translating text into the target language. The encoder and decoder can be implemented by different sub-machine models. For example, the encoder can be based on an Automatic Speech Recognition (ASR) model, and the decoder can be based on a Machine Translation (MT) model. The speech translation model achieves end-to-end speech translation, directly translating source language audio into target language text information without any intermediate output. For example, the speech translation model directly outputs English audio as Chinese text.

[0028] In this embodiment, the encoder and decoder that make up the language translation model can be pre-trained language models. Pre-trained language models can utilize large-scale unlabeled data for pre-training, and then fine-tune the overall ST task. For multiple languages ​​and with a large amount of labeled speech translation data, even with limited text labels during speech translation, initializing the speech translation model using pre-trained weights can significantly improve the convergence accuracy and training performance of the ST model. For example, pre-trained language models can be used for initializing the speech translation model and integrating various sub-tasks (such as speech recognition and text translation) into a unified framework.

[0029] In this embodiment, when initializing the encoder and decoder using a pre-trained language model, the model structure of the pre-trained language model can be implemented using Convolutional Neural Networks (CNNs). The encoder includes at least two computational layers, and the decoder includes at least two computational layers.

[0030] In this embodiment, each computation layer in the encoder is an independent Transformer module, which can perform a single speech recognition computation. Similarly, each computation layer in the decoder is an independent Transformer module, which can perform a single text translation computation.

[0031] Step 102: Speech recognition is performed on the original speech data through the low-level computing layer of the encoder to obtain the low-level recognition result. The speech recognition accuracy of the low-level computing layer is less than the accuracy threshold.

[0032] In this embodiment, the pre-trained language model is a deep network structure. Simply combining two separate pre-trained language models reduces the interpretability of the downstream ST model. For the audio-to-text information flow conversion, the speech recognition accuracy doesn't significantly improve after some computation in the encoder's various layers. Therefore, the encoder can be configured to learn basic functions such as extracting, aligning, and fusing acoustic features in the lower computational layers, and then move to learning important knowledge, such as language attributes, in the higher computational layers for translation in the decoder. Thus, expanding the high-level representation capabilities of the pre-trained speech translation model is key to achieving better performance.

[0033] In one embodiment, for at least two computational layers connected sequentially in the encoder, before acquiring the original speech data and inputting it into the speech translation model, a preset probe dataset is acquired; based on the probe dataset, speech recognition testing is performed on each computational layer in the encoding layer to obtain the speech recognition accuracy corresponding to each computational layer; starting from the computational layer where the data is input, the speech recognition accuracy corresponding to each computational layer is compared with an accuracy threshold sequentially until it is determined that the speech recognition accuracy of any computational layer is greater than or equal to the accuracy threshold; at least one computational layer with a speech recognition accuracy less than the accuracy threshold is determined as a lower-level computational layer; at least one computational layer with a speech recognition accuracy greater than or equal to the accuracy threshold is determined as a higher-level computational layer.

[0034] In this embodiment, a probe dataset is pre-set to detect the speech recognition accuracy of each computational layer in the encoder. This probe dataset is set according to the required language scenario. For example, when speech translation needs to be implemented in the e-commerce field, a probe dataset is set based on the e-commerce field so that the final speech translation model can better adapt to the e-commerce field.

[0035] In one embodiment, the probe dataset is a fine-grained dataset, meaning the data belongs to a specific category and task with finer granularity, in order to better probe encoder performance. Specifically, the data categories in the probe dataset include syntactic information, semantic information, and / or surface information.

[0036] In this embodiment, syntactic information is the most basic and abstract type; it merely represents the phenomenon of things without considering the connotation of the information. Semantic information is a specific description of objective phenomena without making judgments about the merits or demerits of the phenomena themselves. Surface information is the highest level of information. It is based on syntactic and semantic information, and not only considers the relationships between states and their meanings, but also further examines the utility and value of these relationships and meanings for the information user.

[0037] In this embodiment, as shown in Table 1, the probe dataset includes three categories: surface information, syntactic information, and semantic information, and 10 tasks. The syntactic information includes 10 tasks: top constituent, tree depth, and bigram shift; the semantic information includes coordination, object number, past-present tense, subj number, and oddman out; and the surface information includes word content and sentence length. Each task has a corresponding training set, test set, validation set, and categories. The number of data samples for each task is set according to actual conditions and needs, as shown in Table 1.

[0038] Table 1. Detection dataset constructed in this invention

[0039]

[0040] In Table 1, the datasets for all tasks are balanced, with a uniform number of instances for each target category. Preferably, the size of the probe dataset is controlled at 10GB for more efficient speech detection.

[0041] In this embodiment, as Figure 2 As shown, the process of performing fine-grained speech detection tasks using the constructed detection dataset is as follows:

[0042] First, a classifier is pre-trained using different classification datasets. Speech samples from the aforementioned probe dataset are then input into the speech translation model, extracting the output of each computational layer of the encoder. Since the length of each speech segment varies, the length of the output features from each computational layer also differs. To use the output of each computational layer as input data for the classifier, it's necessary to ensure that the feature lengths of all samples are consistent. Therefore, the dimensionality of the output matrices of each computational layer is reduced to better fit the pre-trained classifier. For example, static pooling, a commonly used dimensionality reduction method in speaker recognition, can be employed for dimensionality reduction.

[0043] A pre-trained classifier is used to predict the output of each computational layer, and the predictions are compared with the corresponding sample categories of each speech sample in the probe dataset to obtain the predicted speech recognition accuracy. The speech information capture capability of each computational layer in the encoder is evaluated based on the predicted speech recognition accuracy.

[0044] Starting with the computational layer where the data is input, the speech recognition accuracy of each computational layer is compared with an accuracy threshold until it is determined that the speech recognition accuracy of any computational layer is greater than or equal to the accuracy threshold. At least one computational layer with a speech recognition accuracy less than the accuracy threshold is identified as a lower-level computational layer; at least one computational layer with a speech recognition accuracy greater than or equal to the accuracy threshold is identified as a higher-level computational layer. For example, when the encoder includes 20 computational layers, layers 1 to 12 are identified as lower-level computational layers, and layers 13 to 20 are identified as higher-level computational layers, based on speech recognition accuracy.

[0045] Step 103: Through the high-level computing layer of the coding layer, speech recognition is performed on the low-level recognition results, and prompt words are learned from the low-level recognition results to obtain the learned recognition results. The speech recognition accuracy of the high-level computing layer is greater than or equal to the accuracy threshold.

[0046] In this embodiment, the higher-level computing layer cannot significantly improve the accuracy of speech recognition. In order to strengthen the correlation between speech recognition and text translation, and to improve the utilization of the higher-level computing layer, in addition to implementing the speech recognition function, the higher-level computing layer adds a prompt word learning function to learn prompt words from the lower-level recognition results obtained by the lower-level computing layer, thereby obtaining the learned recognition result.

[0047] In one embodiment, after determining at least one computational layer with a speech recognition accuracy greater than or equal to an accuracy threshold as a high-level computational layer, a prompt word marker is embedded for each high-level computational layer, wherein the prompt word marker is used to learn prompt words from the low-level recognition results.

[0048] In this embodiment, the cue word learning function of the high-level computational layer is achieved by embedding cue word tokens in each high-level computational layer. These cue word tokens are continuous tokens and are trainable. By embedding continuous cue words as prefix tokens in each high-level computational layer, better cue words are automatically searched in the continuous space during machine training of the encoder, avoiding the tedious process of manually designing cue word templates and enhancing the language understanding ability of the pre-trained language model.

[0049] Step 104: Translate the learned recognition results using a decoder to obtain the speech translation result.

[0050] In this embodiment, a learning recognition result is obtained through a higher-level computing layer. This learning recognition result is text data derived from the original speech data, and the language of the original speech data is consistent with that of the learning recognition result. At this point, according to the decoder's preset settings, the learning recognition result can be translated into any desired language text to obtain the final speech translation result.

[0051] In one embodiment, the high-level computational layer of the encoding layer performs speech recognition on the low-level recognition results and learns prompt words from the low-level recognition results. After obtaining the learned recognition results, the decoder translates the learned recognition results. Before obtaining the speech translation results, the learned recognition results are input into the fully connected layer in the speech translation model. The fully connected layer processes the learned recognition results in terms of data dimensions to obtain the learned recognition results in the target dimension. The decoder translates the learned recognition results to obtain the speech translation results. The implementation process is as follows: The decoder translates the learned recognition results in the target dimension to obtain the speech translation results.

[0052] In this embodiment, since cue word markers are added to each high-level computing layer, a dimensionality mismatch occurs between the encoder and the decoder. Therefore, a fully connected layer is added between the encoder and the decoder to reduce the dimensionality of the learned recognition results output by the encoder so as to match the data dimension that the decoder can process.

[0053] In one embodiment, based on the speech translation method provided in the above embodiments, this disclosure also provides a speech translation model, which includes an encoder and a decoder. The encoder includes a low-level computing layer and a high-level computing layer. The low-level computing layer is used to perform speech recognition on the original speech data to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computing layer is less than an accuracy threshold. The high-level computing layer is used to perform speech recognition on the low-level recognition result and to learn prompt words from the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computing layer is greater than or equal to the accuracy threshold. The decoder is used to translate the learned recognition result to obtain a speech translation result.

[0054] In one embodiment, each higher-level computing layer embeds a prompt word marker.

[0055] In one embodiment, the speech translation model further includes a fully connected layer; the fully connected layer is used to perform data dimension processing on the learned recognition results to obtain the learned recognition results in the target dimension.

[0056] In a specific embodiment, such as Figure 3 As shown, the specific structure of the speech translation model is as follows:

[0057] Speech translation models consist of an encoder and a decoder. The encoder comprises a Convolutional Neural Network (CNN) and at least two computational layers (i.e., Transformers). The CNN processes the raw speech data into vector data for subsequent computation. The computational layers are divided into low-level and high-level layers. The low-level layers are used solely for speech recognition, while the high-level layers are used for both speech recognition and prompt word learning. Each high-level layer embeds prompt words.

[0058] The learned recognition result (i.e., Encoder Output) output by the encoder is input into the fully connected layer for dimensionality reduction, and then processed into the target dimension of the learned recognition result, which is then input into the decoder. For example, the learned recognition result is reduced to a 1×2048 feature vector and input into the decoder.

[0059] The decoder translates the target dimension's learned recognition results based on pre-set labeled text to obtain the final translated text (i.e., the speech translation result).

[0060] It should be noted that the cue word learners embedded in the higher-level computation layers are continuous cue words. During the model training phase, the cue word learners can be optimized through continuous reparameterization.

[0061] In one specific embodiment, each computational layer in the encoder or decoder is an independent Transformer module. For example... Figure 4 As shown, the internal structure of the Transformer module is as follows: The Transformer module includes a multi-headed attention mechanism, a feed-forward layer, a layer normalization layer, two feed-forward layers, an optional adapter, and a layer normalization layer.

[0062] The specific parameters of each layer in each Transformer module can be adjusted as needed based on actual circumstances.

[0063] The speech translation method and model disclosed herein acquire raw speech data; input the raw speech data into the encoder of the speech translation model; perform speech recognition on the raw speech data through the low-level computational layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold; perform speech recognition on the low-level recognition result through the high-level computational layer of the encoding layer, and learn prompt words from the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold; input the learned recognition result into the decoder of the speech translation model, and translate the learned recognition result to obtain the speech translation result. In the above process, the high-level computational layer performs prompt word learning, and the decoder translates the learned recognition result. That is, a prompt word learning process is added to the speech recognition process, linking speech recognition and translation, preparing for the translation function of the decoder, reducing the translation difficulty of the decoder, and thus improving the computational efficiency and accuracy of the speech translation result.

[0064] Specifically, by analyzing the performance of different network layers using fine-grained probe datasets, and without interfering with speech recognition, prompt word markers are added to the computational layer with the largest accuracy fluctuation in the probe encoder. By leveraging the advantages of continuous prompt learning, more effective cross-modal alignment and fusion are achieved, thereby improving the performance of the speech translation model, increasing speech translation efficiency, and improving the accuracy of the results.

[0065] The speech translation apparatus provided in the embodiments of this disclosure is described below. The speech translation apparatus described below can be referred to in correspondence with the speech translation method described above. For example... Figure 5 As shown, the voice translation device includes:

[0066] The acquisition module 501 is used to acquire raw speech data and input the raw speech data into the speech translation model, wherein the speech translation model includes an encoder and a decoder;

[0067] The low-level recognition module 502 is used to perform speech recognition on the original speech data through the low-level computing layer of the encoder to obtain the low-level recognition result, wherein the speech recognition accuracy of the low-level computing layer is less than the accuracy threshold.

[0068] The high-level recognition module 503 is used to perform speech recognition on the low-level recognition results through the high-level computing layer of the coding layer, and to learn prompt words from the low-level recognition results to obtain the learned recognition results. The speech recognition accuracy of the high-level computing layer is greater than or equal to the accuracy threshold.

[0069] The translation module 504 is used to translate the learned recognition results through the decoder to obtain the speech translation result.

[0070] In one embodiment, the encoder includes at least two computational layers connected in sequence;

[0071] The speech translation model also includes a preprocessing module for acquiring raw speech data and obtaining a pre-set probe dataset before inputting the raw speech data into the speech translation model. Based on the probe dataset, speech recognition tests are performed on each computational layer in the coding layer to obtain the speech recognition accuracy corresponding to each computational layer. Starting with the computational layer where the data is input, the speech recognition accuracy corresponding to each computational layer is compared with an accuracy threshold until it is determined that the speech recognition accuracy of any computational layer is greater than or equal to the accuracy threshold. At least one computational layer with a speech recognition accuracy less than the accuracy threshold is determined as a lower-level computational layer, and at least one computational layer with a speech recognition accuracy greater than or equal to the accuracy threshold is determined as a higher-level computational layer.

[0072] In one embodiment, the preprocessing module is used to determine at least one computational layer with a speech recognition accuracy greater than or equal to an accuracy threshold as a high-level computational layer, and then embed prompt word markers for each high-level computational layer, wherein the prompt word markers are used to learn prompt words from the low-level recognition results.

[0073] In one embodiment, the speech translation model further includes a connection module, which is used to perform speech recognition on the low-level recognition results through a high-level computational layer of the encoding layer, and to learn prompt words from the low-level recognition results. After obtaining the learned recognition results, the learned recognition results are translated through a decoder. Before obtaining the speech translation results, the learned recognition results are input into the fully connected layer in the speech translation model. Through the fully connected layer, the learned recognition results are processed in the data dimension to obtain the learned recognition results in the target dimension.

[0074] The translation module 504 is used to translate the learning and recognition results of the target dimension through the decoder to obtain the speech translation result.

[0075] In one embodiment, the data categories of the probe dataset include syntactic information, semantic information, and / or surface information.

[0076] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 601, a communications interface 602, a memory 603, and a communication bus 604. The processor 601, communications interface 602, and memory 603 communicate with each other via the communication bus 604. The processor 601 can call logical instructions in the memory 603 to execute a speech translation method. This method includes: acquiring raw speech data and inputting the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; performing speech recognition on the raw speech data through the low-level computational layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold; performing speech recognition on the low-level recognition result through the high-level computational layer of the encoding layer, and learning prompt words from the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold; and translating the learned recognition result through the decoder to obtain a speech translation result.

[0077] Furthermore, the logical instructions in the aforementioned memory 603 can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of this disclosure, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0078] On the other hand, this disclosure also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, enable the computer to perform the speech translation method provided by the above methods. The method includes: acquiring raw speech data and inputting the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; performing speech recognition on the raw speech data through a low-level computational layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold; performing speech recognition on the low-level recognition result through a high-level computational layer of the encoding layer, and performing cue word learning on the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold; and translating the learned recognition result through the decoder to obtain a speech translation result.

[0079] In another aspect, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the aforementioned speech translation methods. The method includes: acquiring raw speech data and inputting the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; performing speech recognition on the raw speech data through a low-level computational layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computational layer is less than an accuracy threshold; performing speech recognition on the low-level recognition result through a high-level computational layer of the encoding layer, and learning prompt words from the low-level recognition result to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computational layer is greater than or equal to the accuracy threshold; and translating the learned recognition result through the decoder to obtain a speech translation result.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A speech translation method, characterized in that, include: Acquire raw speech data and input the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; The low-level computing layer of the encoder performs speech recognition on the original speech data to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computing layer is less than the accuracy threshold. The encoder's high-level computing layer performs speech recognition on the low-level recognition results and learns prompt words from the low-level recognition results to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computing layer is greater than or equal to the accuracy threshold. The decoder translates the learned recognition results to obtain speech translation results.

2. The speech translation method according to claim 1, characterized in that, The encoder includes at least two computing layers connected in sequence; Before acquiring the raw speech data and inputting the raw speech data into the speech translation model, the method further includes: Obtain the preset detection dataset; Based on the probe dataset, a speech recognition test is performed on each of the computational layers in the encoder to obtain the speech recognition accuracy corresponding to each of the computational layers. Starting with the data input computation layer, the speech recognition accuracy rate corresponding to each computation layer is compared with the accuracy threshold in turn, until it is determined that the speech recognition accuracy rate of any computation layer is greater than or equal to the accuracy threshold. At least one of the computing layers whose speech recognition accuracy is less than the accuracy threshold is identified as the low-level computing layer; At least one of the computing layers whose speech recognition accuracy is greater than or equal to the accuracy threshold is determined as the higher-level computing layer.

3. The speech translation method according to claim 2, characterized in that, After determining at least one computing layer whose speech recognition accuracy is greater than or equal to the accuracy threshold as the higher-level computing layer, the method further includes: Each of the higher-level computing layers is embedded with a cue word marker, wherein the cue word marker is used to learn cue words from the lower-level recognition results.

4. The speech translation method according to claim 1, characterized in that, After performing speech recognition on the low-level recognition results through the high-level computing layer of the encoder, and learning cue words from the low-level recognition results to obtain a learned recognition result, before translating the learned recognition results through the decoder to obtain a speech translation result, the method further includes: The learned recognition results are input into the fully connected layer of the speech translation model; The fully connected layer is used to process the learning and recognition results in terms of data dimensions to obtain the learning and recognition results in the target dimension. The learned recognition results are translated using the decoder to obtain speech translation results, including: The decoder is used to translate the learning and recognition results of the target dimension to obtain the speech translation result.

5. The speech translation method according to claim 2, characterized in that, The data categories of the probe dataset include syntactic information, semantic information, and / or surface information.

6. A speech translation model, characterized in that, It includes an encoder and a decoder, wherein the encoder includes a low-level computing layer and a high-level computing layer; The low-level computing layer is used to perform speech recognition on the raw speech data to obtain low-level recognition results, wherein the speech recognition accuracy of the low-level computing layer is less than the accuracy threshold. The higher-level computing layer is used to perform speech recognition on the lower-level recognition results and to learn prompt words on the lower-level recognition results to obtain learned recognition results, wherein the speech recognition accuracy of the higher-level computing layer is greater than or equal to the accuracy threshold. The decoder is used to translate the learned recognition results to obtain speech translation results.

7. The speech translation model according to claim 6, characterized in that, Each of the higher-level computing layers embeds a prompt word marker.

8. The speech translation model according to claim 6, characterized in that, It also includes fully connected layers; The fully connected layer is used to process the learning and recognition results in terms of data dimensions to obtain the learning and recognition results in the target dimension.

9. A voice translation device, characterized in that, include: An acquisition module is used to acquire raw speech data and input the raw speech data into a speech translation model, wherein the speech translation model includes an encoder and a decoder; The low-level recognition module is used to perform speech recognition on the original speech data through the low-level computing layer of the encoder to obtain a low-level recognition result, wherein the speech recognition accuracy of the low-level computing layer is less than an accuracy threshold. The high-level recognition module is used to perform speech recognition on the low-level recognition results through the high-level computing layer of the encoder, and to learn prompt words from the low-level recognition results to obtain a learned recognition result, wherein the speech recognition accuracy of the high-level computing layer is greater than or equal to the accuracy threshold. The translation module is used to translate the learned recognition results through the decoder to obtain speech translation results.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech translation method as described in any one of claims 1 to 5.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech translation method as described in any one of claims 1 to 5.

Images