Speech translation method, electronic equipment and computer readable storage medium

Abstract

The invention discloses a speech translation method, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring an audio clip; performing representation extraction on the audio clip to obtain a plurality of initial high-order speech representations of the audio clip, and performing phoneme classification on each initial high-order speech representation to determine an initial phoneme set; matching phonemes in the initial phoneme set by using each proprietary vocabulary phoneme set in a target database constructed based on the proprietary vocabulary offset list to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio clip; replacing the initial high-order speech representation of the target proprietary vocabulary with a target high-order speech representation to form a high-order speech representation set of the audio clip; and translating the high-order speech table collection to obtain a target translated text of the audio clip. By means of phoneme-level proprietary vocabulary excitation, the generalization problem caused by pronunciation and accent differences in different application scenes in a speech translation task can be relieved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech translation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] As a downstream task of Automatic Speech Recognition (ASR), the performance of end-to-end speech translation is greatly affected by speech recognition. In many scenarios, the speech to be recognized may contain phrases (bias phrases, proprietary vocabulary) that are rarely or completely absent in the training data, such as proper nouns, names, and place names in certain fields. The accuracy of the recognition and translation of these proprietary vocabulary has a great impact on the user experience. Context bias integrates the contextual knowledge of the proprietary domain and incorporates additional proprietary vocabulary information to improve the recognition and translation accuracy of vocabulary in related fields. It is the mainstream solution for identifying personalized vocabulary. The proprietary vocabulary generation scheme mainly includes randomly sampling phrase fragments from text labels to simulate proprietary vocabulary, open source proprietary vocabulary sets (LibriSpeech, Aishell-1-NER) and user-customized proprietary vocabulary sets in the training stage, and finally obtains a proprietary vocabulary set in text format.

[0003] However, retrieving relevant information from an external professional terminology (proprietary vocabulary) database relies on the preliminary translation results of the speech translation system. For proprietary vocabulary outside the training set, the preliminary translation results of the speech translation system are often of poor quality, and it is difficult to retrieve relevant information directly from an external database search. In addition, the construction cost of a professional terminology database is high and difficult to expand, making it difficult to add a bias list of proprietary vocabulary in daily use scenarios. In existing contextual bias schemes, both rule-based proprietary vocabulary excitation and neural network-based proprietary vocabulary excitation generate text-level proprietary vocabulary bias lists. Different user groups have certain pronunciation and accent differences for the same vocabulary in different application scenarios, such as dialect scenarios. Text-level proprietary vocabulary excitation cannot meet the generalization requirements of audio signals with pronunciation and accent differences. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech translation method, an electronic device and a computer-readable storage medium, which can solve the technical problem of low generalization of pronunciation and accent differences in an end-to-end speech translation system in different usage scenarios.

[0005] In order to solve the above technical problems, a technical solution adopted in this application is: the speech translation method comprises:

[0006] Get the audio clip;

[0007] Extracting representations of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and classifying each of the initial high-order speech representations by phonemes to determine an initial phoneme set of the audio segment;

[0008] Using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list, matching the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment;

[0009] Replacing the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment;

[0010] The high-order speech representation set is translated to obtain a target translation text of the audio segment.

[0011] Optionally, extracting the representation of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and performing phoneme classification on each of the initial high-order speech representations to determine an initial phoneme set of the audio segment includes:

[0012] Extracting an initial high-order speech representation of the audio segment through a target speech encoder, and downsampling the initial high-order speech representation through an adapter;

[0013] Using a classifier, performing phoneme classification on each of the downsampled initial high-order speech representations according to a preset phoneme table to determine an initial phoneme set of the audio segment;

[0014] The translating the high-order speech representation set to obtain a target translation text of the audio segment includes:

[0015] The high-order speech representation set is input into a target language model for autoregressive generation to obtain a target translation text of the audio segment.

[0016] Optionally, before the method of using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment, the method further comprises:

[0017] Acquire the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of triplets of proprietary vocabulary;

[0018] Inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and downsampling the high-order speech representation;

[0019] According to the phoneme table, performing phoneme classification on the downsampled high-order speech representation of each of the proprietary words to obtain a proprietary vocabulary phoneme set of each of the proprietary words;

[0020] The target database is constructed based on the high-order speech representation of each of the special vocabulary and its special vocabulary phoneme set.

[0021] Optionally, the obtaining of the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of triples of proprietary vocabulary, including:

[0022] Get a translation dataset containing multiple source text labels;

[0023] Perform word segmentation and part-of-speech tagging on each of the original text labels to obtain a corresponding noun sequence;

[0024] Sampling from the noun sequence to input into the target language model for speech synthesis, to obtain a set of special vocabulary phonemes corresponding to each of the special vocabulary;

[0025] Acquire the source text and the translation text of each of the special words from the translation data set;

[0026] The source text, the special vocabulary phoneme set and the translation text of each of the special vocabulary are combined respectively to obtain the special vocabulary offset list.

[0027] Optionally, the step of inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and before downsampling the high-order speech representation, comprises:

[0028] Obtaining the translation dataset;

[0029] Based on the translation data set, training the first speech encoder to obtain the second speech encoder; and training the first bias encoder to obtain the second bias encoder;

[0030] Training the first language model based on the proprietary vocabulary offset list to obtain a second language model;

[0031] Based on the translation data set, the second speech encoder, the second bias encoder and the second language model are collaboratively trained to obtain the target speech encoder, the target bias encoder and the target language model.

[0032] Optionally, the translation data set includes a plurality of sample audios and corresponding phoneme categories; and the training of the first speech encoder based on the translation data set to obtain the second speech encoder includes:

[0033] Segmenting the sample audio and labeling it according to the phoneme category to obtain a first real phoneme label;

[0034] Inputting the sample audio into the first speech encoder for representation extraction, and using the classifier for phoneme classification to obtain a first predicted phoneme label;

[0035] Calculating a first phoneme classification loss of the first true phoneme label and the first predicted phoneme label, and using the first phoneme classification loss to train the first speech encoder;

[0036] The training is stopped in this way until the first phoneme classification loss reaches convergence, thereby obtaining the second speech encoder;

[0037] Optionally, the training the first bias encoder to obtain the second bias encoder includes:

[0038] Segmenting the special vocabulary audio in the sample audio and labeling it according to the phoneme category to obtain a second real phoneme label;

[0039] Inputting the special vocabulary audio into the first bias encoder for representation extraction, and using the classifier to perform phoneme classification to obtain a second predicted phoneme label;

[0040] Calculating a second phoneme classification loss of the second true phoneme label and the second predicted phoneme label, and using the second phoneme classification loss to train the first bias encoder;

[0041] The training is stopped in this way until the second phoneme classification loss reaches convergence, thereby obtaining the second bias encoder.

[0042] Optionally, the translation data set further includes original text labels, translated text labels, and translated text proprietary vocabulary positions;

[0043] The step of collaboratively training the second speech encoder, the second bias encoder, and the second language model based on the translation data set to obtain the target speech encoder, the target bias encoder, and the target language model comprises:

[0044] Inputting the training data into the second speech encoder for processing, inputting the proper vocabulary offset list into the second offset encoder for processing, so as to output corresponding training original text, training translation, and proper noun position of the training translation through the second language model;

[0045] Obtaining the translation loss of the training translation text and the translation text label, the language recognition loss of the second language model, and the position loss of the training translation text and the translation text proprietary vocabulary position;

[0046] The translation loss, the language identification loss, the position loss, the first phoneme classification loss, and the second phoneme classification loss are combined to calculate a collaborative overall loss, and the collaborative overall loss is used to train the second speech encoder, the second bias encoder, and the second language model;

[0047] The collaborative training is stopped in this way until the collaborative overall loss reaches convergence, and the target speech encoder, the target bias encoder and the target language model are obtained.

[0048] Optionally, the using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment includes:

[0049] Using the special vocabulary phoneme set of each special vocabulary in the target database, matching the phonemes in the initial phoneme set;

[0050] In response to at least one of the proprietary vocabulary phoneme sets in the target database matching the phonemes in the initial phoneme set, determining it as the target proprietary vocabulary, and using the high-order speech representation of the target proprietary vocabulary as the target high-order speech representation;

[0051] In response to each of the special vocabulary phoneme sets in the target database not matching the phonemes in the initial phoneme set, a preset unbiased symbol is used as the target high-order speech representation.

[0052] In order to solve the above technical problems, another technical solution adopted by the present application is: an electronic device, the electronic device comprising:

[0053] A memory for storing executable program codes;

[0054] The processor is used to call and run the executable program code from the memory, so that the electronic device executes any one of the speech translation methods described above.

[0055] To solve the above technical problems, another technical solution adopted in the present application is: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech translation method as described in any one of the above items is implemented.

[0056] Different from the text-level proprietary vocabulary excitation in the prior art, the embodiment of the present application provides a speech translation method, which includes: obtaining an audio segment; extracting representations of the audio segment to obtain multiple initial high-order speech representations of the audio segment, and classifying each of the initial high-order speech representations to determine the initial phoneme set of the audio segment; using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment; replacing the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment; translating the high-order speech representation set to obtain the target translation text of the audio segment. With the help of phoneme-level proprietary vocabulary excitation, the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks can be alleviated, and the generalization ability of the end-to-end speech translation system can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of the first embodiment of the speech translation method of the present application;

[0058] Figure 2 It is a flowchart of the second embodiment of the speech translation method of the present application;

[0059] Figure 3 This is a schematic diagram of the overall framework of the phoneme-level proprietary vocabulary-stimulated end-to-end speech translation of the speech translation method of the present application;

[0060] Figure 4 It is a flowchart of the third embodiment of the speech translation method of the present application;

[0061] Figure 5 It is a flowchart of the fourth embodiment of the speech translation method of the present application;

[0062] Figure 6 It is a flowchart of the fifth embodiment of the speech translation method of the present application;

[0063] Figure 7 is an exemplary structural block diagram of an electronic device of the speech translation method of the present application;

[0064] Figure 8 It is an exemplary structural block diagram of a computer-readable storage medium of the speech translation method of the present application. DETAILED DESCRIPTION

[0065] The present application is described in detail below with reference to the accompanying drawings and implementation methods.

[0066] The present invention relates to the fields of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP), and is a product-level End2End Speech-to-Text Translation (E2EST) system. It uses phoneme-level proprietary vocabulary incentives for customized usage scenarios of proprietary vocabulary in professional fields, can adapt to contextual biasing in proprietary fields, and improve the accuracy of the end-to-end speech translation system in specific scenarios.

[0067] As a downstream task of Automatic Speech Recognition (ASR), the performance of end-to-end speech translation is greatly affected by speech recognition. In many scenarios, the speech to be recognized may contain phrases (bias phrases, proprietary vocabulary) that are rarely or not present in the training data, such as proper nouns, names, and place names in certain fields. The accuracy of identifying and translating these phrases has a great impact on the user experience. However, since the speech context involves proprietary vocabulary phrases that change with the user's environment, identifying the translation context is challenging for end-to-end speech translation systems.

[0068] Context bias integrates domain-specific contextual knowledge and incorporates additional proprietary vocabulary information to improve the recognition and translation accuracy of related domain vocabulary. It is the mainstream solution for identifying personalized vocabulary. The proprietary vocabulary generation scheme mainly includes randomly sampling phrase fragments from text labels to simulate proprietary vocabulary, open source proprietary vocabulary sets (LibriSpeech, Aishell-1-NER) and user-customized proprietary vocabulary sets during the training phase, and finally obtains a proprietary vocabulary set in text format. However, considering that users have certain differences in pronunciation and accent for various words in usage scenarios, text-level proprietary vocabulary incentives will affect the accuracy of speech recognition and translation.

[0069] Context bias can also be called proprietary vocabulary excitation. Existing proprietary vocabulary excitation schemes that incorporate context information can be divided into two categories: rule-based proprietary vocabulary excitation and neural network-based proprietary vocabulary excitation methods. The details are as follows.

[0070] The rule-based proprietary vocabulary incentive gives a certain reward to the decoding path when matching a word in the proprietary vocabulary list during the decoding process, which can intuitively increase the posterior probability of the token related to the proprietary vocabulary. The rule-based proprietary vocabulary incentive is essentially shallow fusion (also known as post-fusion). If the proprietary vocabulary in the bias list is not involved in the training process, it will not appear in the decoding candidate path. The use of the bias list has limited improvement on performance and it is difficult to solve the problem of rare characters.

[0071] The neural network-based proprietary vocabulary excitation introduces an independent bias module for modeling and integrating biased proprietary vocabulary information. Proprietary vocabulary excitation is implemented during model training and inference. Compared with rule-based proprietary vocabulary excitation, it is more flexible and can quickly adapt to different proprietary vocabulary bias lists.

[0072] The end-to-end simultaneous interpretation framework (Cross Language Agent-Simultaneous Interpretation, CLASI) performs neural network-based proprietary vocabulary enhancement in the classic E2E ASR model LAS. The core idea is to randomly sample text phrases from text labels to simulate proprietary vocabulary during the training phase; add an additional bias encoder module, and introduce additional attention in the decoder to establish an attention connection between the text hidden state and the proprietary vocabulary embedding.

[0073] The early solution for speech translation was to cascade the ASR system and the machine translation (MT) system, first transcribing the speech into text, and then translating the text into the target language. However, in the continuous transcription or translation steps, the errors in the previous step will be accumulated and amplified in the subsequent steps, eventually leading to a decrease in translation quality. Compared with the cascade system, end-to-end speech translation can significantly reduce error accumulation and delay, and can have more context modeling and better generalization capabilities in various languages. The Large Language Model (LLM) has powerful generation capabilities. The latest end-to-end speech translation system CLASI uses an architecture that combines an audio encoder and LLM. The audio encoder converts the input speech into a continuous representation. The LLM receives the representation generated by the audio encoder and the retrieved related information and context, and generates the corresponding translation results.

[0074] However, the most advanced end-to-end speech translation system, taking CLASI as an example, solves the translation problem of professional vocabulary by building a corresponding professional terminology database. However, retrieving relevant information from an external professional terminology (proprietary vocabulary) database relies on the preliminary translation results of the speech translation system. For proprietary vocabulary outside the training set, the preliminary translation results of the speech translation system are often of poor quality, and it is difficult to retrieve relevant information directly by using external database retrieval. In addition, the construction cost of the professional terminology database is high and difficult to expand, and it is difficult to add a bias list of proprietary vocabulary in daily use scenarios. Therefore, this solution applies contextual bias to the end-to-end speech translation system, uses fixed proprietary vocabulary incentives during training, and adds proprietary vocabulary incentives outside the training set as needed during reasoning.

[0075] In existing contextual bias schemes, both rule-based and neural network-based proprietary vocabulary excitation generate text-level proprietary vocabulary bias lists. However, different user groups have certain pronunciation and accent differences for the same vocabulary in different application scenarios, such as dialect scenarios. Text-level proprietary vocabulary excitation cannot meet the generalization requirements of audio signals with pronunciation and accent differences. In order to ensure that the end-to-end speech translation system can significantly excite proprietary vocabulary with different pronunciations and accents, this scheme constructs a speech representation database of proprietary vocabulary and uses phoneme-level proprietary vocabulary excitation to alleviate the interference of pronunciation and accent on the speech translation system.

[0076] See also Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the speech translation method of the present application. It should be noted that if there are substantially the same results, the method of the present application is not limited to the first embodiment. Figure 1 The process sequence shown is limited. Figure 1 As shown, the speech translation method comprises:

[0077] S110, obtaining an audio clip;

[0078] Specifically, the audio segment can be a complete input audio, or a segment of the input audio. The language of the audio segment can be Chinese, English, French, etc., and can further be a dialect of various places.

[0079] S120, extracting representations of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and classifying each of the initial high-order speech representations into phonemes to determine an initial phoneme set of the audio segment;

[0080] Specifically, the representation extraction can convert the audio segment into a feature vector to capture the sound features in the audio. The initial high-order speech representation can be a plurality of high-order speech representations corresponding to the audio segment before matching the target proper vocabulary in the audio segment, and these initial high-order speech representations can be used to reflect the features of the semantic information of the audio segment. The initial phoneme set can be a plurality of phonemes corresponding to the audio segment. A phoneme can be the smallest speech unit in a language, such as each vowel and each consonant.

[0081] S130, using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set, to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment;

[0082] Among them, the proprietary vocabulary offset list can be used to construct a target database to stimulate proprietary vocabulary in the audio clip. The target database can be a database including high-order speech representations of proprietary vocabulary. The high-order speech representations in the target database can include a proprietary vocabulary phoneme set and a translation text of the proprietary vocabulary, which can be used to match the phonemes in the initial phoneme set of the audio clip to identify whether there is proprietary vocabulary in the audio clip, such as terms, names or names in a specific field or application. At least one target proprietary vocabulary can be one or more proprietary vocabulary in the audio clip.

[0083] Specifically, in order to ensure that proprietary words in audio clips can be correctly identified and processed, thereby improving the generalization effect of speech translation, the high-order speech representations of each proprietary word in the target database can be matched according to the high-order speech representations and phoneme categories in the audio clips, and then the proprietary words contained in the audio clips can be retrieved and matched. In this way, it can be identified whether specific proprietary words are mentioned in the audio clip, thereby realizing phoneme-level proprietary word excitation.

[0084] S140, replacing the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment;

[0085] Among them, the high-order speech representation set includes the initial high-order speech representation corresponding to the common words in the audio clip and the target high-order speech representation corresponding to the proprietary words, which can fully represent the speech content in the audio clip.

[0086] Specifically, the initial high-level representation of the proprietary words in the audio clip is replaced with the corresponding high-level speech representation of the proprietary words in the target database, thereby ensuring that the high-level speech representation set corresponding to the newly formed audio clip can accurately represent these proprietary words.

[0087] S150: Translate the high-order speech representation set to obtain a target translation text of the audio segment.

[0088] The target translation text may be a translation result of the audio segment.

[0089] Specifically, a set of high-order speech representations can be input into a language model for translation to generate a text representation of the audio clip, that is, the output result can at least include the target translated text, so that the audio clip can be understood and presented in written form while ensuring the consistency of proprietary vocabulary in the translation, thereby improving the generalization of speech translation.

[0090] In addition, the user can customize the output result through the operation instruction, and the output result can include not only the target translation text, but also the target source text and / or the position of the target special vocabulary in the target translation text. Among them, the target source text can be the content of the audio clip converted into written text.

[0091] In an exemplary embodiment, an audio clip is "You must bite the bullet", and a representation extraction is performed on the audio clip to obtain a plurality of initial high-order speech representations, and these initial high-order speech representations are classified by phonemes to obtain an initial phoneme set of the audio clip as "y u...b ai t ze...te". Assuming that a high-order speech representation of a proprietary word in the target database is (bite the bullet, b-ai-ze-ze-...-te, bite the bullet), the proprietary vocabulary phoneme set "b-ai-ze-ze-te" of the proprietary word corresponds to "b aize...te" in the initial phoneme set "y u...b ai ze...te", it can be determined that the proprietary word exists in the audio clip, and the corresponding initial high-order speech features in the audio clip are replaced with the target high-order speech representation of the proprietary word to form an updated high-order speech representation set of the audio clip. The target high-order speech representation may include, then, translating the high-order speech representation set to obtain the target source text "You must bite the bullet" of the audio segment "You must bite the bullet", the target translation text "You must bite your teeth", and the position of the target proper word "bite your teeth" in the target translation text "### bite your teeth". It should be noted that if the audio segment "You must bite the bullet" is not stimulated by the proper noun of this embodiment, the direct translation text is "You must bite the bullet".

[0092] This embodiment can alleviate the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks by using phoneme-level proprietary vocabulary excitation, thereby greatly enhancing the generalization capability of the end-to-end speech translation system.

[0093] Furthermore, the present embodiment takes into account that end-to-end speech translation is a downstream task of automatic speech recognition (ASR), and its performance is greatly affected by speech recognition. In many scenarios, the speech to be recognized may contain phrases (proper vocabulary, biasphrases) that are rarely or completely absent in the training data, such as proper nouns, names, and place names in certain fields. The accuracy of these phrase recognition and translation has a great impact on the user experience. However, since the speech context involves proprietary vocabulary phrases that change with the user's environment, it is challenging to recognize the translation context for an end-to-end speech translation system.

[0094] However, taking the end-to-end simultaneous interpretation system (Cross Language Agent-Simultaneous Interpretation, CLASI) as an example, the translation problem of professional vocabulary is solved by building a corresponding professional terminology database. However, retrieving relevant information from an external professional terminology (proprietary vocabulary) database relies on the preliminary translation results of the speech translation system. For proprietary vocabulary outside the training set, the preliminary translation results of the speech translation system are often of poor quality, and it is difficult to retrieve relevant information directly by using it for external database retrieval. In addition, the construction cost of the professional terminology database is high and difficult to expand, making it difficult to add a bias list of proprietary vocabulary in daily use scenarios.

[0095] Therefore, in order to ensure that the end-to-end speech translation system can significantly stimulate proprietary vocabulary with different pronunciations and accents, the embodiment of the present application proposes an end-to-end speech translation method with phoneme-level proprietary vocabulary stimulation, constructs a speech representation database of proprietary vocabulary, and adopts phoneme-level proprietary vocabulary stimulation to alleviate the interference of pronunciation and accent on the speech translation system.

[0096] See also Figure 2 , Figure 2 1 is a flow chart of the second embodiment of the speech translation method of the present application. In this embodiment, the method comprises the following steps:

[0097] S210, obtaining an audio clip;

[0098] The above-mentioned S110 may be described in detail, which will not be described again.

[0099] S220, extracting an initial high-order speech representation of the audio segment through a target speech encoder, and downsampling the initial high-order speech representation through an adapter;

[0100] Specifically, the target speech encoder can be a trained speech encoder (Audio Encoder), which can be used to extract high-order speech representations of audio clips. The adapter can downsample the high-order speech representations of the audio clips, maintain the key information of the high-order speech representations, and reduce the complexity of the data; and then adapt the dimension of the input layer of the large language model through linear projection, so that the large language model can process this information more efficiently. The adapter usually uses linear projection to complete this dimensionality reduction process.

[0101] S230, performing phoneme classification on each of the downsampled initial high-order speech representations according to a preset phoneme table by a classifier to determine an initial phoneme set of the audio segment;

[0102] Specifically, a phoneme table can be constructed based on prior knowledge. The phoneme table includes multiple minimum units of speech, such as vowels and consonants. By determining the initial phoneme set of the audio segment based on the phoneme table at the phoneme level, the speech data can be finely processed at the phoneme level.

[0103] S240, using the special vocabulary phoneme set of each special vocabulary in the target database to match the phonemes in the initial phoneme set;

[0104] Specifically, in an end-to-end speech translation system, some proprietary words do not often appear in general training data, resulting in inaccurate translation results of the translation system. Therefore, this embodiment constructs a target database through a proprietary word offset list, providing an incentive mechanism for these proprietary words. The proprietary word phoneme set of each proprietary word in the target database is matched with the phonemes in the initial phoneme set of the audio segment.

[0105] S251, in response to at least one of the proprietary vocabulary phoneme sets in the target database matching the phonemes in the initial phoneme set, determining it as the target proprietary vocabulary, and using the high-order speech representation of the target proprietary vocabulary as the target high-order speech representation;

[0106] Specifically, when the proprietary phoneme set of each proprietary word in the target database successfully matches a certain segment of phonemes in the initial phoneme set of the audio segment, the target high-order speech features of the target proprietary words can be used to replace the inaccurate initial high-order speech representation in the audio segment.

[0107] S252: In response to each of the proprietary vocabulary phoneme sets in the target database not matching the phonemes in the initial phoneme set, using a preset unbiased symbol as the target high-order speech representation.

[0108] Among them, the unbiased symbol is used as the default choice when there is no proprietary vocabulary and / or unknown vocabulary. Specifically, when the proprietary vocabulary phoneme set of each proprietary vocabulary in the target database fails to match a certain segment of phonemes in the initial phoneme set of the audio clip, the default unbiased symbol can be used for replacement. This mechanism can ensure that the system can still generate a reasonable high-level representation even when encountering unknown vocabulary, and avoid system crashes or translation failures due to the absence of proprietary vocabulary and / or unknown vocabulary or the inability to process unknown vocabulary.

[0109] S260, replacing the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment;

[0110] Specifically, since the traditional model in end-to-end speech translation relies on the intermediate step of speech-to-text, it is difficult to effectively convert speech signals into features that are useful for downstream translation tasks. Therefore, this embodiment extracts deep initial high-order speech features from the audio clip, and when the existence of proprietary words in the audio clip is detected, the target high-order speech features corresponding to the proprietary words replace the corresponding initial high-order speech features, thereby directly translating without the need for intermediate text representation.

[0111] S270: Input the high-order speech representation set into a target language model for autoregressive generation to obtain a target translation text of the audio segment.

[0112] Specifically, the large speech model only uses a decoder. Since the decoder uses an autoregressive generation method, that is, it gradually generates new outputs based on the previous input, generates a word or symbol each time, and then generates the next word based on the previous context. Therefore, this embodiment can input the high-order speech representation set corresponding to the audio segment into the large speech model for translation; it can also input the initial high-order speech features of the audio segment and the updated target high-order speech features into the large speech model in the order of sentences for translation.

[0113] The user can customize the output results through the operation instructions. The output results of the target language model may include not only the target translation text, but also the target source text and / or the position of the target proprietary vocabulary in the target translation text. The target source text may be the content of the audio clip converted into written text.

[0114] In an exemplary embodiment, see Figure 3First, the input audio clip is divided into 25ms audio sequences, and input into the speech encoder and adapter to obtain the initial high-order speech representation; then, the classifier is used to classify these initial high-order speech representations, and the phoneme category in the phoneme table corresponding to each initial high-order speech representation is obtained to form an initial phoneme set. By searching and comparing whether the audio clip contains proprietary vocabulary in the target database, if the phonemes in a proprietary vocabulary phoneme set in the target database appear continuously in the initial phoneme set, it means that the proprietary vocabulary appears in the audio clip, so the target high-order speech representation corresponding to the target proprietary vocabulary can be used to replace the corresponding initial high-order speech representation in the audio clip. Then, the initial high-order speech representation that has not been replaced and the updated target high-order speech representation are input into the large language model for translation, and the target source text, the target translation text and the position of the target proprietary vocabulary in the target translation text are output. Among them, each proprietary vocabulary phoneme set in the target database can be pre-input into the bias encoder through the proprietary vocabulary bias list and perform representation extraction and adapter to obtain multiple target high-order speech representations corresponding to each proprietary vocabulary, and then these target high-order speech representations are phoneme classified to obtain the proprietary vocabulary phoneme set corresponding to each proprietary vocabulary.

[0115] This embodiment, by using phoneme-level proprietary vocabulary excitation, can alleviate the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks, greatly enhance the generalization ability of the end-to-end speech translation system, and build a proprietary vocabulary speech representation database to meet the proprietary vocabulary customization needs of different user groups when using the speech translation system.

[0116] See also Figure 4 , Figure 4 It is a flowchart of the third embodiment of the speech translation method of the present application. Based on the above first embodiment, the construction process of the target database in this embodiment can be implemented before S130, for example, it can be implemented synchronously with S110 or S120, or it can be implemented before S110; based on the above second embodiment, the construction process of the target database can be implemented before S240, for example, it can be implemented synchronously with S210 or S220 or S230, or it can be implemented before S210. In this embodiment, the construction process of the target database can specifically include:

[0117] S410, obtaining the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of triplets of proprietary vocabulary;

[0118] Among them, triples can be used to construct a comprehensive representation of proprietary vocabulary, which can include the source text of the proprietary vocabulary, the phoneme set of the proprietary vocabulary, and the translated text. Therefore, by combining triples of different dimensions of proprietary vocabulary, it is ensured that the translation system can take into account both speech features and semantic information when processing complex speech translation tasks, thereby improving the processing effect of proprietary vocabulary.

[0119] In an exemplary embodiment, the special vocabulary offset list may be as follows: (bite the bullet, b-ai-ze-...-te, bite the bullet); (break the ice, bu-rei-k-...-ss, break the deadlock); (hit the nail, he-ei-t-...-ll, hit the nail on the head); (burn the oil, bo-e-rn-...-yao, burn the midnight oil); (between a rock, b-ei-tu-...-k, be caught between a rock and a hard place); (let the cat out, le-t-ze-...-t, let the cat slip); (jump on the wagon, zh-ang-m-...-en, go with the flow); (under the weather, ang-de-r-...-er, feel unwell); (piece of cake, pi-s-ev-...-k, a simple thing); (cost a leg, kao-st-...-g, cost a lot).

[0120] S420, inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and downsampling the high-order speech representation;

[0121] Specifically, the bias encoder is used to extract the high-order speech representation of the proprietary vocabulary audio. The triple of each proprietary vocabulary in the proprietary bias list is input into the target bias encoder to extract the high-order speech representation of the proprietary vocabulary. The high-order speech representation can also be downsampled using an adapter so that the high-order speech representation output by the bias encoder can also adapt to the target language model.

[0122] S430, performing phoneme classification on the down-sampled high-order speech representation of each of the proprietary words according to the phoneme table to obtain a proprietary vocabulary phoneme set of each of the proprietary words;

[0123] Specifically, a classifier can be used to classify the downsampled high-order speech representations according to the phoneme table. By dividing the high-order speech representations of proprietary vocabulary into smaller phoneme units to obtain a proprietary vocabulary phoneme set, the pronunciation characteristics of these vocabulary can be described more accurately.

[0124] S440: constructing the target database based on the high-order speech representation of each of the special words and the special word phoneme set.

[0125] Specifically, based on the high-order speech representation of each proprietary vocabulary and the phoneme set of the proprietary vocabulary, a target database is constructed, which records the high-order speech representation of the proprietary vocabulary and its corresponding phoneme representation information, so that these proprietary vocabulary can be quickly referenced and matched in the speech translation task.

[0126] In this embodiment, a target database of proprietary vocabulary speech representation is constructed by using a phoneme-level proprietary vocabulary offset list, so as to adopt phoneme-level proprietary vocabulary excitation and alleviate the generalization problem caused by pronunciation and accent in speech translation tasks.

[0127] See also Figure 5 , Figure 5 1 is a flow chart of the fourth embodiment of the speech translation method of the present application. In this embodiment, the steps of constructing the proprietary vocabulary offset list may include:

[0128] S510, obtaining a translation data set including a plurality of original text labels;

[0129] Specifically, the translation data set may include at least sample audio, original text labels corresponding to the sample audio, and translated text.

[0130] S520, performing word segmentation and part-of-speech tagging on each of the original text labels to obtain a corresponding noun sequence;

[0131] Specifically, since nouns occupy an important position in the proper vocabulary, especially in terms of domain, place names, personal names, etc., filtering out noun sequences can help the system focus on those words that are most likely to belong to the proper vocabulary and reduce unnecessary interference. Therefore, the system first performs part-of-speech tagging on the word sequences in the text and filters out all noun sequences. Part-of-speech tagging can be used to assign part-of-speech tags to given word sequences.

[0132] S530, sampling from the noun sequence to input into the target language model for speech synthesis, to obtain a set of special vocabulary phonemes corresponding to each of the special vocabulary;

[0133] Specifically, indefinite nouns are randomly sampled from all noun sequences, and low-frequency indefinite nouns are sampled from all noun sequences. Among them, by randomly sampling indefinite nouns, noun samples of different lengths and complexities can be obtained, which helps to improve the generalization ability of various proper nouns. Randomness can avoid overfitting to specific noun structures. Low-frequency nouns are more likely to be proper words, so by sampling low-frequency nouns, those proper words can be better covered.

[0134] Then, the text to speech (TTS) function of the large model can be used to generate audio of the sampled proprietary vocabulary, and the audio is cut into 25 ms frames to obtain the proprietary vocabulary phoneme set corresponding to the proprietary vocabulary.

[0135] S540, acquiring a source text and a translation text of each of the special words from the translation data set;

[0136] Specifically, the corresponding source text and translation text of the sampled special words are found in the original data set, so that the special words correspond to the corresponding source text and translation text.

[0137] S550: Combining the source text, the special vocabulary phoneme set and the translation text of each of the special vocabulary respectively to obtain the special vocabulary offset list.

[0138] Specifically, the final speech translation proprietary vocabulary bias list is composed of several triplets (proper vocabulary source text, proprietary vocabulary phoneme set, proprietary vocabulary translation text). At the same time, the bias lists of open source corpora such as Wikipedia and LibriSpeech can also be incorporated into the proprietary vocabulary bias list.

[0139] In addition, you can increase <no-bias>The character is used as the default choice when there is no special vocabulary.

[0140] In an exemplary implementation, the sample audio is: A mountain is not high, but it is famous if there are immortals. A water is not deep, but it is magical if there are dragons. This is a humble room, but my virtue is fragrant. The moss on the steps is green, and the grass is green in the curtains. There are great scholars talking and laughing, and there are no unlearned people coming and going. You can tune the plain zither and read the golden scriptures. There is no noise from stringed instruments, and no fatigue from paperwork. Zhuge Lu in Nanyang and Ziyun Pavilion in Western Shu. Confucius said: "What is humble about it?"

[0141] If part-of-speech tagging is not performed, random sampling from the label text will result in the following random proprietary words: Gao Youxian, Shi Loushi, Jie Lucaose, An Duzhi, Confucius, <no-bias>Among them, Gao Youxian, Jie Lucaose, and An Duzhi are all unreasonable random proprietary words.

[0142] If only nouns are randomly sampled after part-of-speech tagging, we will get: humble room, moss marks, great scholar, white man, plain zither, golden scripture, Ziyun Pavilion, Confucius, <no-bias>.in, <no-bias>Unbiased label.

[0143] In this embodiment, by extracting and sampling proprietary vocabulary from natural language processing and generating speech representations with the help of speech synthesis technology, a proprietary vocabulary offset list is constructed, wherein the proprietary vocabulary offset list includes the original text of the proprietary vocabulary corresponding to each proper noun, the audio phoneme set of the proprietary vocabulary, and the translation of the proprietary vocabulary. This allows the speech translation system to better recognize and translate proprietary vocabulary, thereby improving the performance when processing domain-specific or low-frequency vocabulary.

[0144] See also Figure 6 , Figure 6 1 is a flow chart of the fifth embodiment of the speech translation method of the present application. In this embodiment, the training process of the target speech encoder, the target bias encoder and the target language model includes:

[0145] S610, based on the translation data set, training the first speech encoder to obtain a second speech encoder; and training the first bias encoder to obtain a second bias encoder;

[0146] The translation data set may include each sample audio, the original text label corresponding to each sample audio, the sample audio phoneme set, the translation label and the translation-specific vocabulary position.

[0147] Specifically, in the first stage, the first speech encoder and the first bias encoder are trained separately to obtain the trained second speech encoder and the second bias encoder, so that the second speech encoder has the ability to extract representations and classify phonemes for the input audio clips, while ensuring accurate capture of different speech signals; at the same time, the second bias encoder has the ability to extract representations and classify phonemes for the input proprietary vocabulary in the proprietary vocabulary bias list, while ensuring that the proprietary vocabulary is given appropriate weights in the translation task.

[0148] In addition, in the initial individual training, there is less interdependence between encoders, which can effectively simplify the training process and enable each encoder to reach an ideal state on a single task.

[0149] Furthermore, the translation data set may include a plurality of sample audios and corresponding phoneme categories, and the training process of the second speech encoder may specifically include:

[0150] The sample audio is segmented and labeled according to the phoneme category to obtain a first real phoneme label; the sample audio is input into the first speech encoder for representation extraction, and the phoneme classification is performed using the classifier to obtain a first predicted phoneme label; the first phoneme classification loss of the first real phoneme label and the first predicted phoneme label is calculated, and the first speech encoder is trained using the first phoneme classification loss; this cycle is repeated until the first phoneme classification loss reaches convergence and the training is stopped to obtain the second speech encoder;

[0151] Specifically, the first speech encoder is trained using the translation data set, the sample audio in the translation data set is segmented, and the segmented sample audio segments are labeled with phonemes according to the phoneme table to obtain a first real phoneme label; the classifier performs phoneme classification on the high-order speech representation (embedding) output by the first speech encoder to obtain a first predicted phoneme label. Then, the first phoneme classification loss L between the first predicted phoneme label and the first real phoneme label is calculated using the cross-entropy loss (CE). AE , and back propagate the first phoneme classification loss to update the first speech encoder. Then, return to segment the sample audio and label it according to the phoneme category to obtain the first true phoneme label and the subsequent steps, and repeat until the first factor classification loss reaches convergence and stops training to obtain a trained second speech encoder.

[0152] Furthermore, the training process of the second bias encoder may specifically include:

[0153] The special vocabulary audio in the sample audio is segmented and labeled according to the phoneme category to obtain a second real phoneme label; the special vocabulary audio is input into the first bias encoder for representation extraction, and the phoneme classification is performed using the classifier to obtain a second predicted phoneme label; the second phoneme classification loss of the second real phoneme label and the second predicted phoneme label is calculated, and the first bias encoder is trained using the second phoneme classification loss; this cycle is repeated until the second phoneme classification loss reaches convergence and the training is stopped to obtain the second bias encoder.

[0154] Specifically, the first bias encoder is trained using the translation dataset, the proprietary vocabulary audio in the translation dataset is segmented, and the segmented proprietary vocabulary audio is labeled with phonemes according to the phoneme table to obtain a second true phoneme label; the classifier performs phoneme classification on the high-order speech representation output by the first bias encoder to obtain a second predicted phoneme label. Then, the second phoneme classification loss L between the second predicted phoneme label and the second true phoneme label is calculated using the cross entropy loss BE , and back propagate the second phoneme classification loss to update the first bias encoder. Then, return to segment the special vocabulary audio in the sample audio and mark it according to the phoneme category and the subsequent steps, and repeat this cycle until the second factor classification loss reaches convergence and stops training to obtain a trained second bias encoder.

[0155] S620, training the first language model based on the proprietary vocabulary offset list to obtain a second language model;

[0156] The first language model may be a large language model before fine-tuning training, and the second language model may be a large language model after fine-tuning training.

[0157] Specifically, because the frequency of proprietary vocabulary bias lists in the overall translation dataset is low, in order to enhance the large language model's proprietary translation capabilities, the first language model needs to be fine-tuned using the proprietary vocabulary bias list before collaborative training to obtain the second language model, ensuring that during the inference process, the second language model can correctly translate the target high-order speech representation output by the target bias encoder.

[0158] S630: Based on the translation data set, collaboratively train the second speech encoder, the second bias encoder, and the second language model to obtain the target speech encoder, the target bias encoder, and the target language model.

[0159] Specifically, in the third stage, the second speech encoder, the second bias encoder and the second language model are trained collaboratively, and the information between these models is shared and transmitted to ensure that they can work together consistently and efficiently when processing tasks. After collaborative training, the target speech encoder, target bias encoder and target language model are obtained. As a result, the target speech encoder and target bias encoder retain the ability to extract high-order implicit expressions and classifications of audio, while also training the target language model's translation ability for the overall translation data set and its ability to perceive the location of proprietary vocabulary in the translation.

[0160] Furthermore, S630 may specifically include:

[0161] The translation data set also includes original text labels, translated text labels, and translated text proprietary vocabulary positions;

[0162] The step of collaboratively training the second speech encoder, the second bias encoder, and the second language model based on the translation data set to obtain the target speech encoder, the target bias encoder, and the target language model comprises:

[0163] Inputting the training data into the second speech encoder for processing, inputting the proper vocabulary offset list into the second offset encoder for processing, so as to output corresponding training original text, training translation, and proper noun position of the training translation through the second language model;

[0164] Obtaining the translation loss of the training translation text and the translation text label, the language recognition loss of the second language model, and the position loss of the training translation text and the translation text proprietary vocabulary position;

[0165] The translation loss, the language identification loss, the position loss, the first phoneme classification loss, and the second phoneme classification loss are combined to calculate a collaborative overall loss, and the collaborative overall loss is used to train the second speech encoder, the second bias encoder, and the second language model;

[0166] The collaborative training is stopped in this way until the collaborative overall loss reaches convergence, and the target speech encoder, the target bias encoder and the target language model are obtained.

[0167] Specifically, the second speech encoder, the second bias encoder, and the second language model are trained collaboratively: using L Trans represents the loss between the training translation output by the second language model and the actual translation label, L ASR represents the speech recognition loss of the second language model, L Pos The position loss of the exclusive vocabulary position in the translation. At the same time, the first phoneme classification loss L of the second speech encoder and the second bias encoder AE and the second phoneme classification loss L BE Also participate in the training, the collaborative overall loss L can be expressed as: L = L Trans +L AE +L BE +L Pos +L ASR ;

[0168] Among them, the cross entropy loss is used to calculate the loss L of the position of the exclusive vocabulary in the perceived translation Pos .

[0169] In an exemplary embodiment, the sample audio is "What a genius Einsten was?", the original text label is "What a genius Einsten was?", the translated text label is "Einstein is really a genius", the translated text proper word position is "Einstein#####", and the sample audio phoneme set is "hh weins ta n zz"; the proper word offset list is (Einsten, eins-ta-n, Einstein); the second language model outputs "What a genius Einstenwas?", and the output training translation text is "Einstein is really a genius", and the translated text proper noun position is "Einstein#####".

[0170] In this embodiment, first, each model is trained individually to achieve the best state in its respective task, ensuring that they can handle basic tasks; the large language model is fine-tuned through a proprietary vocabulary bias list to make it more suitable for processing proprietary vocabulary and low-frequency vocabulary in specific fields, thereby improving the accuracy of translation; finally, through collaborative training, information sharing and coordination between various models are achieved, which improves the performance of the system in processing complex scenarios, ensures that the speech encoder, bias encoder and language model work closely together, and achieves higher quality speech translation effects.

[0171] See also Figure 7 , Figure 7 is an exemplary structural block diagram of an electronic device of the speech translation method of the present application. Figure 7 As shown, the electronic device 700 of the present application may include a processor 701 and a memory 702, wherein the processor 701 and the memory 702 communicate with each other via a bus. The memory 702 stores program instructions for speech translation, and when the program instructions are executed by the processor 701, the processor executes the above-mentioned related method steps to implement a speech translation method in the above-mentioned embodiment.

[0172] See also Figure 8 , Figure 8 is an exemplary structural block diagram of a computer-readable storage medium of the speech translation method of the present application. Figure 8 As shown, the computer-readable storage medium 800 stores a computer program 801. When the computer program 801 is executed by a processor on a computer, the computer executes the above-mentioned related method steps to implement a speech translation method in the above-mentioned embodiment.

[0173] The above scheme obtains an audio clip; extracts representations of the audio clip to obtain multiple initial high-order speech representations of the audio clip, and classifies each of the initial high-order speech representations into phonemes to determine the initial phoneme set of the audio clip; uses each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio clip; replaces the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio clip; translates the high-order speech representation set to obtain the target translation text of the audio clip. With the help of phoneme-level proprietary vocabulary excitation, the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks can be alleviated, greatly enhancing the generalization ability of the end-to-end speech translation system.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed methods, electronic devices and storage media can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the speech translation method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0178] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech translation method, characterized in that: The speech translation method comprises: Get the audio clip; Extracting representations of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and classifying each of the initial high-order speech representations by phonemes to determine an initial phoneme set of the audio segment; Using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list, matching the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment; Replacing the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment; The high-order speech representation set is translated to obtain a target translation text of the audio segment.

2. The speech translation method according to claim 1, characterized in that: The extracting the representation of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and classifying each of the initial high-order speech representations to determine an initial phoneme set of the audio segment, comprises: Extracting an initial high-order speech representation of the audio segment through a target speech encoder, and downsampling the initial high-order speech representation through an adapter; Using a classifier, performing phoneme classification on each of the downsampled initial high-order speech representations according to a preset phoneme table to determine an initial phoneme set of the audio segment; The translating the high-order speech representation set to obtain a target translation text of the audio segment includes: The high-order speech representation set is input into a target language model for autoregressive generation to obtain a target translation text of the audio segment.

3. The speech translation method according to claim 1, characterized in that: Before the method of using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment, the method includes: Acquire the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of triplets of proprietary vocabulary; Inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and downsampling the high-order speech representation; According to the phoneme table, performing phoneme classification on the downsampled high-order speech representation of each of the proprietary words to obtain a proprietary vocabulary phoneme set of each of the proprietary words; The target database is constructed based on the high-order speech representation of each of the special vocabulary and its special vocabulary phoneme set.

4. The speech translation method according to claim 3, characterized in that: Before obtaining the proprietary vocabulary offset list, the method includes: Get a translation dataset containing multiple source text labels; Perform word segmentation and part-of-speech tagging on each of the original text labels to obtain a corresponding noun sequence; Sampling from the noun sequence to input into the target language model for speech synthesis, to obtain a set of special vocabulary phonemes corresponding to each of the special vocabulary; Acquire the source text and the translation text of each of the special words from the translation data set; The source text, the special vocabulary phoneme set and the translation text of each of the special vocabulary are combined respectively to obtain the special vocabulary offset list.

5. The speech translation method according to any one of claims 2 to 4, characterized in that: The step of inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and before downsampling the high-order speech representation, includes: Obtaining the translation dataset; Based on the translation data set, training the first speech encoder to obtain the second speech encoder; and training the first bias encoder to obtain the second bias encoder; Training the first language model based on the proprietary vocabulary offset list to obtain a second language model; Based on the translation data set, the second speech encoder, the second bias encoder and the second language model are collaboratively trained to obtain the target speech encoder, the target bias encoder and the target language model.

6. The speech translation method according to claim 5, characterized in that: The translation data set includes a plurality of sample audios and corresponding phoneme categories; the training of the first speech encoder based on the translation data set to obtain the second speech encoder includes: Segmenting the sample audio and labeling it according to the phoneme category to obtain a first real phoneme label; Inputting the sample audio into the first speech encoder for representation extraction, and using the classifier for phoneme classification to obtain a first predicted phoneme label; Calculating a first phoneme classification loss of the first true phoneme label and the first predicted phoneme label, and training the first speech encoder using the first phoneme classification loss; The training is stopped in this way until the first phoneme classification loss reaches convergence, thereby obtaining the second speech encoder; The step of training the first bias encoder to obtain the second bias encoder includes: Segmenting the special vocabulary audio in the sample audio and labeling it according to the phoneme category to obtain a second real phoneme label; Inputting the special vocabulary audio into the first bias encoder for representation extraction, and using the classifier to perform phoneme classification to obtain a second predicted phoneme label; Calculating a second phoneme classification loss of the second true phoneme label and the second predicted phoneme label, and using the second phoneme classification loss to train the first bias encoder; The training is stopped in this way until the second phoneme classification loss reaches convergence, thereby obtaining the second bias encoder.

7. The speech translation method according to claim 6, characterized in that: The translation data set also includes original text labels, translated text labels, and translated text proprietary vocabulary positions; The step of collaboratively training the second speech encoder, the second bias encoder, and the second language model based on the translation data set to obtain the target speech encoder, the target bias encoder, and the target language model comprises: Inputting the training data into the second speech encoder for processing, inputting the proper vocabulary offset list into the second offset encoder for processing, so as to output corresponding training original text, training translation, and proper noun position of the training translation through the second language model; Obtaining the translation loss of the training translation text and the translation text label, the language recognition loss of the second language model, and the position loss of the training translation text and the translation text proprietary vocabulary position; The translation loss, the language identification loss, the position loss, the first phoneme classification loss, and the second phoneme classification loss are combined to calculate a collaborative overall loss, and the collaborative overall loss is used to train the second speech encoder, the second bias encoder, and the second language model; The collaborative training is stopped in this way until the collaborative overall loss reaches convergence, and the target speech encoder, the target bias encoder and the target language model are obtained.

8. The speech translation method according to claim 4, characterized in that: The method of using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment includes: Using the special vocabulary phoneme set of each special vocabulary in the target database, matching the phonemes in the initial phoneme set; In response to at least one of the proprietary vocabulary phoneme sets in the target database matching the phonemes in the initial phoneme set, determining it as the target proprietary vocabulary, and using the high-order speech representation of the target proprietary vocabulary as the target high-order speech representation; In response to each of the special vocabulary phoneme sets in the target database not matching the phonemes in the initial phoneme set, a preset unbiased symbol is used as the target high-order speech representation.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable program codes; A processor is used to call and run the executable program code from the memory, so that the electronic device executes the speech translation method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech translation method according to any one of claims 1 to 8 is implemented.

Images