Speech translation method, electronic device, and computer-readable storage medium

Abstract

The application discloses a speech translation method, an electronic device and a computer readable storage medium. The method comprises: obtaining an audio segment; performing feature extraction on the audio segment to obtain a plurality of initial high-order speech features of the audio segment, and performing phoneme classification on each initial high-order speech feature to determine an initial phoneme set; matching phonemes in the initial phoneme set with each special vocabulary phoneme set in a target database constructed based on a special vocabulary bias list to obtain a target high-order speech feature of at least one target special vocabulary in the audio segment; replacing the initial high-order speech feature of the target special vocabulary with the target high-order speech feature to form a high-order speech feature set of the audio segment; and performing translation on the high-order speech feature set to obtain a target translation text of the audio segment. With the phoneme-level special vocabulary excitation, the generalization problem caused by pronunciation and accent differences in different application scenarios in the speech translation task can be relieved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech translation method, electronic device, and computer-readable storage medium. Background Art

[0002] As a downstream task of Automatic Speech Recognition (ASR), the performance of end-to-end speech translation is significantly influenced by speech recognition. In many scenarios, the speech to be recognized may contain phrases (bias phrases, proprietary vocabulary) that are rarely or completely absent in the training data, such as proper nouns, names, and place names in certain fields. The accuracy of the recognition and translation of these proprietary vocabulary significantly impacts the user experience. Contextual bias integrates domain-specific contextual knowledge and incorporates additional proprietary vocabulary information to improve the recognition and translation accuracy of related domain vocabulary. It is a mainstream solution for identifying personalized vocabulary. Proprietary vocabulary generation schemes mainly involve randomly sampling phrase fragments from text labels to simulate proprietary vocabulary during the training phase, using open source proprietary vocabulary sets (LibriSpeech, Aishell-1-NER), and user-customized proprietary vocabulary sets, ultimately resulting in a text-based proprietary vocabulary set.

[0003] However, searching for relevant information in an external database of professional terminology (proprietary vocabulary) relies on the preliminary translation results of the speech translation system. For proprietary vocabulary outside the training set, the preliminary translation results of the speech translation system are often of poor quality, and it is difficult to retrieve relevant information directly when used for external database retrieval. In addition, the construction cost of a professional terminology database is high and difficult to expand, making it difficult to add a bias list of proprietary vocabulary in daily use scenarios. In existing contextual bias schemes, whether it is rule-based proprietary vocabulary excitation or neural network-based proprietary vocabulary excitation, both generate text-level proprietary vocabulary bias lists. Different user groups have certain pronunciation and accent differences for the same vocabulary in different application scenarios, such as dialect scenarios. Text-level proprietary vocabulary excitation cannot meet the generalization requirements of audio signals with pronunciation and accent differences. Summary of the Invention

[0004] The main technical problem solved by this application is to provide a speech translation method, electronic device and computer-readable storage medium, which can solve the technical problem of low generalization of pronunciation and accent differences in end-to-end speech translation systems in different usage scenarios.

[0005] To solve the above technical problems, a technical solution adopted in this application is: the speech translation method includes:

[0006] Get the audio clip;

[0007] Extracting representations of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and performing phoneme classification on each of the initial high-order speech representations to determine an initial phoneme set of the audio segment;

[0008] Matching the phonemes in the initial phoneme set with the phoneme sets of the target vocabulary in the target database constructed based on the proprietary vocabulary offset list to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment;

[0009] Replacing the initial high-order speech representation of the target proper vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment;

[0010] The high-order speech representation set is translated to obtain a target translated text of the audio segment.

[0011] Optionally, extracting the representation of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and performing phoneme classification on each of the initial high-order speech representations to determine the initial phoneme set of the audio segment includes:

[0012] extracting an initial high-order speech representation of the audio segment through a target speech encoder, and downsampling the initial high-order speech representation through an adapter;

[0013] Performing phoneme classification on each of the downsampled initial high-order speech representations according to a preset phoneme table using a classifier to determine an initial phoneme set of the audio segment;

[0014] The translating the high-order speech representation set to obtain a target translated text of the audio segment includes:

[0015] The high-order speech representation set is input into a target language model for autoregressive generation to obtain a target translation text of the audio segment.

[0016] Optionally, before obtaining a target high-order speech representation of at least one target proprietary vocabulary in the audio segment by matching the phonemes in the initial phoneme set with the phonemes in the target database constructed based on the proprietary vocabulary offset list, the method further comprises:

[0017] Acquire the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of proprietary vocabulary triples;

[0018] Inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and downsampling the high-order speech representation;

[0019] According to the phoneme table, performing phoneme classification on the downsampled high-order speech representation of each of the proprietary words to obtain a proprietary vocabulary phoneme set of each of the proprietary words;

[0020] The target database is constructed based on the high-order speech representation of each of the proprietary words and the proprietary word phoneme set.

[0021] Optionally, the obtaining of the proprietary vocabulary offset list includes, before the triples of the plurality of proprietary vocabulary, the proprietary vocabulary offset list includes:

[0022] Obtain a translation dataset containing multiple source text labels;

[0023] Perform word segmentation and part-of-speech tagging on each of the original text labels to obtain a corresponding noun sequence;

[0024] Sampling from the noun sequence to input into the target language model for speech synthesis, to obtain a set of proprietary vocabulary phonemes corresponding to each of the proprietary vocabulary;

[0025] Acquire the source text and the translated text of each of the proprietary words from the translation dataset;

[0026] The source text, the special vocabulary phoneme set and the translation text of each special vocabulary are respectively combined to obtain the special vocabulary offset list.

[0027] Optionally, the step of inputting each triple into a target bias encoder for representation extraction to obtain a high-order speech representation of each proprietary word, and before downsampling the high-order speech representation, comprises:

[0028] Obtaining the translation dataset;

[0029] Based on the translation data set, training the first speech encoder to obtain a second speech encoder; and training the first bias encoder to obtain a second bias encoder;

[0030] Training the first language model based on the proprietary vocabulary offset list to obtain a second language model;

[0031] Based on the translation data set, the second speech encoder, the second bias encoder, and the second language model are collaboratively trained to obtain the target speech encoder, the target bias encoder, and the target language model.

[0032] Optionally, the translation dataset includes a plurality of sample audios and corresponding phoneme categories; and the training of the first speech encoder based on the translation dataset to obtain the second speech encoder includes:

[0033] Segmenting the sample audio and labeling it according to the phoneme category to obtain a first true phoneme label;

[0034] Inputting the sample audio into the first speech encoder for characterization extraction, and performing phoneme classification using the classifier to obtain a first predicted phoneme label;

[0035] Calculating a first phoneme classification loss of the first true phoneme label and the first predicted phoneme label, and training the first speech encoder using the first phoneme classification loss;

[0036] Repeat this cycle until the first phoneme classification loss reaches convergence, stopping the training to obtain the second speech encoder;

[0037] Optionally, the training the first bias encoder to obtain the second bias encoder includes:

[0038] Segmenting the proper vocabulary audio in the sample audio and labeling it according to the phoneme category to obtain a second true phoneme label;

[0039] Inputting the proprietary vocabulary audio into the first bias encoder for representation extraction, and performing phoneme classification using the classifier to obtain a second predicted phoneme label;

[0040] Calculating a second phoneme classification loss for the second true phoneme label and the second predicted phoneme label, and training the first bias encoder using the second phoneme classification loss;

[0041] This cycle is repeated until the second phoneme classification loss reaches convergence, and the training is stopped to obtain the second bias encoder.

[0042] Optionally, the translation dataset further includes original text labels, translated text labels, and locations of specific vocabulary in the translated text;

[0043] The step of collaboratively training the second speech encoder, the second bias encoder, and the second language model based on the translation dataset to obtain the target speech encoder, the target bias encoder, and the target language model includes:

[0044] Inputting the training data into the second speech encoder for processing, and inputting the proper vocabulary offset list into the second offset encoder for processing, so as to output corresponding training original text, training translation, and proper noun positions of the training translation through the second language model;

[0045] Obtaining translation loss of the training translation text and the translation text label, language recognition loss of the second language model, and position loss of the training translation text and the translation text proper vocabulary position;

[0046] Calculating a collaborative overall loss by combining the translation loss, the language identification loss, the position loss, the first phoneme classification loss, and the second phoneme classification loss, and using the collaborative overall loss to train the second speech encoder, the second bias encoder, and the second language model;

[0047] This cycle is repeated until the collaborative overall loss reaches convergence, and the collaborative training is stopped to obtain the target speech encoder, the target bias encoder, and the target language model.

[0048] Optionally, the step of matching the phonemes in the initial phoneme set with the phoneme sets of the target vocabulary in the target database constructed based on the proprietary vocabulary offset list to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment includes:

[0049] Matching the phonemes in the initial phoneme set using the proprietary phoneme set of each proprietary word in the target database;

[0050] In response to at least one of the proprietary vocabulary phoneme sets in the target database matching the phonemes in the initial phoneme set, determining the target proprietary vocabulary as the target proprietary vocabulary, and using the high-order speech representation of the target proprietary vocabulary as the target high-order speech representation;

[0051] In response to each of the proprietary phoneme sets in the target database not matching the phonemes in the initial phoneme set, a preset unbiased symbol is used as the target high-order speech representation.

[0052] To solve the above technical problems, another technical solution adopted in this application is: an electronic device, comprising:

[0053] a memory for storing executable program code;

[0054] The processor is configured to call and run the executable program code from the memory, so that the electronic device executes any one of the above speech translation methods.

[0055] To solve the above technical problems, another technical solution adopted in this application is: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech translation method as described in any one of the above is implemented.

[0056] Different from the text-level proprietary vocabulary excitation in the prior art, the embodiment of the present application provides a speech translation method, which includes: obtaining an audio clip; extracting representations of the audio clip to obtain multiple initial high-order speech representations of the audio clip, and performing phoneme classification on each of the initial high-order speech representations to determine the initial phoneme set of the audio clip; using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio clip; replacing the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio clip; translating the high-order speech representation set to obtain a target translation text of the audio clip. With the help of phoneme-level proprietary vocabulary excitation, the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks can be alleviated, thereby enhancing the generalization capability of the end-to-end speech translation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flowchart of the first embodiment of the speech translation method of the present application;

[0058] Figure 2 This is a flow chart of the second embodiment of the speech translation method of the present application;

[0059] Figure 3 This is a schematic diagram of the overall framework of the phoneme-level proprietary vocabulary-stimulated end-to-end speech translation method of the present application;

[0060] Figure 4 This is a flowchart of the third embodiment of the speech translation method of the present application;

[0061] Figure 5 This is a flowchart of the fourth embodiment of the speech translation method of the present application;

[0062] Figure 6 This is a flowchart of the fifth embodiment of the speech translation method of the present application;

[0063] Figure 7 This is an exemplary structural block diagram of an electronic device for the speech translation method of the present application;

[0064] Figure 8 This is an exemplary structural block diagram of a computer-readable storage medium for the speech translation method of the present application. DETAILED DESCRIPTION

[0065] The present application is described in detail below with reference to the accompanying drawings and implementation methods.

[0066] The present invention relates to the fields of Automatic Speech Recognition (ASR) and Natural Language Processing (NLP), and is aimed at a product-level end-to-end speech-to-text translation (E2EST) system. This system uses phoneme-level proprietary vocabulary incentives for customized usage scenarios of specialized domains, can adapt to contextual biasing in specialized domains, and improve the accuracy of the end-to-end speech translation system in specific scenarios.

[0067] As a downstream task of Automatic Speech Recognition (ASR), the performance of end-to-end speech translation is largely influenced by speech recognition. In many scenarios, the speech to be recognized may contain phrases (bias phrases, proprietary vocabulary) that are rarely or completely absent in the training data, such as proper nouns in certain fields, names of people, and places. The accuracy of identifying and translating these phrases has a significant impact on the user experience. However, because the speech context involves proprietary phrases that vary with the user's environment, identifying the translation context is challenging for end-to-end speech translation systems.

[0068] Context bias integrates domain-specific contextual knowledge and incorporates additional proprietary vocabulary information to improve the recognition and translation accuracy of related domain vocabulary. It is a mainstream solution for identifying personalized vocabulary. Proprietary vocabulary generation schemes primarily involve randomly sampling phrase fragments from text labels during the training phase to simulate proprietary vocabulary, open source proprietary vocabulary sets (LibriSpeech, Aishell-1-NER), and user-customized proprietary vocabulary sets, ultimately resulting in a text-based proprietary vocabulary set. However, given the varying pronunciation and accents of various words in user usage scenarios, text-level proprietary vocabulary incentives can affect speech recognition and translation accuracy.

[0069] Context bias is also known as proprietary word excitation. Existing proprietary word excitation schemes that incorporate contextual information can be divided into two main categories: rule-based proprietary word excitation and neural network-based proprietary word excitation. These are described in detail below.

[0070] Rule-based proprietary word incentives reward each decoding path when it matches a word in the proprietary word list during decoding. This can intuitively increase the posterior probability of tokens associated with the proprietary word. This rule-based proprietary word incentive is essentially a shallow fusion (also known as post-fusion). Proprietary words in the bias list that were not included in the training process will not appear in the decoding candidate path. Using a bias list has limited performance improvements and is difficult to solve for rare characters.

[0071] Neural network-based proprietary vocabulary excitation introduces a separate bias module for modeling and integrating biased proprietary vocabulary information. Proprietary vocabulary excitation is implemented during model training and inference. Compared to rule-based proprietary vocabulary excitation, this is more flexible and can quickly adapt to different proprietary vocabulary bias lists.

[0072] The Cross Language Agent-Simultaneous Interpretation (CLASI) framework for end-to-end simultaneous interpretation incorporates neural network-based proprietary vocabulary enhancement into the classic end-to-end automatic speech recognition (ASR) model, LAS. The core idea is to simulate proprietary vocabulary by randomly sampling text phrases from text labels during training. Furthermore, an additional bias encoder module is added, and additional attention is introduced in the decoder to establish an attentive connection between the text latent state and the proprietary vocabulary embedding.

[0073] Early solutions for speech translation involved cascading ASR systems and machine translation (MT) systems, first transcribing speech into text and then translating the text into the target language. However, in consecutive transcription or translation steps, errors in the previous step will accumulate and amplify in subsequent steps, ultimately leading to a decline in translation quality. Compared to cascade systems, end-to-end speech translation can significantly reduce error accumulation and latency, and can have more context modeling and better generalization capabilities across languages. Large Language Models (LLMs) have powerful generation capabilities, and the latest end-to-end speech translation system, CLASI, uses an architecture that combines an audio encoder and LLM. The audio encoder converts the input speech into a continuous representation. The LLM receives the representation generated by the audio encoder and the retrieved relevant information and context, and generates the corresponding translation results.

[0074] However, the most advanced end-to-end speech translation system, taking CLASI as an example, solves the translation problem of professional vocabulary by building a corresponding professional domain terminology database. However, retrieving relevant information from an external professional domain terminology (proprietary vocabulary) database relies on the preliminary translation results of the speech translation system. For proprietary vocabulary outside the training set, the preliminary translation results of the speech translation system are often of poor quality, and it is difficult to retrieve relevant information by directly using it for external database retrieval. In addition, the construction cost of the professional domain terminology database is high and difficult to expand, making it difficult to add a bias list of proprietary vocabulary in daily use scenarios. Therefore, this solution applies contextual bias to the end-to-end speech translation system, using fixed proprietary vocabulary incentives during training and adding proprietary vocabulary incentives outside the training set as needed during inference.

[0075] Existing contextual biasing schemes, whether rule-based or neural network-based, all generate text-level lists of proprietary vocabulary biases. However, different user groups may have varying pronunciations and accents for the same vocabulary in different application scenarios, such as dialects. Text-level proprietary vocabulary excitation cannot meet the generalization requirements for audio signals with varying pronunciations and accents. To ensure that end-to-end speech translation systems can effectively excite proprietary vocabulary with varying pronunciations and accents, this solution constructs a phonetic representation database for proprietary vocabulary and employs phoneme-level proprietary vocabulary excitation to mitigate the interference of pronunciation and accent on speech translation systems.

[0076] See also Figure 1 , Figure 1 It is a flowchart of the first embodiment of the speech translation method of the present application. It should be noted that if there is substantially the same result, the method of the present application is not based on Figure 1 The process sequence shown is limited. Figure 1 As shown, the speech translation method includes:

[0077] S110, obtaining an audio clip;

[0078] Specifically, the audio clip can be a complete input audio or a segment of the input audio. The language of the audio clip can be Chinese, English, French, etc., and can also be a local dialect.

[0079] S120: extracting representations of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and performing phoneme classification on each of the initial high-order speech representations to determine an initial phoneme set of the audio segment;

[0080] Specifically, representation extraction can convert an audio clip into a feature vector, capturing the acoustic features of the audio. Initial high-level speech representations can be multiple high-level speech representations corresponding to the audio clip before matching the target proper vocabulary in the audio clip. These initial high-level speech representations can be used to reflect the semantic information of the audio clip. The initial phoneme set can be multiple phonemes corresponding to the audio clip. Phonemes can be the smallest speech units in a language, such as vowels and consonants.

[0081] S130, using each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment;

[0082] The proprietary vocabulary offset list can be used to construct a target database to stimulate proprietary vocabulary in the audio clip. The target database can be a database including high-order speech representations of proprietary vocabulary. The high-order speech representations in the target database can include a proprietary vocabulary phoneme set and a translation text of the proprietary vocabulary, which can be used to match the phonemes in the initial phoneme set of the audio clip to identify whether the audio clip contains proprietary vocabulary, such as terms, names, or personal names in a specific field or application. The at least one target proprietary vocabulary can be one or more proprietary words in the audio clip.

[0083] Specifically, in order to ensure that proprietary words in audio clips can be correctly identified and processed, thereby improving the generalization effect of speech translation, the high-order speech representations of each proprietary word in the target database can be matched according to the high-order speech representations and phoneme categories in the audio clips, and then the proprietary words contained in the audio clips can be retrieved and matched. In this way, it is possible to identify whether specific proprietary words are mentioned in the audio clips, thereby realizing phoneme-level proprietary word excitation.

[0084] S140, replacing the initial high-order speech representation of the target proper vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment;

[0085] Among them, the high-order speech representation set includes the initial high-order speech representation corresponding to the common words in the audio clip and the target high-order speech representation corresponding to the proprietary words, which can fully represent the speech content in the audio clip.

[0086] Specifically, the initial high-level representation of the proprietary words in the audio clip is replaced with the corresponding high-level speech representation of this proprietary word in the target database, thereby ensuring that the high-level speech representation set corresponding to the newly formed audio clip can accurately represent these proprietary words.

[0087] S150: Translate the high-order speech representation set to obtain a target translation text of the audio segment.

[0088] The target translation text may be a translation result of the audio segment.

[0089] Specifically, a set of high-order speech representations can be input into a language model for translation to generate a text representation of the audio clip. That is, the output result can at least include the target translated text, so that the audio clip can be understood and presented in written form while ensuring the consistency of proprietary vocabulary in the translation, thereby improving the generalization of speech translation.

[0090] In addition, users can customize the output results through operation instructions. The output results can include not only the target translation text, but also the target source text and / or the location of the target vocabulary in the target translation text. The target source text can be the content of the audio clip converted into written text.

[0091] In an exemplary embodiment, a representation extraction is performed on the audio clip "You must bite the bullet" to obtain multiple initial high-order speech representations. These initial high-order speech representations are then subjected to phoneme classification to obtain an initial phoneme set for the audio clip, "y u...b ai t ze...te." Assuming that the high-order speech representation of a proprietary word in the target database is (bite the bullet, b-ai-ze-ze-...-te, bite your teeth), the proprietary word phoneme set "b-ai-ze-ze-te" corresponds to "b aize...te" in the initial phoneme set "y u...b ai ze...te." Therefore, the presence of the proprietary word in the audio clip is determined, and the corresponding initial high-order speech features in the audio clip are replaced with the target high-order speech representation of the proprietary word to form an updated high-order speech representation set for the audio clip. The target high-order speech representation may include: then, translating the high-order speech representation set to obtain the target source text "You must bite the bullet" of the audio clip "You must bite the bullet", the target translated text "You must bite the bullet", and the position of the target proper noun "bite the bullet" in the target translated text "### bite the bullet". It should be noted that if the audio clip "You must bite the bullet" is not subjected to the proper noun excitation of this embodiment, the translated text is directly "You must bite the bullet".

[0092] This embodiment can alleviate the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks by using phoneme-level proprietary vocabulary excitation, greatly enhancing the generalization capability of the end-to-end speech translation system.

[0093] Furthermore, this embodiment takes into account that end-to-end speech translation is a downstream task of automatic speech recognition (ASR), and its performance is greatly affected by speech recognition. In many scenarios, the speech to be recognized may contain phrases (proper vocabulary, biasphrases) that are rarely or completely absent in the training data, such as proper nouns, names, and place names in certain fields. The accuracy of these phrase recognition and translation has a great impact on the user experience. However, since the speech context involves proprietary vocabulary phrases that change with the user's environment, identifying the translation context is challenging for end-to-end speech translation systems.

[0094] However, taking the end-to-end simultaneous interpretation system (Cross Language Agent-Simultaneous Interpretation, CLASI) as an example, the translation problem of professional vocabulary is solved by building a corresponding professional domain terminology database. However, retrieving relevant information from an external professional domain terminology (proprietary vocabulary) database relies on the preliminary translation results of the speech translation system. For proprietary vocabulary outside the training set, the preliminary translation results of the speech translation system are often of poor quality, and it is difficult to retrieve relevant information directly through external database retrieval. In addition, the construction cost of the professional domain terminology database is high and difficult to expand, making it difficult to add a bias list of proprietary vocabulary in daily use scenarios.

[0095] Therefore, in order to ensure that the end-to-end speech translation system can significantly excite proprietary vocabulary with different pronunciations and accents, the embodiment of the present application proposes an end-to-end speech translation method with phoneme-level proprietary vocabulary excitation, constructs a speech representation database of proprietary vocabulary, and adopts phoneme-level proprietary vocabulary excitation to alleviate the interference of pronunciation and accent on the speech translation system.

[0096] See also Figure 2 , Figure 2 1 is a flow chart of a second embodiment of the speech translation method of the present application. In this embodiment, the method includes the following steps:

[0097] S210, obtaining an audio clip;

[0098] The above-mentioned S110 may be repeated here.

[0099] S220, extracting an initial high-order speech representation of the audio segment through a target speech encoder, and downsampling the initial high-order speech representation through an adapter;

[0100] Specifically, the target speech encoder can be a trained audio encoder, which can be used to extract high-order speech representations from the audio clip. The adapter can downsample the high-order speech representations of the audio clip, preserving key information while reducing data complexity. This is then adapted to the dimensionality of the large language model input layer through linear projection, enabling the large language model to process this information more efficiently. The adapter typically uses linear projection to achieve this dimensionality reduction.

[0101] S230, performing phoneme classification on each of the downsampled initial high-order speech representations according to a preset phoneme table using a classifier to determine an initial phoneme set of the audio segment;

[0102] Specifically, a phoneme table can be constructed based on prior knowledge. The phoneme table includes multiple minimum units of speech, such as vowels and consonants. By determining the initial phoneme set of the audio segment based on the phoneme-level phoneme table, speech data can be processed in detail at the phoneme level.

[0103] S240, using the proper vocabulary phoneme set of each proper vocabulary in the target database to match the phonemes in the initial phoneme set;

[0104] Specifically, in an end-to-end speech translation system, some proprietary words rarely appear in general training data, leading to inaccurate translation results. Therefore, this embodiment constructs a target database using a proprietary word offset list, providing an incentive mechanism for these proprietary words. The proprietary word phoneme set of each proprietary word in the target database is used to match the phonemes in the initial phoneme set of the audio clip.

[0105] S251, in response to at least one of the proprietary vocabulary phoneme sets in the target database matching the phonemes in the initial phoneme set, determining the target proprietary vocabulary as the target proprietary vocabulary, and using the high-order speech representation of the target proprietary vocabulary as the target high-order speech representation;

[0106] Specifically, when the proprietary phoneme set of each proprietary word in the target database successfully matches a certain segment of phonemes in the initial phoneme set of the audio clip, the target high-order speech features of the target proprietary word can be used to replace the inaccurate initial high-order speech representation in the audio clip.

[0107] S252: In response to each of the proprietary phoneme sets in the target database not matching the phonemes in the initial phoneme set, using a preset unbiased symbol as the target high-order speech representation.

[0108] Unbiased symbols are used as a default choice when proprietary and / or unknown vocabulary is absent. Specifically, when the proprietary phoneme set of each proprietary vocabulary in the target database fails to match a certain segment of phonemes in the initial phoneme set of the audio clip, the default unbiased symbol can be used as a replacement. This mechanism ensures that even when the system encounters unknown vocabulary, it can still generate a reasonable high-level representation, avoiding system crashes or translation failures due to the absence of proprietary and / or unknown vocabulary or the inability to process unknown vocabulary.

[0109] S260: replacing the initial high-order speech representation of the target proper vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment;

[0110] Specifically, because traditional models in end-to-end speech translation rely on an intermediate speech-to-text step, it's difficult to effectively convert speech signals into features useful for downstream translation tasks. Therefore, this embodiment extracts deep, initial high-order speech features from audio clips. When proprietary vocabulary is detected in the audio clip, the target high-order speech features corresponding to the proprietary vocabulary replace the corresponding initial high-order speech features. This allows direct translation without the need for an intermediate text representation.

[0111] S270: Input the high-order speech representation set into a target language model for autoregressive generation to obtain a target translation text of the audio segment.

[0112] Specifically, the large speech model only uses the decoder. Because the decoder uses an autoregressive generation method, that is, it gradually generates new outputs based on previous inputs, generating one word or symbol at a time, and then generating the next word based on the previous context. Therefore, this embodiment can input the high-order speech representation set corresponding to the audio clip into the large speech model for translation; it can also input the initial high-order speech features of the audio clip and the updated target high-order speech features into the large speech model in sentence order for translation.

[0113] The user can customize the output results through operation instructions. The output results of the target language model can include not only the target translation text, but also the target source text and / or the location of the target proprietary vocabulary in the target translation text. The target source text can be the content of the audio clip converted into written text.

[0114] In an exemplary embodiment, see Figure 3First, the input audio clip is divided into 25ms audio sequences and input into the speech encoder and adapter to obtain initial high-order speech representations; then, a classifier is used to classify these initial high-order speech representations to obtain the phoneme categories in the phoneme table corresponding to each initial high-order speech representation, forming an initial phoneme set. By searching and comparing whether the audio clip contains proprietary vocabulary in the target database, if the phonemes in a proprietary vocabulary phoneme set in the target database appear continuously in the initial phoneme set, it means that the proprietary vocabulary appears in the audio clip. Therefore, the target high-order speech representation corresponding to the target proprietary vocabulary can be used to replace the corresponding initial high-order speech representation in the audio clip. Then, the initial high-order speech representation that has not been replaced and the updated target high-order speech representation are input into the large language model for translation, and the target source text, target translation text and the position of the target proprietary vocabulary in the target translation text are output. Among them, each proprietary vocabulary phoneme set in the target database can be pre-input into the bias encoder through the proprietary vocabulary bias list and subjected to representation extraction and adapter to obtain multiple target high-order speech representations corresponding to each proprietary vocabulary, and then these target high-order speech representations are subjected to phoneme classification to obtain the proprietary vocabulary phoneme set corresponding to each proprietary vocabulary.

[0115] This embodiment, by using phoneme-level proprietary vocabulary excitation, can alleviate the generalization problems caused by pronunciation and accent differences in different application scenarios in speech translation tasks, greatly enhance the generalization capabilities of the end-to-end speech translation system, and build a proprietary vocabulary speech representation database to meet the proprietary vocabulary customization needs of different user groups when using the speech translation system.

[0116] See also Figure 4 , Figure 4 : This is a flow chart of the third embodiment of the speech translation method of the present application. Based on the above-mentioned first embodiment, the target database construction process in this embodiment can be implemented before S130, for example, it can be implemented simultaneously with S110 or S120, or it can be implemented before S110; based on the above-mentioned second embodiment, the target database construction process can be implemented before S240, for example, it can be implemented simultaneously with S210 or S220 or S230, or it can be implemented before S210. In this embodiment, the target database construction process can specifically include:

[0117] S410, obtaining the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of proprietary vocabulary triples;

[0118] Triples can be used to construct a comprehensive representation of proprietary vocabulary, including the source text, the phoneme set, and the translated text. By combining triplets from different dimensions of proprietary vocabulary, the translation system can balance phonetic features and semantic information when handling complex speech translation tasks, thereby improving the processing of proprietary vocabulary.

[0119] In an exemplary embodiment, the list of proprietary word offsets may be as follows: (bite the bullet, b-ai-ze-...-te, bite the bullet); (break the ice, bu-rei-k-...-ss, break the deadlock); (hit the nail, he-ei-t-...-ll, hit the nail on the head); (burn the oil, bo-e-rn-...-yao, burn the oil); (between a rock, b-ei-tu-...-k, be caught between a rock and a hard place); (let the cat out, le-t-ze-...-t, let the cat out); (jump on the wagon, zh-ang-m-...-en, go with the flow); (under the weather, ang-de-r-...-er, feel unwell); (piece of cake, pi-s-ev-...-k, a simple thing); (cost a leg, kao-st-...-g, cost a leg).

[0120] S420, inputting each triple into a target bias encoder for representation extraction to obtain a high-order speech representation of each proprietary word, and downsampling the high-order speech representation;

[0121] Specifically, the bias encoder is used to extract higher-order speech representations for proprietary vocabulary audio. Each triplet of proprietary vocabulary in the proprietary bias list is input into the target bias encoder to extract the higher-order speech representations for the proprietary vocabulary. An adapter can also be used to downsample the higher-order speech representations so that the higher-order speech representations output by the bias encoder can also be adapted to the target language model.

[0122] S430, performing phoneme classification on the downsampled high-order speech representation of each of the proprietary words according to the phoneme table to obtain a proprietary vocabulary phoneme set of each of the proprietary words;

[0123] Specifically, a classifier can be used to classify the downsampled high-level speech representations into phoneme-based categories based on a phoneme table. By dividing the high-level speech representations of proprietary vocabulary into smaller phoneme units, a proprietary vocabulary phoneme set is obtained, which can more accurately describe the pronunciation characteristics of these vocabulary words.

[0124] S440: constructing the target database based on the high-order phonetic representation of each of the proprietary words and the proprietary word phoneme set.

[0125] Specifically, based on the high-level speech representations and phoneme sets of each proprietary vocabulary, a target database is constructed. This database records the high-level speech representations of the proprietary vocabulary and their corresponding phoneme representations. This allows for rapid reference and matching of these proprietary vocabulary in speech translation tasks.

[0126] In this embodiment, a target database of proprietary vocabulary speech representation is constructed by using a phoneme-level proprietary vocabulary offset list to achieve the use of phoneme-level proprietary vocabulary excitation and alleviate the generalization problem caused by pronunciation and accent in speech translation tasks.

[0127] See also Figure 5 , Figure 5 1 is a flow chart of the fourth embodiment of the speech translation method of the present application. In this embodiment, the steps of constructing the proprietary vocabulary offset list may include:

[0128] S510, obtaining a translation dataset including multiple original text labels;

[0129] Specifically, the translation dataset may include at least sample audio, original text labels corresponding to the sample audio, and translated text.

[0130] S520, performing word segmentation and part-of-speech tagging on each of the original text labels to obtain a corresponding noun sequence;

[0131] Specifically, nouns play an important role in proprietary vocabulary, especially in domain terms, place names, and personal names. Therefore, filtering out noun sequences can help the system focus on words most likely to belong to proprietary vocabulary, reducing unnecessary interference. Therefore, the system first performs part-of-speech tagging on the word sequences in the text and filters out all noun sequences. Part-of-speech tagging can be used to assign a part-of-speech label to a given word sequence.

[0132] S530, sampling from the noun sequence to input into the target language model for speech synthesis, to obtain a set of proprietary vocabulary phonemes corresponding to each of the proprietary vocabulary;

[0133] Specifically, we randomly sample indefinite-length nouns from all noun sequences, and we also sample low-frequency indefinite-length nouns from all noun sequences. Randomly sampling indefinite-length nouns yields noun samples of varying lengths and complexities, helping to improve generalization across a wide range of proper nouns. The randomness prevents overfitting to specific noun structures. Low-frequency nouns are more likely to be proper nouns, so sampling them better covers those proper nouns.

[0134] Then, the text to speech (TTS) function of the large model can be used to generate audio of the sampled proprietary vocabulary, and the audio can be cut into 25ms frames to obtain the proprietary vocabulary phoneme set corresponding to the proprietary vocabulary.

[0135] S540, obtaining the source text and the translated text of each of the proprietary words from the translation dataset;

[0136] Specifically, the corresponding source text and translation text of the sampled proprietary vocabulary are found in the original data set, so that the proprietary vocabulary corresponds to its corresponding source text and translation text.

[0137] S550 : Combining the source text, the phoneme set, and the translation text of each of the proprietary words to obtain the proprietary word offset list.

[0138] Specifically, the final speech translation proprietary vocabulary bias list consists of several triplets (proper vocabulary source text, proprietary vocabulary phoneme set, proprietary vocabulary translation text). At the same time, bias lists from open-source corpora such as Wikipedia and LibriSpeech can also be incorporated into the proprietary vocabulary bias list.

[0139] In addition, you can add <no-bias>The character is used as the default choice when there is no proprietary vocabulary.

[0140] In an exemplary embodiment, the sample audio is as follows: A mountain is not famous for its height, but for the immortals who inhabit it. A river is not magical for its depth, but for the dragons who inhabit it. This is a humble room, yet my virtue is fragrant. Moss grows green on the steps, and the grass is green through the curtains. Great scholars chat and laugh among us, and there are no unlearned people among us. I can tune my zither and read the Golden Sutra. No music to disturb my ears, no paperwork to tire my body. Zhuge Liang's Hut in Nanyang, Ziyun Pavilion in Western Shu. Confucius said, "What is humble about this?"

[0141] If part-of-speech tagging is not performed, random sampling from the label text will yield the following random proprietary words: Gao Youxian, Shi Loushi, Jie Lucaose, An Duzhi, Confucius, <no-bias>Among them, Gao Youxian, Jie Lucaose, and An Duzhi are all unreasonable random proprietary words.

[0142] If only random sampling of nouns is performed after part-of-speech tagging, we will get: humble room, moss marks, great scholar, commoner, plain zither, golden scripture, Ziyun Pavilion, Confucius, <no-bias>.in, <no-bias>Unbiased label.

[0143] This embodiment extracts and samples proprietary vocabulary from natural language processing and generates speech representations using speech synthesis technology to construct a proprietary vocabulary offset list. This list contains the original text of each proper noun, the audio phoneme set of the proper vocabulary, and the translation of the proper vocabulary. This allows the speech translation system to better recognize and translate proprietary vocabulary, improving performance when processing domain-specific or low-frequency vocabulary.

[0144] See also Figure 6 , Figure 6 1 is a flow chart of the fifth embodiment of the speech translation method of the present application. In this embodiment, the training process of the target speech encoder, the target bias encoder and the target language model includes:

[0145] S610, based on the translation data set, training the first speech encoder to obtain a second speech encoder; and training the first bias encoder to obtain a second bias encoder;

[0146] The translation data set may include each sample audio, the original text label corresponding to each sample audio, the sample audio phoneme set, the translation label and the translation-specific vocabulary position.

[0147] Specifically, in the first stage, the first speech encoder and the first bias encoder are trained separately to obtain a trained second speech encoder and a trained second bias encoder. This enables the second speech encoder to have the ability to extract representations and classify phonemes for the input audio clips while ensuring accurate capture of different speech signals. At the same time, the second bias encoder has the ability to extract representations and classify phonemes for the proprietary vocabulary in the input proprietary vocabulary bias list while ensuring that the proprietary vocabulary is appropriately weighted in the translation task.

[0148] In addition, in the initial individual training, there is less interdependence between encoders, which can effectively simplify the training process and enable each encoder to reach an ideal state on a single task.

[0149] Furthermore, the translation dataset may include multiple sample audios and corresponding phoneme categories, and the training process of the second speech encoder may specifically include:

[0150] The sample audio is segmented and labeled according to the phoneme category to obtain a first true phoneme label; the sample audio is input into the first speech encoder for representation extraction, and the phoneme classification is performed using the classifier to obtain a first predicted phoneme label; a first phoneme classification loss is calculated for the first true phoneme label and the first predicted phoneme label, and the first speech encoder is trained using the first phoneme classification loss; this cycle is repeated until the first phoneme classification loss reaches convergence, and the training is terminated to obtain the second speech encoder;

[0151] Specifically, the first speech encoder is trained using the translation dataset, the sample audio in the translation dataset is segmented, and the segmented sample audio segments are labeled with phonemes according to the phoneme table to obtain a first true phoneme label; the classifier performs phoneme classification on the high-order speech representation (embedding) output by the first speech encoder to obtain a first predicted phoneme label. Then, the first phoneme classification loss L is calculated using the cross-entropy loss (CE) between the first predicted phoneme label and the first true phoneme label. AE The first phoneme classification loss is back-propagated to update the first speech encoder. Then, the sample audio is segmented and labeled according to the phoneme category to obtain the first true phoneme label, and the following steps are repeated until the first factor classification loss reaches convergence, resulting in a trained second speech encoder.

[0152] Furthermore, the training process of the second bias encoder may specifically include:

[0153] The proprietary vocabulary audio in the sample audio is segmented and labeled according to the phoneme category to obtain a second true phoneme label; the proprietary vocabulary audio is input into the first bias encoder for representation extraction, and the phoneme classification is performed using the classifier to obtain a second predicted phoneme label; the second phoneme classification loss of the second true phoneme label and the second predicted phoneme label is calculated, and the first bias encoder is trained using the second phoneme classification loss; this cycle is repeated until the second phoneme classification loss reaches convergence and the training is stopped to obtain the second bias encoder.

[0154] Specifically, the first bias encoder is trained using the translation dataset, the proprietary vocabulary audio in the translation dataset is segmented, and the segmented proprietary vocabulary audio is labeled with phonemes according to the phoneme table to obtain a second true phoneme label; the classifier performs phoneme classification on the high-order speech representation output by the first bias encoder to obtain a second predicted phoneme label. Then, the second phoneme classification loss L is calculated using the cross entropy loss to compare the second predicted phoneme label with the second true phoneme label. BE The second phoneme classification loss is back-propagated to update the first bias encoder. Then, the proper vocabulary audio in the sample audio is segmented and labeled according to the phoneme category, and the subsequent steps are repeated until the second factor classification loss reaches convergence, resulting in a trained second bias encoder.

[0155] S620, training the first language model based on the proprietary vocabulary offset list to obtain a second language model;

[0156] The first language model may be a large language model before fine-tuning training, and the second language model may be a large language model after fine-tuning training.

[0157] Specifically, because proprietary vocabulary bias lists appear less frequently in the overall translation dataset, in order to enhance the large language model's proprietary translation capabilities, the first language model needs to be fine-tuned using the proprietary vocabulary bias list before collaborative training to obtain a second language model. This ensures that during the inference process, the second language model can correctly translate the target high-order speech representation output by the target bias encoder.

[0158] S630: Based on the translation dataset, collaboratively train the second speech encoder, the second bias encoder, and the second language model to obtain the target speech encoder, the target bias encoder, and the target language model.

[0159] Specifically, in the third stage, the second speech encoder, second bias encoder, and second language model are collaboratively trained. Information is shared and transferred between these models to ensure they can work together consistently and efficiently when processing tasks. After collaborative training, the target speech encoder, target bias encoder, and target language model are obtained. This allows the target speech encoder and target bias encoder to retain the ability to extract high-order implicit expressions and classifications from audio, while also training the target language model's ability to translate the entire translation dataset and its ability to perceive the location of proprietary vocabulary in the translation.

[0160] Furthermore, S630 may specifically include:

[0161] The translation dataset also includes original text labels, translated text labels, and translated text proprietary vocabulary positions;

[0162] The step of collaboratively training the second speech encoder, the second bias encoder, and the second language model based on the translation dataset to obtain the target speech encoder, the target bias encoder, and the target language model includes:

[0163] Inputting the training data into the second speech encoder for processing, and inputting the proper vocabulary offset list into the second offset encoder for processing, so as to output corresponding training original text, training translation, and proper noun positions of the training translation through the second language model;

[0164] Obtaining translation loss of the training translation text and the translation text label, language recognition loss of the second language model, and position loss of the training translation text and the translation text proper vocabulary position;

[0165] Calculating a collaborative overall loss by combining the translation loss, the language identification loss, the position loss, the first phoneme classification loss, and the second phoneme classification loss, and using the collaborative overall loss to train the second speech encoder, the second bias encoder, and the second language model;

[0166] This cycle is repeated until the collaborative overall loss reaches convergence, and the collaborative training is stopped to obtain the target speech encoder, the target bias encoder, and the target language model.

[0167] Specifically, the second speech encoder, the second bias encoder and the second language model are trained collaboratively: using L Trans Represents the loss between the training translation output by the second language model and the actual translation label, L ASR Represents the speech recognition loss of the second language model, L Pos The position loss of the position of the specific word in the translation. At the same time, the first phoneme classification loss L of the second speech encoder and the second bias encoder AE and the second phoneme classification loss L BE Also participate in the training, the collaborative overall loss L can be expressed as: L = L Trans +L AE +L BE +L Pos +L ASR ;

[0168] Among them, the cross entropy loss is used to calculate the loss L of the position of the proprietary word in the perceived translation Pos .

[0169] In an exemplary embodiment, the sample audio is "What a genius Einsten was?", the original text label is "What a genius Einsten was?", the translated text label is "Einstein is really a genius", the translated text proper word position is "Einstein#####", the sample audio phoneme set is "hh weins ta n zz", the proper word offset list is (Einsten, eins-ta-n, Einstein), the second language model outputs "What a genius Einsten was?", the output training translation text is "Einstein is really a genius", and the translated text proper noun position is "Einstein#####".

[0170] In this embodiment, first, through individual training, each model is optimized for its respective task, ensuring that they can handle basic tasks. Then, the large language model is fine-tuned through a proprietary vocabulary bias list to make it more suitable for processing proprietary vocabulary and low-frequency vocabulary in specific fields, thereby improving translation accuracy. Finally, through collaborative training, information sharing and coordination between the various models are achieved, which improves the system's performance in handling complex scenarios and ensures that the speech encoder, bias encoder, and language model work closely together to achieve higher-quality speech translation effects.

[0171] See also Figure 7 , Figure 7 This is an exemplary structural block diagram of an electronic device for the speech translation method of this application. Figure 7 As shown, the electronic device 700 of the present application may include a processor 701 and a memory 702, wherein the processor 701 and the memory 702 communicate with each other via a bus. The memory 702 stores program instructions for speech translation. When the program instructions are executed by the processor 701, the processor performs the above-mentioned related method steps to implement a speech translation method in the above-mentioned embodiment.

[0172] See also Figure 8 , Figure 8 This is an exemplary structural block diagram of the computer-readable storage medium of the speech translation method of the present application. Figure 8 As shown, the computer-readable storage medium 800 stores a computer program 801. When the computer program 801 is executed by a processor on a computer, the computer executes the above-mentioned related method steps to implement a speech translation method in the above-mentioned embodiment.

[0173] The above scheme obtains an audio clip; extracts representations from the audio clip to obtain multiple initial high-order speech representations of the audio clip, and performs phoneme classification on each of the initial high-order speech representations to determine the initial phoneme set of the audio clip; uses each proprietary vocabulary phoneme set in the target database constructed based on the proprietary vocabulary offset list to match the phonemes in the initial phoneme set to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio clip; replaces the initial high-order speech representation of the target proprietary vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio clip; and translates the high-order speech representation set to obtain a target translation text of the audio clip. Using phoneme-level proprietary vocabulary excitation can alleviate the generalization problem caused by pronunciation and accent differences in different application scenarios in speech translation tasks, greatly enhancing the generalization capability of the end-to-end speech translation system.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed methods, electronic devices and storage media can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0175] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0176] In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the speech translation method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0178] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech translation method, characterized in that: The speech translation method comprises: Get the audio clip; Extracting representations of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and performing phoneme classification on each of the initial high-order speech representations to determine an initial phoneme set of the audio segment; Matching the phonemes in the initial phoneme set with the phoneme sets of the target vocabulary in the target database constructed based on the proprietary vocabulary offset list to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment; Replacing the initial high-order speech representation of the target proper vocabulary with the target high-order speech representation to form a high-order speech representation set corresponding to the audio segment; Translating the high-order speech representation set to obtain a target translated text of the audio segment; Before obtaining the proprietary vocabulary offset list, the following steps are included: Obtain a translation dataset containing multiple source text labels; Perform word segmentation and part-of-speech tagging on each of the original text labels to obtain a corresponding noun sequence; Sampling the noun sequence and inputting the sampling into the target language model for speech synthesis to obtain a set of phonemes corresponding to each of the specific words; Acquire the source text and the translated text of each of the proprietary words from the translation dataset; The source text, the special vocabulary phoneme set and the translation text of each special vocabulary are respectively combined to obtain the special vocabulary offset list.

2. The speech translation method according to claim 1, wherein: The extracting the representation of the audio segment to obtain a plurality of initial high-order speech representations of the audio segment, and performing phoneme classification on each of the initial high-order speech representations to determine an initial phoneme set of the audio segment includes: extracting an initial high-order speech representation of the audio segment through a target speech encoder, and downsampling the initial high-order speech representation through an adapter; Performing phoneme classification on each of the downsampled initial high-order speech representations according to a preset phoneme table using a classifier to determine an initial phoneme set of the audio segment; The translating the high-order speech representation set to obtain a target translated text of the audio segment includes: The high-order speech representation set is input into the target language model for autoregressive generation to obtain a target translation text of the audio segment.

3. The speech translation method according to claim 2, wherein: Before obtaining a target high-order speech representation of at least one target proprietary vocabulary in the audio segment by matching the phonemes in the initial phoneme set with the phonemes in the target database constructed based on the proprietary vocabulary offset list, the method includes: Acquire the proprietary vocabulary offset list; wherein the proprietary vocabulary offset list includes a plurality of proprietary vocabulary triples; Inputting each of the triples into a target bias encoder for representation extraction to obtain a high-order speech representation of each of the proprietary words, and downsampling the high-order speech representation; According to the phoneme table, performing phoneme classification on the downsampled high-order speech representation of each of the proprietary words to obtain a proprietary vocabulary phoneme set of each of the proprietary words; The target database is constructed based on the high-order speech representation of each of the proprietary words and the proprietary word phoneme set.

4. The speech translation method according to claim 3, wherein: The step of inputting each triple into a target bias encoder for representation extraction to obtain a high-order speech representation of each proprietary word, and before downsampling the high-order speech representation, includes: Obtaining the translation dataset; Based on the translation data set, training the first speech encoder to obtain a second speech encoder; and training the first bias encoder to obtain a second bias encoder; Training the first language model based on the proprietary vocabulary offset list to obtain a second language model; Based on the translation data set, the second speech encoder, the second bias encoder, and the second language model are collaboratively trained to obtain the target speech encoder, the target bias encoder, and the target language model.

5. The speech translation method according to claim 4, wherein: The translation dataset includes a plurality of sample audios and corresponding phoneme categories; and the training of the first speech encoder based on the translation dataset to obtain the second speech encoder includes: Segmenting the sample audio and labeling it according to the phoneme category to obtain a first true phoneme label; Inputting the sample audio into the first speech encoder for characterization extraction, and performing phoneme classification using the classifier to obtain a first predicted phoneme label; Calculating a first phoneme classification loss of the first true phoneme label and the first predicted phoneme label, and training the first speech encoder using the first phoneme classification loss; Repeat this cycle until the first phoneme classification loss reaches convergence, stopping the training to obtain the second speech encoder; The training of the first bias encoder to obtain the second bias encoder includes: Segmenting the proper vocabulary audio in the sample audio and labeling it according to the phoneme category to obtain a second true phoneme label; Inputting the proprietary vocabulary audio into the first bias encoder for representation extraction, and performing phoneme classification using the classifier to obtain a second predicted phoneme label; Calculating a second phoneme classification loss for the second true phoneme label and the second predicted phoneme label, and training the first bias encoder using the second phoneme classification loss; This cycle is repeated until the second phoneme classification loss reaches convergence, and the training is stopped to obtain the second bias encoder.

6. The speech translation method according to claim 5, characterized in that: The translation dataset also includes original text labels, translated text labels, and translated text proprietary vocabulary positions; The step of collaboratively training the second speech encoder, the second bias encoder, and the second language model based on the translation dataset to obtain the target speech encoder, the target bias encoder, and the target language model includes: Inputting the training data into the second speech encoder for processing, and inputting the proper vocabulary offset list into the second offset encoder for processing, so as to output corresponding training original text, training translation, and proper noun positions of the training translation through the second language model; Obtaining a translation loss between a training translation text and a translation text label, a language recognition loss of the second language model, and a position loss between the training translation text and a position of a specific vocabulary in the translation text; Calculating a collaborative overall loss by combining the translation loss, the language identification loss, the position loss, the first phoneme classification loss, and the second phoneme classification loss, and using the collaborative overall loss to train the second speech encoder, the second bias encoder, and the second language model; This cycle is repeated until the collaborative overall loss reaches convergence, and the collaborative training is stopped to obtain the target speech encoder, the target bias encoder, and the target language model.

7. The speech translation method according to claim 1, wherein: The method of matching the phonemes in the initial phoneme set with the phoneme sets of the target vocabulary in the target database constructed based on the proprietary vocabulary offset list to obtain a target high-order speech representation of at least one target proprietary vocabulary in the audio segment includes: Matching the phonemes in the initial phoneme set using the proprietary phoneme set of each proprietary word in the target database; In response to at least one of the proprietary vocabulary phoneme sets in the target database matching the phonemes in the initial phoneme set, determining the target proprietary vocabulary as the target proprietary vocabulary, and using the high-order speech representation of the target proprietary vocabulary as the target high-order speech representation; In response to each of the proprietary phoneme sets in the target database not matching the phonemes in the initial phoneme set, a preset unbiased symbol is used as the target high-order speech representation.

8. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable program code; A processor is configured to call and run the executable program code from the memory, so that the electronic device executes the speech translation method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech translation method according to any one of claims 1 to 7 is implemented.

Images