Speech translation method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses a speech translation method and device, electronic equipment and a computer readable storage medium. The speech translation method comprises the following steps: acquiring a first speech coding vector of speech to be translated; performing real-time smoothing processing on the first voice coding vector to obtain marking information and segmentation information corresponding to the first voice coding vector; generating a smooth second voice coding vector with key point information based on the first voice coding vector, the mark information, the segmentation information and at least one cue word; decoding the second voice coding vector to obtain target voice; wherein the to-be-translated speech is different from the target speech in corresponding languages. According to the method, the source voice can be translated based on the key point information, so that the translation fluency and accuracy are improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of machine translation, and in particular to a speech translation method and device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Simultaneous interpretation refers to a translation method that continuously translates speech content to the audience without interrupting the speaker. Simultaneous interpretation technology is designed to perform high-quality and low-latency translation from source language speech to target language speech, and is widely used in many scenarios.

[0003] In scenarios that require real-time and seamless communication, such as international conferences and cross-border business negotiations, high-accuracy and low-latency machine simultaneous interpretation is crucial. In related technologies, simultaneous interpretation systems are usually divided into cascade systems and end-to-end systems.

[0004] However, the above-mentioned simultaneous interpretation systems have many problems such as high latency or low translation quality, which results in low real-time and accuracy of voice translation, thus affecting user experience. Summary of the invention

[0005] The main technical problem solved by the present application is to provide a speech translation method and device, an electronic device and a computer-readable storage medium, which can solve the problems of low real-time and low accuracy of speech translation existing in related simultaneous interpretation technologies.

[0006] In order to solve the above technical problems, the first technical solution adopted in the present application is to provide a speech translation method, including: obtaining a first speech coding vector of the speech to be translated; performing real-time smooth processing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector; generating a second speech coding vector with key information and smoothness based on the first speech coding vector, marking information, segmentation information and at least one prompt word; decoding the second speech coding vector to obtain the target speech; wherein the speech to be translated and the target speech correspond to different languages.

[0007] Among them, the step of performing real-time smoothing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector includes: marking the semantically irrelevant feature vector in the first speech coding vector in real time to obtain marking information; and segmenting the first speech coding vector to obtain segmentation information.

[0008] The speech translation method is implemented by a real-time end-to-end simultaneous interpretation system, which includes an audio encoder, an event encoder, a large language model and a decoder; the step of obtaining a first speech coding vector of a speech to be translated includes: encoding the speech to be translated by an audio encoder in the simultaneous interpretation system to obtain a first speech coding vector; performing real-time smooth processing on the first speech coding vector to obtain tag information and segmentation information corresponding to the first speech coding vector, including: performing real-time smooth processing on the first speech coding vector by an event encoder to obtain tag information and segmentation information corresponding to the first speech coding vector; the step of generating a smooth second speech coding vector with key information based on the first speech coding vector, tag information, segmentation information and at least one prompt word includes: processing the first speech coding vector, tag information, segmentation information and prompt word by a large language model to obtain a second speech coding vector; the step of decoding the second speech coding vector to obtain a target speech includes: decoding the second speech coding vector by a decoder to obtain the target speech.

[0009] Among them, the large language model is obtained by fine-tuning the large language base model with speech data annotated with key information; the method for fine-tuning the large language base model with speech data annotated with key information includes: obtaining a training speech sample data set; wherein the training speech sample data set includes multiple training speech samples corresponding to multiple languages, and each training speech sample includes invalid information irrelevant to semantics; based on each training speech sample, a corresponding training speech coding vector is obtained; each training speech coding vector is processed smoothly in real time to obtain the marking information and segmentation information corresponding to the training speech coding vector; the training speech coding vector, marking information, segmentation information and prompt words corresponding to each training speech sample are input into the large language base model for fine-tuning training to obtain a trained large language model.

[0010] Among them, the training speech coding vector, tag information, segmentation information and prompt word corresponding to each training speech sample are input into the large language base model for fine-tuning training to obtain the trained large language model, including: adding a low-rank adaptation matrix next to the large language base model; initializing the low-rank adaptation matrix with a random Gaussian distribution to obtain a first matrix; initializing the low-rank adaptation matrix with a zero matrix to obtain a second matrix; training the first matrix and the second matrix through the training speech coding vector, tag information, segmentation information, prompt word and speech parameters corresponding to each training speech sample; multiplying the trained first matrix with the second matrix and merging the pre-trained parameters of the large language base model to obtain the fine-tuned model parameters; fine-tuning the large language base model based on the fine-tuned model parameters to obtain the large language model.

[0011] Among them, the event encoder is obtained by pre-training a test-time adaptive base model with large-scale simultaneous interpretation data, and then fully fine-tuning the pre-trained base model with real-time speech data and feedback information corresponding to different individuals.

[0012] Among them, the simultaneous interpretation system also includes an adapter. After encoding the to-be-translated speech through the audio encoder in the simultaneous interpretation system to obtain the first speech coding vector, the step includes: processing the first speech coding vector through the adapter to generate a first speech vector representation adapted to the large language model; processing the first speech coding vector, marking information, segmentation information and prompt words through the large language model to obtain the second speech coding vector, the step includes: processing the first speech vector representation, marking information, segmentation information and prompt words through the large language model to obtain the second speech coding vector.

[0013] In order to solve the above technical problems, the second technical solution adopted in the present application is to provide a speech translation device, including: a first speech coding vector acquisition module, used to acquire the first speech coding vector of the speech to be translated; a smooth processing module, used to perform real-time smooth processing on the first speech coding vector to obtain the marking information and segmentation information corresponding to the first speech coding vector; a second speech coding vector generation module, used to generate a second speech coding vector with key information and smoothness based on the first speech coding vector, marking information, segmentation information and at least one prompt word; a target speech generation module, used to decode the second speech coding vector to obtain a target speech; wherein the speech to be translated and the target speech correspond to different languages.

[0014] To solve the above technical problems, the third technical solution adopted in the present application is to provide an electronic device, comprising: a memory for storing program data, which implements the steps in the speech translation method as described in any one of the above items when the stored program data is executed; and a processor for executing program instructions stored in the memory to implement the steps in the speech translation method as described in any one of the above items.

[0015] In order to solve the above technical problems, the fourth technical solution adopted in the present application is to provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the speech translation method described in any one of the above items are implemented.

[0016] The beneficial effects of the present application are as follows: Different from the related art, the present application provides a speech translation method and device, an electronic device, and a computer-readable storage medium, which can summarize and accurately extract the key information in the source speech in real time by performing real-time smooth processing on the first speech coding vector corresponding to the speech to be translated to obtain marking information and segmentation information, so as to reduce translation delay, thereby improving the real-time and fluency of translation. Based on the first speech coding vector, marking information, segmentation information, and prompt words, a smooth second speech coding vector with key information is generated, and the second speech coding vector is decoded to obtain the target speech, which can translate the source speech based on the key information, thereby improving the accuracy of translation and then improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a principle block diagram of an implementation method of a simultaneous interpretation system of the present application;

[0019] Figure 2 yes Figure 1 An architectural diagram of an embodiment of an event encoder;

[0020] Figure 3 is a schematic diagram of training an event encoder;

[0021] Figure 4 yes Figure 1 The architecture diagram of the first implementation of the CUHK language model;

[0022] Figure 5 It is a flowchart of the first implementation method of the speech translation method of the present application;

[0023] Figure 6 It is a flowchart of the second implementation method of the speech translation method of the present application;

[0024] Figure 7 is a flow chart of an implementation method of fine-tuning a large language base model;

[0025] Figure 8 This is a schematic diagram of fine-tuning the pre-trained large language model based on LoRA in this application;

[0026] Fig. 9 It is a structural diagram of an implementation method of a speech translation device of the present application;

[0027] Fig.10 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0028] Fig.11 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms of "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless otherwise clearly indicated above, and "multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0031] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0032] It should be understood that the terms "include", "comprises" or any other variations used herein are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "includes..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0033] Among the related technologies, the biggest challenge currently facing machine simultaneous interpretation is still to achieve a balance between low latency and high translation quality.

[0034] Specifically, on the one hand, the delay is not low enough. For cascade systems, due to the processing delay between modules, it cannot be applied to scenarios with high real-time requirements for machine simultaneous interpretation. In order to maintain low latency, existing cascade systems usually use small models with poor performance, which poses a bottleneck when dealing with complex and changeable practical application scenarios. For end-to-end systems, the existing system translates all sentences "heard" by all machines, so there is a lot of translation redundancy, such as: word repetition caused by stuttering (for example, "this, this is") and modal particles (for example, "um, ah, then") are all irrelevant information to the meaning of the sentence and do not need to be translated; for example, when the speaker makes a mistake and then corrects it (for example, "this seems to be something that happened last year, but it is not something that happened in February this year"), the wrong part is irrelevant information to the meaning of the sentence and does not need to be translated.

[0035] On the other hand, the translated information is not accurate enough and lacks the ability to extract key information. From a user-centric perspective, these current simultaneous interpretation systems only convey less than 42% of effective information to the audience, which seriously affects the communication effect. In contrast, professional human interpreters usually provide more than 70% of the necessary information, and ideally provide 95% of the necessary information.

[0036] Based on the above situation, the present application provides a speech translation method and device, an electronic device and a computer-readable storage medium, which can solve the problems of low real-time and low accuracy of speech translation existing in related simultaneous interpretation technologies.

[0037] The present application first provides a simultaneous interpretation system.

[0038] Specifically, see Figure 1 , Figure 1 The schematic diagram of the simultaneous interpretation system of the present application is a block diagram of a principle of an embodiment of the simultaneous interpretation system of the present application. In the embodiment, the simultaneous interpretation system 100 includes an audio encoder 11, an event encoder 12, a large language model (LLM) 13 and a decoder 14 connected to each other.

[0039] In this embodiment, the simultaneous interpretation system 100 is a real-time end-to-end simultaneous interpretation system.

[0040] In this embodiment, the audio encoder 11 is used to extract relevant feature representations from the original audio signal corresponding to the speech to be translated, so as to convert the audio signal of the source speech into high-dimensional speech features. In some embodiments, the audio encoder 11 can be a pre-trained model such as Wav2Vec, HuBERT, etc.

[0041] In this embodiment, the event encoder 12 is used to capture and process key events in the speech stream, such as pauses, repetitions, division of semantic units, etc., to output event-related vector representations to the large language model 13.

[0042] Specifically, see Figure 2 , Figure 2 yes Figure 1 In this embodiment, the event encoder 12 uses a multi-layer Transformer as the core framework, including a self-attention layer (Self Attention) and a multi-layer perceptron (MLP, Multilayer Perception) and two detectors Detector1 and Detector2.

[0043] Among them, Detector1 is a speech smoothing detector, which is used to mark events according to event trigger mechanisms in the speech to be translated (such as pauses, wrong words, corrections, redundant information, etc.). For example, when the speaker uses hesitant words such as "um" and "uh", the event encoder 12 can mark this speech feature vector and pass it to LLM, so that LLM can further process or ignore these features according to the information, so that LLM can smooth the speech features in real time, thereby making the translation result more natural.

[0044] Among them, Detector2 is a speech segmentation event detector, which is used to perform semantic segmentation based on acoustic signals (such as speech amplitude, frequency, energy change, etc.) and combined with grammatical and syntactic information, thereby improving the accuracy and real-time performance of translation. In some embodiments, by combining a pre-trained language model, the event encoder can determine in real time whether a complete sentence, phrase or semantic unit appears in the speech stream. For example, when a complete syntactic structure is detected, the event encoder automatically outputs segmentation information to the LLM.

[0045] It can be understood that by implementing streaming speech segmentation through the event encoder 12, the simultaneous interpretation system 100 can more intelligently identify semantic boundaries in speech, thereby meeting the real-time requirements of the speech stream.

[0046] In some implementations, the event encoder 12 is obtained by pre-training a test-time adaptive base model using large-scale simultaneous interpretation data, and then fully fine-tuning the pre-trained base model using real-time speech data and feedback information corresponding to different individuals.

[0047] Specifically, see Figure 3 , Figure 3It is a schematic diagram of training the event encoder. In this embodiment, the base model based on test-time adaptation (TTA) is first pre-trained through large-scale simultaneous interpretation data, and then the pre-trained base model is fully fine-tuned during the user's use, and the pre-trained base model is continuously learned from the speech characteristics of new users during actual use, so that the system is more adapted to the user's voice feature distribution, thereby obtaining the event encoder 12. Specifically, there are differences in speaking habits between different users. The base model pre-trained on large-scale simultaneous interpretation data may not perform well during the use of each specific user. Therefore, it is necessary to continuously optimize the performance of the event encoder 12 during the simultaneous interpretation process. For the voice data input by the user in real time, the event encoder 12 first encodes it and promptly provides it to the feature vector required by the LLM, and then uses these real-time data and user feedback to asynchronously optimize the model to ensure that the model used by the user next time is a model that conforms to his language style.

[0048] It is understandable that the event encoder 12 can gradually learn the speaker's speech speed, pause habits and language style during user use, and make adaptive adjustments to provide more accurate event information to the LLM, thereby helping to improve the accuracy of translation. Furthermore, through the adaptive algorithm, the event encoder 12 is continuously optimized during user use, and a user-customized simultaneous interpretation model can be obtained, thereby fully meeting the needs of the user.

[0049] In this embodiment, the large language model 13 is used to learn and capture key information in the speech segment, and convert the audio vector of the speech to be translated into the audio vector of the target language.

[0050] Specifically, see Figure 4 , Figure 4 yes Figure 1 In this embodiment, the base model of the large language model 13 uses a large language model (Pre-trained LLM) pre-trained on a large-scale simultaneous interpretation corpus, and combines LoRA (low rank adaptation) to perform model fine-tuning optimization.

[0051] In an actual simultaneous interpretation scenario, the speech coding feature vector, prompt words and event-related information obtained from the event encoder 12 are used as inputs to the large language model 13, so that the large language model can refine and summarize the input speech key points according to the actual situation, and finally convert it into a target speech vector representation in the target language.

[0052] Among them, prompt words can further enhance the key information extraction ability of the large language model.

[0053] In some implementations, prompts can be designed for specific scenarios. In some specific implementations, prompts that can be designed for speech scenarios include: summarizing the core arguments of the following speech and the key facts that support these arguments, and summarizing the main points conveyed by the speaker in concise language. In other implementations, prompts that can be designed for business negotiation scenarios include: extracting contract-related terms and conditions from this business negotiation. In still other implementations, prompts that can be designed for general scenarios include: translating after simplifying the voice content, and reducing modal particles in translation.

[0054] It can be understood that by enabling the large language model 13 to perform predictive generation in combination with the event information output by the event encoder 12, the large language model 13 can predict the speaker's next expression based on the context information to ensure the continuity and coherence of the translation, reduce unnecessary pauses, and improve the naturalness of the simultaneous interpretation results, thereby further improving the fluency of the translation.

[0055] In this embodiment, the decoder 14 is used to convert the generated target speech vector representation into speech in the target language. In some embodiments, the decoder 14 can be a pre-trained model such as WaveNet.

[0056] In some implementations, the simultaneous interpretation system 100 further includes an adapter 15. The adapter 15 acts as a bridge between the audio encoder 11 and the large language model 13, and is used to adjust the high-dimensional speech features output by the audio encoder 11 into vector representations that are easier for the LLM to process, thereby making the large language model 13 more adaptable when processing multilingual information.

[0057] In some specific implementations, the speech coding feature vector adjusted by the adapter 15, the prompt word and the event-related information obtained from the event encoder 12 are used as inputs of the large language model 13, so that the large language model can refine and summarize the input speech key points information according to the actual situation, and finally convert it into the target speech vector representation of the target language.

[0058] In this embodiment, when training the above-mentioned simultaneous interpretation system 100, the parameters of the audio encoder 11 and the decoder 14 are frozen so that they do not participate in the update. During the training process, the adapter 15 is fully fine-tuned so that it can convert the speech coding vectors into vector representations that are compatible with the large language model 13. Furthermore, the large language model 13 is fine-tuned using an enhanced fine-tuning scheme based on LoRA design to adapt it to the needs of capturing key information.

[0059] It can be understood that the simultaneous interpretation system 100 provided in this embodiment can capture and extract the key points of the speaker's speech information in real time during the machine simultaneous interpretation process, thereby realizing a simultaneous interpretation mode of extracting key points while translating. The above-mentioned simultaneous interpretation mode can effectively improve the accuracy of machine simultaneous interpretation in translating effective information, reduce translation delay, and provide an efficient simultaneous interpretation technical solution for some scenarios such as international conferences and cross-border business negotiations that have high requirements for simultaneous interpretation accuracy and delay.

[0060] See also Figure 5 , Figure 5 : is a flowchart of the first implementation of the speech translation method of the present application. In this implementation, the speech translation method includes:

[0061] S11: Obtain a first speech coding vector of the speech to be translated.

[0062] In this implementation, the speech to be translated may be the speech audio of the target object during speech, or may be a song or opera, and the specific type of the speech to be translated is not limited here.

[0063] Among them, the target object is a specific natural person who can occur.

[0064] In this implementation, the first speech coding vector is a high-dimensional speech feature corresponding to the speech to be translated.

[0065] S12: Perform real-time smoothing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector.

[0066] In this implementation, smoothing processing refers to real-time marking of semantically irrelevant feature vectors in the first speech coding vector to obtain marking information, and segmenting the first speech coding vector to obtain segmentation information.

[0067] Among them, the feature vectors unrelated to semantics refer to the information corresponding to events such as pauses, wrong words, corrections, and redundant information in the streaming speech.

[0068] In some implementations, the first speech coding vector is semantically segmented based on acoustic signals (such as speech amplitude, frequency, energy change, etc.) and combined with grammatical and syntactic information. In some specific implementations, when a complete syntactic structure is detected, the segmentation information is automatically output.

[0069] It can be understood that by performing real-time smooth processing on the first speech coding vector, event information in the streaming speech, such as stuttering, slip of the tongue, etc., can be detected so that these situations can be processed in real time during the subsequent translation process, thereby increasing the smoothness of simultaneous interpretation.

[0070] S13: Generate a second smooth speech coding vector with key information based on the first speech coding vector, the tag information, the segmentation information and at least one prompt word.

[0071] In this embodiment, the first speech coding vector is extracted and translated based on the tag information, segmentation information and prompt words to generate a second speech coding vector with key information and smoothness.

[0072] It can be understood that by extracting and translating the key information in the source speech, the translation of irrelevant information can be reduced, thereby indirectly reducing the delay between the target speech and the source speech, and then achieving the purpose of simultaneous speech.

[0073] S14: Decode the second speech coding vector to obtain a target speech; wherein the speech to be translated and the target speech correspond to different languages.

[0074] In some implementations, the language corresponding to the speech to be translated is Chinese, and the language corresponding to the target speech is English. In other implementations, the language corresponding to the speech to be translated is English, and the language corresponding to the target speech is Chinese. In still other implementations, the language corresponding to the speech to be translated is Chinese, and the language corresponding to the target speech is other Western languages ​​other than English, or Japanese, Korean, etc.

[0075] Different from the related art, this embodiment can summarize and accurately extract the key information in the source speech in real time by performing real-time smooth processing on the first speech coding vector corresponding to the speech to be translated to obtain marking information and segmentation information, so as to reduce translation delay, thereby improving the real-time and fluency of translation. Based on the first speech coding vector, marking information, segmentation information and prompt words, a smooth second speech coding vector with key information is generated, and the second speech coding vector is decoded to obtain the target speech, which can translate the source speech based on the key information, thereby improving the accuracy of translation and then improving the user experience.

[0076] See also Figure 6 , Figure 6 : is a flow chart of the second embodiment of the speech translation method of the present application. In this embodiment, the speech translation method is implemented by a real-time end-to-end simultaneous interpretation system, which includes an audio encoder, an event encoder, a large language model and a decoder. The speech translation method includes:

[0077] S21: Encode the speech to be translated by an audio encoder in the simultaneous interpretation system to obtain a first speech coding vector.

[0078] In this implementation, the audio encoder is used to extract relevant feature representations from the original audio signal corresponding to the speech to be translated, so as to convert the audio signal of the source speech into high-dimensional speech features.

[0079] In some implementations, the simultaneous interpretation system further includes an adapter.

[0080] In some specific implementations, after encoding the speech to be translated by an audio encoder in a simultaneous interpretation system to obtain a first speech encoding vector, the first speech encoding vector is processed by an adapter to generate a first speech vector representation adapted to a large language model.

[0081] Among them, the first speech vector representation is a vector representation that is easier to process by a large language model.

[0082] S22: Perform real-time smoothing on the first speech coding vector through an event encoder to obtain marking information and segmentation information corresponding to the first speech coding vector.

[0083] In some implementations, information corresponding to events such as pauses, wrong words, corrections, and redundant information in the first speech coding vector is detected by an event encoder, and the above information is marked to transmit the marked information to the large language model.

[0084] In some implementations, the first speech coding vector is semantically segmented by an event encoder based on acoustic signals (e.g., speech amplitude, frequency, energy change, etc.) and combined with grammatical and syntactic information. In some specific implementations, when a complete syntactic structure is detected, the segmentation information is automatically output.

[0085] S23: Processing the first speech coding vector, the marking information, the segmentation information and the prompt word through a large language model to obtain a second speech coding vector.

[0086] In some implementations, the first speech coding vector is processed by an adapter to generate a first speech vector representation adapted to a large language model, and then the first speech vector representation, tag information, segmentation information, and prompt words are processed by the large language model to obtain a second speech coding vector.

[0087] S24: Decode the second speech coding vector through a decoder to obtain the target speech.

[0088] Different from the related art, this embodiment uses an event encoder to smoothly process the first speech coding vector corresponding to the speech to be translated in real time to obtain marking information and segmentation information, and can detect event information in the streaming speech, such as stuttering, slip of the tongue, etc., to summarize and accurately extract the key information in the source speech in real time, so that the large language model can process these situations in real time, increase the smoothness of simultaneous interpretation, and at the same time, the streaming speech segmentation is realized through the event encoder, which can more intelligently identify the semantic boundaries in the speech, so as to better cope with the real-time requirements of the speech stream, reduce translation delays, and thus improve the real-time and fluency of translation. Furthermore, the large language model generates a smooth second speech coding vector with key information based on the first speech coding vector, marking information, segmentation information and prompt words, and decodes the second speech coding vector through the decoder to obtain the target speech, which can translate the source speech based on the key information, thereby improving the accuracy of the translation and then improving the user experience.

[0089] See also Figure 7 , Figure 7 1 is a flow chart of an implementation method of fine-tuning a large language base model. In this implementation method, the method includes:

[0090] S31: Acquire a training speech sample data set; wherein the training speech sample data set includes a plurality of training speech samples corresponding to a plurality of languages, and each training speech sample includes invalid information irrelevant to semantics.

[0091] In this embodiment, in order to obtain these training speech samples, the data processing process to be performed includes: original speech data collection, speech transcription and key information extraction, alignment and slicing, text translation, sound reproduction and speech synthesis, and data screening.

[0092] Among them, part of the training speech samples are real data collected from public corpora and real scenarios such as meetings and speeches, and the other part is a large amount of pseudo-labeled data generated by existing speech recognition models and speech synthesis models.

[0093] Understandably, simultaneous interpretation involves real-time translation of multiple languages, and training data needs to cover various language pairs. In order to meet the needs of some scenarios that require real-time and seamless communication, such as international conferences and cross-border business negotiations, in addition to collecting data from public corpora, meetings, speeches and other real-world scenarios, it is also necessary to ensure that these data contain repetitions, slips of the tongue and other phenomena to ensure that the model can be smoothly applied in actual scenarios.

[0094] Understandably, it is extremely expensive to specially record original voice data and target voice data containing only valid information. Generating some training voice samples can reduce training costs.

[0095] Specifically, please refer to Table 1, which is an example of original speech and target speech training data in a real communication scenario.

[0096] Table 1 Examples of original speech and target speech training data in real communication scenarios

[0097]

[0098] In actual speech, some information is key information, while some is secondary information. However, generally, only the original speech containing secondary information or stuttering can be obtained, and the target speech containing only valid information cannot be obtained. In order to obtain data containing only valid information, firstly, the key information in the corpus is extracted and summarized by using specific prompt words on the large language model, so as to obtain the key information text annotation corresponding to the speech.

[0099] Specifically, please refer to Table 2, which is an example of original speech and key information in a real communication scenario.

[0100] Table 2 Examples of original speech and key information in real communication scenarios

[0101]

[0102] Furthermore, in order to simulate the real-time requirements of simultaneous interpretation, the training speech samples should have streaming characteristics, that is, the input is audio or text fragments that arrive gradually, and the output should also be generated synchronously. Therefore, the text is sliced ​​according to the key information, and then the original speech is sliced ​​and aligned according to the mapping relationship between the text and the original speech content.

[0103] Specifically, please refer to Table 3, which is an example of conversion to streaming input and output.

[0104] Table 3 Example of conversion to streaming input and output

[0105]

[0106] S32: Obtain a corresponding training speech coding vector based on each training speech sample.

[0107] In this implementation, each training speech sample is encoded by an audio encoder to obtain a corresponding training speech coding vector for each training speech sample.

[0108] S33: Perform real-time smoothing processing on each training speech coding vector to obtain labeling information and segmentation information corresponding to the training speech coding vector.

[0109] For the specific process, please refer to the description in S12 and S22, which will not be repeated here.

[0110] S34: Input the training speech coding vector, tag information, segmentation information and prompt words corresponding to each training speech sample into the large language base model for fine-tuning training to obtain a trained large language model.

[0111] In this implementation, a low-rank adaptation matrix is ​​added next to the large language base model (i.e., the pre-trained large language model). The low-rank adaptation matrix is ​​initialized with a random Gaussian distribution to obtain a first matrix. The low-rank adaptation matrix is ​​initialized with a zero matrix to obtain a second matrix. The first matrix and the second matrix are trained using the training speech coding vector, tag information, segmentation information, prompt words, and speech parameters corresponding to each training speech sample. The trained first matrix is ​​multiplied by the second matrix and the pre-trained parameters of the large language base model are combined to obtain the fine-tuned model parameters. The large language base model is fine-tuned based on the fine-tuned model parameters to obtain a large language model.

[0112] Specifically, see Figure 8 , Figure 8 This is a schematic diagram of fine-tuning the pre-trained large language model based on LoRA in this application. In this implementation, a bypass is added next to the original pre-trained language model to simulate the intrinsic rank by performing dimensionality reduction and dimensionality increase operations, initializing matrix A with a random Gaussian distribution and initializing matrix B with a zero matrix. After the training is completed, the B matrix is ​​multiplied by the A matrix and the pre-trained model parameters are combined as the fine-tuned model parameters W, W = W0 + AB.

[0113] Among them, during the training process, more speech parameters are added to target the speech conditions that may occur in actual application scenarios in order to enhance the accuracy of the simultaneous interpretation system: 1) Noise injection: Add interference factors such as environmental noise, background sound, and echo to the training speech samples to simulate the suboptimal conditions in real scenarios; 2) Speech rate adjustment: Speed ​​up or slow down the training speech samples to enhance the model's adaptability to speeches at different speeds; 3) Pitch and intonation changes: Randomly change the pitch and intonation of the training speech samples to make the model more robust to different pronunciation styles; 4) Word order perturbation: In text input, moderately disrupt the word order or introduce synonym transformations so that the model can cope with different syntactic structures and expressions.

[0114] Furthermore, the speech of the target language is synthesized using text through a large language model, and the target language speech is made as similar to the original speech as possible through a sound replication method, while the synthesized target speech is controlled to be close in duration to the original speech.

[0115] Furthermore, in order to improve the quality of translation, the present embodiment formulates rules to filter out data of poor quality, such as language inconsistency, large sound difference between the source voice and the target voice, inconsistency between the target voice and the source voice in meaning, and large difference in duration between the target voice and the original voice.

[0116] Different from the related art, this embodiment can summarize and accurately extract the key information in the source speech in real time by performing real-time smooth processing on the first speech coding vector corresponding to the speech to be translated to obtain marking information and segmentation information, so as to reduce translation delay, thereby improving the real-time and fluency of translation. Based on the first speech coding vector, marking information, segmentation information and prompt words, a smooth second speech coding vector with key information is generated, and the second speech coding vector is decoded to obtain the target speech, which can translate the source speech based on the key information, thereby improving the accuracy of translation and then improving the user experience.

[0117] Correspondingly, the present application provides a speech translation device.

[0118] See also Fig. 9 , Fig. 9 In this embodiment, the speech translation device 40 includes a first speech coding vector acquisition module 41 , a smoothing processing module 42 , a second speech coding vector generation module 43 and a target speech generation module 44 .

[0119] The first speech coding vector acquisition module 41 is used to acquire the first speech coding vector of the speech to be translated.

[0120] A smoothing processing module 42, configured to perform real-time smoothing processing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector;

[0121] A second speech coding vector generating module 43, configured to generate a second speech coding vector having key information and being smooth based on the first speech coding vector, the marking information, the segmentation information and at least one prompt word;

[0122] The target speech generation module 44 is used to decode the second speech coding vector to obtain the target speech; wherein the speech to be translated and the target speech correspond to different languages.

[0123] For the specific process, please refer to the relevant text descriptions in S11 to S14 and S21 to S24, which will not be repeated here.

[0124] Different from the related art, the present embodiment performs real-time smooth processing on the first speech coding vector corresponding to the speech to be translated by the smooth processing module 42 to obtain the marking information and segmentation information, and can summarize and accurately extract the key information in the source speech in real time to reduce the translation delay, thereby improving the real-time and fluency of the translation. Based on the first speech coding vector, marking information, segmentation information and prompt words, the second speech coding vector with key information and smoothness is generated by the second speech coding vector generation module 43, and the second speech coding vector is decoded by the target speech generation module 44 to obtain the target speech, which can translate the source speech based on the key information, thereby improving the accuracy of the translation and then improving the user experience.

[0125] Correspondingly, the present application provides an electronic device.

[0126] See also Fig.10 , Fig.10 Schematic diagram of the structure of an electronic device of the present application. Fig.10 As shown, in this embodiment, the electronic device 50 includes a memory 51 and a processor 52.

[0127] In this embodiment, the memory 51 is used to store program data, and when the program data is executed, the steps in the above-mentioned speech translation method are implemented. The processor 52 is used to execute the program instructions stored in the memory 51 to implement the steps in the above-mentioned speech translation method.

[0128] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps in the speech translation method as described above. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by multiple integrated circuit chips.

[0129] Different from the related art, the present embodiment performs real-time and smooth processing on the first speech coding vector corresponding to the speech to be translated by the processor 52 to obtain marking information and segmentation information, and can summarize and accurately extract the key information in the source speech in real time to reduce translation delay, thereby improving the real-time and fluency of translation. Based on the first speech coding vector, marking information, segmentation information and prompt words, a smooth second speech coding vector with key information is generated, and the second speech coding vector is decoded to obtain the target speech, and the source speech can be translated based on the key information, thereby improving the accuracy of translation and then improving the user experience.

[0130] Correspondingly, the present application provides a computer-readable storage medium.

[0131] See also Fig.11 , Fig.11 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application.

[0132] The computer-readable storage medium 60 includes a computer program 601 stored on the computer-readable storage medium 60, and the computer program 601 implements the steps in the above-mentioned speech translation method when executed by the above-mentioned processor. Specifically, if the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium 60. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium 60, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned computer-readable storage medium 60 includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0135] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0136] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0137] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech translation method, characterized in that: include: Obtaining a first speech coding vector of the speech to be translated; Performing real-time smoothing processing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector; Generate a smooth second speech coding vector with key information based on the first speech coding vector, the tag information, the segmentation information and at least one prompt word; The second speech coding vector is decoded to obtain a target speech; wherein the speech to be translated and the target speech correspond to different languages.

2. The speech translation method according to claim 1, characterized in that: The step of performing real-time smoothing processing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector includes: Marking the feature vector irrelevant to semantics in the first speech coding vector in real time to obtain the marking information; and The first speech coding vector is segmented to obtain the segmentation information.

3. The speech translation method according to claim 1, characterized in that: The speech translation method is implemented by a real-time end-to-end simultaneous interpretation system, which includes an audio encoder, an event encoder, a large language model and a decoder; The step of obtaining a first speech coding vector of the speech to be translated comprises: Encoding the speech to be translated by the audio encoder in the simultaneous interpretation system to obtain the first speech coding vector; The step of performing real-time smoothing processing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector includes: Performing real-time smoothing processing on the first speech coding vector by the event encoder to obtain the marking information and the segmentation information corresponding to the first speech coding vector; The step of generating a smooth second speech coding vector having key information based on the first speech coding vector, the tag information, the segmentation information and at least one prompt word comprises: Processing the first speech coding vector, the tag information, the segmentation information and the prompt word through the large language model to obtain the second speech coding vector; The step of decoding the second speech coding vector to obtain the target speech comprises: The second speech coding vector is decoded by the decoder to obtain the target speech.

4. The speech translation method according to claim 3, characterized in that: The large language model is obtained by fine-tuning the large language base model through speech data annotated with key information; The method for fine-tuning a large language base model using speech data annotated with key information comprises: A training speech sample data set is obtained; wherein the training speech sample data set includes a plurality of training speech samples corresponding to a plurality of languages, and each of the training speech samples includes invalid information irrelevant to semantics; Obtaining a corresponding training speech coding vector based on each of the training speech samples; Performing real-time smoothing processing on each of the training speech coding vectors to obtain label information and segmentation information corresponding to the training speech coding vector; The training speech coding vector, the marking information, the segmentation information and the prompt word corresponding to each training speech sample are input into the large language base model for fine-tuning training to obtain the trained large language model.

5. The speech translation method according to claim 4, characterized in that: The step of inputting the training speech coding vector, the tag information, the segmentation information and the prompt word corresponding to each training speech sample into the large language base model for fine-tuning training to obtain the trained large language model comprises: Adding a low-rank adaptation matrix next to the large language base model; Initializing the low-rank adaptive matrix with random Gaussian distribution to obtain a first matrix; Performing zero matrix initialization on the low-rank adaptive matrix to obtain a second matrix; Training the first matrix and the second matrix by using the training speech coding vector, the marking information, the segmentation information, the prompt word and the speech parameter corresponding to each training speech sample; Multiplying the trained first matrix with the second matrix and combining the pre-trained parameters of the large language base model to obtain fine-tuned model parameters; Fine-tune the large language base model based on the fine-tuned model parameters to obtain the large language model.

6. The speech translation method according to claim 3, characterized in that: The event encoder is obtained by pre-training a test-time adaptive base model with large-scale simultaneous interpretation data, and fully fine-tuning the pre-trained base model with real-time speech data and feedback information corresponding to different individuals.

7. The speech translation method according to claim 3, characterized in that: The simultaneous interpretation system further includes an adapter. After the step of encoding the speech to be translated by the audio encoder in the simultaneous interpretation system to obtain the first speech coding vector, the method further includes: Processing the first speech encoding vector by the adapter to generate a first speech vector representation adapted to the large language model; The step of processing the first speech coding vector, the tag information, the segmentation information and the prompt word by the large language model to obtain the second speech coding vector includes: The first speech vector representation, the tag information, the segmentation information and the prompt word are processed by the large language model to obtain the second speech coding vector.

8. A speech translation device, characterized in that: include: A first speech coding vector acquisition module, used to acquire a first speech coding vector of the speech to be translated; A smoothing processing module, used for performing real-time smoothing processing on the first speech coding vector to obtain marking information and segmentation information corresponding to the first speech coding vector; A second speech coding vector generating module, configured to generate a second speech coding vector having key information and being smooth based on the first speech coding vector, the marking information, the segmentation information and at least one prompt word; The target speech generation module is used to decode the second speech coding vector to obtain the target speech; wherein the to-be-translated speech and the target speech correspond to different languages.

9. An electronic device, characterized in that: include: A memory for storing program data, wherein the stored program data, when executed, implements the steps of the speech translation method according to any one of claims 1 to 7; A processor is used to execute the program instructions stored in the memory to implement the steps in the speech translation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the speech translation method according to any one of claims 1 to 7 are implemented.

Images