Speech translation method, related device, equipment and storage medium

Abstract

The invention discloses a speech translation method, a related device, equipment and a storage medium, and the speech translation method comprises the steps: obtaining a plurality of initial recognition texts of a to-be-translated speech and a thinking chain used for indicating translation; processing the plurality of initial recognition texts according to the thinking chain by utilizing a large language model to obtain a final translation text of the to-be-translated speech; wherein the thinking chain at least comprises the following steps: carrying out error correction based on a plurality of initial recognition texts to obtain a voice recognition text, translating the voice recognition text into a target language to obtain a first translated text, translating the first translated text into a source language of the voice to be translated to obtain a second translated text, and translating the second translated text into a target language; and correcting the first translated text at least based on the difference between the second translated text and the speech recognition text to obtain a final translated text. According to the scheme, the accuracy of realizing speech translation by the large language model can be improved, especially under the condition of data resource limitation.

Description

Technical Field

[0001] The present application relates to the field of speech translation technology, and in particular to a speech translation method and related devices, equipment and storage media. Background Art

[0002] In recent years, the field of natural language processing has undergone a paradigm shift due to major advances in large language models, which are speech-based models pre-trained for processing speech data and can perform various speech-related tasks such as speech recognition and speech-to-text translation.

[0003] However, the above technologies usually rely on specialized data resources to adapt to specific tasks and perform fine-tuning training, which cannot be directly implemented in many cases. This limits the use of large language models in practical applications such as speech translation, especially in resource-limited environments. In view of this, how to improve the accuracy of speech translation using large language models, especially when data resources are limited, has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech translation method and related devices, equipment and storage media, which can improve the accuracy of speech translation achieved by large language models, especially when data resources are limited.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech translation method, including: obtaining a number of initial recognition texts of speech to be translated and a thought chain for indicating translation; using a large language model to process the number of initial recognition texts according to the thought chain to obtain a final translation text of the speech to be translated; wherein the thought chain at least includes: performing error correction based on the number of initial recognition texts to obtain a speech recognition text, and translating the speech recognition text into a target language to obtain a first translation text, translating the first translation text into a source language of the speech to be translated to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speech translation device, including: an acquisition module and a processing module, the acquisition module is used to acquire a number of initial recognition texts of the speech to be translated and a thought chain for indicating the translation; the processing module is used to use a large language model to process the number of initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated; wherein the thought chain at least includes: error correction based on the number of initial recognition texts to obtain a speech recognition text, and translation into a target language based on the speech recognition text to obtain a first translation text, translation into the source language of the speech to be translated based on the first translation text to obtain a second translation text, and correction of the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, which at least includes a memory and a processor coupled to each other, the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech translation method in the above first aspect.

[0008] In order to solve the above technical problem, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the speech translation method of the first aspect.

[0009] The above scheme obtains several initial recognition texts of the speech to be translated and a thought chain for indicating translation, and uses the large language model to process the several initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated, and the thought chain indication includes: performing error correction based on the several initial recognition texts to obtain a speech recognition text, and translating into the target language based on the speech recognition text to obtain a first translation text, translating into the source language of the speech to be translated based on the first translation text to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text. Therefore, on the one hand, during the translation process of the large language model, the large language model is guided by the thought chain for indicating translation, which can reduce the dependence of the large language model on data resources required for realizing speech translation as much as possible. On the other hand, since the large language model corrects the first translation text according to the difference between the speech recognition text in the source language and the second translation text translated into the target language through reverse translation (i.e., from the target language to the source language) as indicated by the thought chain, the two-way translation capability of the large language model can be used to achieve self-correction, which helps to improve the accuracy of speech translation as much as possible. Therefore, it is possible to improve the accuracy of speech translation achieved by large language models, especially when data resources are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1It is a flowchart of an embodiment of the speech translation method of the present application; Figure 2a This is a process diagram of an embodiment of the speech translation method of the present application; Figure 2b It is a process diagram of another embodiment of the speech translation method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the speech translation device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the speech translation method of the present application. Specifically, it may include the following steps: Step S11: obtaining a number of initial recognition texts of the speech to be translated and a thought chain for indicating the translation.

[0015] In one implementation scenario, as a possible implementation method, any speech recognition model such as WavLM, Whistler, etc. can be used to recognize the speech to be translated to obtain several initial recognition texts of the speech to be translated. It should be noted that the speech recognition model can use the beam search algorithm in the decoding process to generate N-best as several initial recognition inverse texts for each input speech. For example, the default beam width can be set to 60. After deduplication, the top N (such as 5) decoded texts with the highest probability can be selected as N-best, that is, several initial recognition texts. For details, please refer to the technical details of the beam search, which will not be repeated here.

[0016] In another implementation scenario, different from the aforementioned implementation, as another possible implementation, several speech recognition models can also be obtained, and the configuration information of each speech recognition model includes the applicable scenario of the speech recognition model. Exemplarily, taking several speech recognition models including WavLM and Whisper as examples, WavLM performs well in tasks such as reading text aloud, because it is very well trained on LibriSpeech, and its training data comes from a restricted environment, which may cause "domain mismatch" problems in more complex or noisy real application scenarios; Whisper uses a wider range of training data, so it is robust when processing speech in different languages, accents and background noise, but Whisper's versatility means that its performance on specific tasks in specific fields may not be as good as models optimized for those fields. Of course, the above example is only a possible example of several speech recognition models in actual application, and the specific model types of several speech recognition models are not limited here. On this basis, the speech recognition model can be selected as the target recognition model based on the translation scenario where the speech to be translated is located and the configuration information of each speech recognition model. Exemplarily, still taking several speech recognition models including WavLM and Whisper as an example, when there is background noise in the translation scene where the speech to be translated is located, Whisper can be selected as the target recognition model; or, when the translation scene where the speech to be translated is located is a specific task such as reading aloud text, WavLM can be selected as the target recognition model. Of course, the above example is only a possible example of selecting a target recognition model when several speech recognition models include WavLM and Whisper, and other possible situations are not given examples one by one here. After the target recognition model is selected, the speech to be translated can be recognized based on the target recognition model to obtain several initial recognition texts. As mentioned above, in the process of recognizing the speech to be translated using the target recognition model, it can be specifically decoded by, for example, a beam search algorithm to obtain several initial recognition texts. In the above method, several speech recognition models are obtained, and the configuration information of each speech recognition model includes the applicable scenario of the speech recognition model. Based on the translation scenario of the speech to be translated and the configuration information of each speech recognition model, the speech recognition model is selected as the target recognition model, and then the speech to be translated is recognized based on the target recognition model to obtain several initial recognition texts. The target recognition model can be adaptively selected according to the translation scenario of the speech to be translated, which helps to improve the accuracy of speech recognition.

[0017] In one implementation scenario, the thought chain used to indicate translation can describe the translation process in natural language text combined with reference examples. Taking the thought chain including error proofreading and comparison reverse translation as an example, the thought chain can include but is not limited to the following natural language text and reference examples: The following text is a number of initial recognition texts output by a speech recognition model. Please proofread the errors in the following initial recognition texts and translate them into [Target Language], and ensure the relevance and accuracy of the translation through back-translation. The specific steps are as follows: Proofread the potential errors in a number of initial recognition texts to obtain the speech recognition text, and translate the speech recognition text into [Target Language] as the first translation text, and then back-translate the first translation text into [Source Language] as the second translation text, and correct the first translation text according to the differences between the second translation text and the speech recognition text to obtain the final translation text. The specific steps are as follows: Step 1. Error proofreading: Proofread the potential errors in a number of initial recognition texts to obtain the speech recognition text.

Initial recognition text

Speech recognition text

[0018]

Initial recognition text

[0019]

Speech recognition text

[0020] Step 2. Text translation: Translate the speech recognition text into the target language as the first translation text.

[0021]

Speech recognition text

[0022]

First translation text

[0023] Step 3. Back-translation and translation correction: Back-translate the first translation text into [Source Language] as the second translation text, and correct the first translation text according to the differences between the second translation text and the speech recognition text to obtain the final translation text.

Speech translation text

First translation text

[0024]

Second translation text

[0025]

Final translation text

[0026]

Final translation text

[0027]

Final translation text

[0028] It should be noted that the above example is only a possible example when the thinking chain includes error proofreading and comparison reverse translation, and the specific content of the thinking chain is not limited here. Of course, when the thinking chain also includes other processing, the thinking chain can also refer to other contents, and no more examples are given here.

[0029] In an implementation scenario, each step in the translation process described by the thinking chain for instructing translation is provided with a prompt instruction described in natural language, and the prompt instruction is used to instruct the large language model to execute the corresponding step. Still taking the aforementioned thinking chain as an example, the prompt instructions contained in the thinking chain may include but are not limited to: the prompt instruction of the first step "error proofreading" "proofreading the potential errors of several initial recognition texts to obtain speech recognition text", the prompt instruction of the second step "text translation" "translating the speech recognition text into the target language as the first translation text", and the prompt instruction of the third step "reverse translation and translation correction" "reverse translating the first translation text into [source language] as the second translation text, and correcting the first translation text according to the difference between the second translation text and the speech recognition text as the final translation text". Of course, the above example is only an example of the aforementioned thinking chain, a possible example of the prompt instructions contained in the thinking chain, and does not limit the specific content of the prompt instructions contained in the thinking chain. Here, the prompt instructions contained in the thinking chain will not be given one by one.

[0030] Step S12: using the large language model to process a number of initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated.

[0031] In the disclosed embodiment, the thought chain may at least include: performing error correction based on a number of initial recognition texts to obtain a speech recognition text, translating the speech recognition text into a target language to obtain a first translation text, translating the first translation text into a source language of the speech to be translated to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text. In addition, the large language model may include but is not limited to open source large models such as LLAMA, Bloom, or a custom large model, or a paid large model, and the specific source of the large language model is not limited here.

[0032] In an implementation scenario, in the process of error correction of several initial recognition texts, the potential error categories of each initial recognition text can be identified first, and then the potential error categories of the initial texts can be corrected based on reference information to obtain speech recognition texts, and the reference information can include: context information and a knowledge base of the source language. The above method, by first identifying the potential error categories of the initial recognition texts, and then correcting the potential error categories in combination with the context information and the knowledge base of the source language to obtain the speech recognition text, can improve the internal coherence of the speech recognition text itself on the one hand, and on the other hand, can ensure that the speech recognition model conforms to the reasonable expression of the source language as much as possible.

[0033] In a specific implementation scenario, the large language model can be used to identify potential error categories in the initial recognition text. It should be noted that potential error categories may include but are not limited to: recognition errors, vocabulary errors, semantic errors, etc., and the specific types of error categories are not limited here.

[0034] In a specific implementation scenario, taking Chinese as the source language as an example, the knowledge base of the source language may include but is not limited to: relevant knowledge of Chinese grammar, relevant knowledge of Chinese idioms / proverbs, Chinese Internet buzzwords, etc. The specific contents of the knowledge base of the source language will not be given one by one here.

[0035] In a specific implementation scenario, as a possible example, in this case, the "first step, error proofreading" in the thinking chain can be specifically expressed as including but not limited to: Step 1. Error proofreading: Each initial recognition text is marked with potential error categories. Please proofread the potential errors in a number of initial recognition texts by using the context information and the knowledge base of the source language to obtain the speech recognition text.

Initial recognition text

[0036]

Initial recognition text

[0037]

Speech recognition text

[0038] Of course, the above example is only a possible example of error correction in the thinking chain when the above process steps are specifically adopted for error correction. Other possible situations will not be given one by one here. Taking the voice to be translated "The weather is really nice today" as an example, it can contain the following initial recognition texts: "The weather is really nice today", "The air is really nice today", "The weather is wrong today", among which, the first initial recognition text "The weather is really nice today" can be determined to have a potential error category of "none" after being recognized by the large language model, the second initial recognition text "The air is really nice today" can be determined to have a potential error category of "recognition error" after being recognized by the large language model, and the third initial recognition text "The weather is wrong today" can be determined to have a potential error category of "semantic error" after being recognized by the large language model. Then, after being processed by the large language model according to the above thinking chain, the following content can be output: Step 1. Error proofreading

Initial recognition text

[0039] The weather is really nice today.

[0040] The weather is really good today.

[0041]

Speech recognition text

[0042] It should be noted that the above example is only a possible example of the output content of the large language model according to the chain of thought when the speech to be translated is "The weather is really nice today" in actual application, and it is not limited to the output content of the large language model according to the chain of thought when the speech to be translated is other speech, and no examples will be given one by one here.

[0043] In a specific implementation scenario, after the large language model performs error proofreading on several initially recognized texts according to the thought chain, it can also output the correction content and correction reason of each initially recognized text. In this case, the specific content of "error proofreading" in the thought chain can be expressed as: Step 1. Error proofreading: Proofread the potential errors in a number of initial recognition texts to obtain the speech recognition text.

Initial recognition text

[0044]

Speech recognition text

[0045]

Correction content and reason

[0046]

Initial recognition text

[0047] Still taking the aforementioned voice to be translated "Today's weather is really nice" as an example, after being processed by the large language model according to the aforementioned thinking chain, the following content can be output:

Speech recognition text

Initial recognition text

[0048] The weather is really nice today.

[0049] The weather is really good today.

[0050]

Speech recognition text

[0051]

Correction content and reason

[0052] Step 1. Error proofreading: Proofread the potential errors in a number of initial recognition texts to obtain the speech recognition text.

Initial recognition text

[0053] It should be noted that the above example is only a possible example of the output content of the correction content and the correction reason of the large language model according to the chain of thought when the speech to be translated is "The weather is really nice today" in actual application. It is not limited to the output content of the correction content and the correction reason of the large language model according to the chain of thought when the speech to be translated is other speech, and no examples are given here one by one.

[0054] In one implementation scenario, different from the aforementioned implementation, as another possible example, when the speech to be translated is long, each initial recognition text may contain multiple initial recognition sentences. Then, the initial recognition sentences of the same speech segment in the speech to be translated can be extracted from each initial recognition text to obtain a recognition sentence set corresponding to the speech segment, and error correction is performed based on each recognition sentence set to obtain the speech recognition sentence of the speech segment corresponding to the recognition sentence set, and then the speech recognition sentences of each speech segment in the speech to be translated are combined to obtain the speech recognition text. The above method can process several recognition texts of long speech through a series of operations such as sentence separation, sentence correction, and sentence merging.

[0055] In a specific implementation scenario, when the speech to be translated is long, the specific content of "error proofreading" in the thought chain can be expressed as follows: The initial recognition statement of speech segment 1 in the first initial recognition text, the initial recognition statement of speech segment 1 in the second initial recognition text, ……, the initial recognition statement of speech segment 1 in the Nth initial recognition text; The initial recognition statement of speech segment 2 in the first initial recognition text, the initial recognition statement of speech segment 2 in the second initial recognition text, ……, the initial recognition statement of speech segment 2 in the Nth initial recognition text;

[0056]

Initial recognition text

Speech Recognition Text

[0057] It should be noted that the above example is only a possible example of the specific content of error proofreading in the chain of thought when the speech to be translated is long, and it does not limit the use of other contents in the chain of thought regarding error proofreading. The specific content of error proofreading in the chain of thought will not be given one by one here.

[0058] In a specific implementation scenario, the specific process of error correction of the recognition sentence set can refer to the aforementioned description of error correction of several initial recognition texts, which will not be repeated here. In addition, as mentioned above, after the error correction, the correction content and correction reason of each initial recognition text can also be output. Similarly, in the case where the speech to be translated is long, after the error correction, the correction content and correction reason of each initial recognition sentence in the recognition sentence set can also be output. For details, please refer to the aforementioned related description, which will not be repeated here.

[0059] In an implementation scenario, as described above, after obtaining the speech recognition text, the large language model can translate the speech recognition text into the target language as indicated by the thought chain to obtain a first translated text. As a possible example, before the first translated text is reversely translated, relevance verification and accuracy verification can also be performed based on the first translated text. It should be noted that the relevance verification is used to verify whether the first translated text contains the semantic information and contextual information of the speech recognition text, and the accuracy verification is used to verify whether the first translated text accurately reflects all elements in the speech recognition text. In the above method, after translating the speech recognition text into the target language and before reverse translation, the first translated text in the target language is first subjected to relevance verification and accuracy verification, which can be pre-processed before reverse translation, and helps to improve the processing efficiency of speech translation according to the thought chain.

[0060] In a specific implementation scenario, when the thought chain further includes relevance verification and accuracy verification, the thought chain can be further expressed as: The texts provided below are some initial recognition texts output by the speech recognition model. Proofread the text for errors and translate it into the [target language], as well as ensure the relevance and accuracy of the translation through back translation The specific steps are as follows: Step 1: Error proofreading: Each initial recognition text is marked with potential error categories. Please use contextual information The potential errors of several initial recognition texts are proofread using the knowledge base of the source language to obtain speech recognition text.

[0061]

Initial recognition text

[0062]

Speech recognition text

[0063] [Correction content and reason]:…….

[0064] Step 2: Text translation and verification: Translate the speech recognition text into the target language as the first translation text. And check the relevance and accuracy of the first translation text.

[0065]

Speech recognition text

[0066]

First translation text

[0067]

Correlation check and accuracy check

[0068] Step 3: Back-translation and translation correction: Back-translate the first translation text into the [source language] as the second The first translated text is corrected according to the difference between the second translated text and the speech recognition text as the final translated text. Translated text.

[0069]

Voice translation text

[0070]

First translation text

[0071]

Second translation text

[0072]

Final translation text

[0073] It should be noted that the above example is only a possible example of the specific content of the thinking chain when the thinking chain further includes relevance verification and accuracy verification, and the specific content of the thinking chain is not limited here.

[0074] In a specific implementation scenario, still taking the speech recognition text "The weather is extremely good today" as an example, if the first translation text translated into English from the speech recognition text is "The weather is extremely good today", after relevance verification and accuracy verification, it can be determined that the first translation text is correctly translated. Of course, the above example is only a special example, and other possible situations will not be given one by one here.

[0075] In an implementation scenario, as a possible example, after obtaining a first translation text in which a large language model translates a speech recognition text into a target language and a second translation text in which a large language model reversely translates the first translation text into a source language, the first translation text can be directly corrected based on the difference between the speech recognition text and the second translation text to obtain a final translation text. It should be noted that in this case, the prompt instruction about "reverse translation and translation correction" in the thinking chain can directly indicate that the first translation text can be corrected based on the difference between the speech recognition text and the second translation text as the final translation text, and the specific examples can be referred to in the above examples, which will not be repeated here. In addition, for ease of understanding, the aforementioned voice to be translated "The weather is really good today" is still taken as an example. As mentioned above, the speech recognition text is "The weather is really good today", the first translation text is "The weather is extremely good today", and the first translation text is reversely translated into the second translation text of the source language as "The weather is extremely good today". Since there is basically no difference between the speech recognition text and the second translation text (that is, "really good" and "extremely good" are basically expressed in the same way), the first translation text can be directly used as the final translation text without being corrected. Of course, the above example is only a special example in actual application, and other possible situations will not be given one by one here.

[0076] In another implementation scenario, different from the aforementioned implementation, as another possible example, after obtaining the first translation text in which the large language model translates the speech recognition text into the target language and the second translation text in which the large language model reversely translates the first translation text into the source language, the first translation text can be corrected based on the difference between the second translation text and the speech recognition text to obtain a third translation text, and then the third translation text is reviewed and corrected with the consistency of the third translation text itself as the desired goal to obtain the final translation text. It should be noted that in this case, the prompt instruction about "reverse translation and translation correction" in the thinking chain can indicate that after correcting the first translation text, the translation text itself is further reviewed and corrected with the consistency of the translation text itself as the desired goal. For details, please refer to the aforementioned example, which will not be repeated here. In addition, for ease of understanding, the aforementioned speech to be translated "The weather is extremely good today" is still taken as an example. As mentioned above, the speech recognition text is "The weather is extremely good today", the first translation text is "The weather is extremely good today", and the first translation text is reversely translated into the second translation text of the source language as "The weather is extremely good today". Since there is basically no difference between the speech recognition text and the second translation text (that is, "really good" and "extremely good" are basically expressed in the same way), the first translation text can be left unchanged. At this time, the first translation text "The weather is extremely good today" is used as the third translation text, and based on the consistency of the third translation text itself as the desired goal, it is reviewed and corrected. Since the third translation text itself is expressed in the same context, the third translation text "The weather is extremely good today" can be used as the final translation text. Of course, the above example is only a special example in the actual application process, and other possible situations are not given examples one by one here. The above method corrects the first translation text based on the difference between the second translation text and the speech recognition text to obtain the third translation text, and then reviews and corrects the third translation text with the consistency of the third translation text itself as the desired goal to obtain the final translation text. It can combine a series of operations such as reverse translation and self-correction to further improve the quality and reliability of the translation content and reduce translation errors.

[0077] In an implementation scenario, please refer to Figure 2a and Figure 2b , Figure 2a is a process diagram of an embodiment of the speech translation method of the present application, Figure 2b It is a process diagram of another embodiment of the speech translation method of the present application. Specifically, Figure 2b 1 is a process diagram for indicating an embodiment of a thought chain of translation. Figure 2a and Figure 2bAs shown, the speech to be translated is subjected to speech recognition by the speech recognition model to obtain several initial recognition texts. In the case where the speech to be translated is a long speech, the several initial recognition texts can be decomposed based on the principle that they belong to the same speech segment in the speech to be translated to obtain several recognition sentence sets, and the initial recognition sentences corresponding to the same speech segment in the speech to be translated in each initial recognition text can be attributed to the same recognition sentence set, such as recognition sentence set 1 can include initial recognition sentences corresponding to speech segment 1 in each initial recognition text, recognition sentence set 2 can include initial recognition sentences corresponding to speech segment 2 in each initial recognition text, ..., recognition sentence set M can include initial recognition sentences corresponding to speech segment M in each initial recognition text. On this basis, each recognition sentence set can be combined with context information and the knowledge base of the source language to perform error correction respectively to obtain speech recognition sentences of each speech segment, and the speech recognition text of the speech to be translated can be obtained by combining them. The speech recognition text can be translated into the target language through the large language model to obtain the first translation text, and the first translation text can be reversely translated into the source language through the large language model to obtain the second translation text. The second translation text can be compared with the speech recognition text to correct the first translation text according to the difference between the two to obtain the third translation text. Finally, the third translation text can be reviewed and corrected with the consistency of the third translation text itself as the expected goal. After the review and correction, the final translation text can be obtained.

[0078] The above scheme obtains several initial recognition texts of the speech to be translated and a thought chain for indicating translation, and uses the large language model to process the several initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated, and the thought chain indication includes: performing error correction based on the several initial recognition texts to obtain a speech recognition text, and translating into the target language based on the speech recognition text to obtain a first translation text, translating into the source language of the speech to be translated based on the first translation text to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text. Therefore, on the one hand, during the translation process of the large language model, the large language model is guided by the thought chain for indicating translation, which can reduce the dependence of the large language model on data resources required for realizing speech translation as much as possible. On the other hand, since the large language model corrects the first translation text according to the difference between the speech recognition text in the source language and the second translation text translated into the target language through reverse translation (i.e., from the target language to the source language) as indicated by the thought chain, the two-way translation capability of the large language model can be used to achieve self-correction, which helps to improve the accuracy of speech translation as much as possible. Therefore, it is possible to improve the accuracy of speech translation achieved by large language models, especially when data resources are limited.

[0079] See also Figure 3 , Figure 3It is a schematic diagram of a framework of an embodiment of a speech translation device of the present application. The speech translation device 30 includes: an acquisition module 31 and a processing module 32, wherein the acquisition module 31 is used to acquire a number of initial recognition texts of the speech to be translated and a thought chain for indicating translation; the processing module 32 is used to process a number of initial recognition texts according to the thought chain using a large language model to obtain a final translation text of the speech to be translated; wherein the thought chain at least includes: performing error correction based on a number of initial recognition texts to obtain a speech recognition text, and translating the speech recognition text into a target language to obtain a first translation text, translating the first translation text into a source language of the speech to be translated to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text.

[0080] In the above scheme, the speech translation device 30 obtains a number of initial recognition texts of the speech to be translated and a thought chain for indicating translation, and uses the large language model to process the number of initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated, and the thought chain indication includes: error correction based on the number of initial recognition texts to obtain a speech recognition text, and translation into the target language based on the speech recognition text to obtain a first translation text, translation into the source language of the speech to be translated based on the first translation text to obtain a second translation text, and correction of the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text. Therefore, on the one hand, the large language model is guided by the thought chain for indicating translation during the translation process of the large language model, which can reduce the dependence of the large language model on data resources required for speech translation as much as possible. On the other hand, because the large language model corrects the first translation text according to the difference between the speech recognition text in the source language and the second translation text translated into the target language through reverse translation (i.e., from the target language to the source language) as indicated by the thought chain, the two-way translation capability of the large language model can be used to achieve self-correction, which helps to improve the accuracy of speech translation as much as possible. Therefore, it is possible to improve the accuracy of speech translation achieved by large language models, especially when data resources are limited.

[0081] In some disclosed embodiments, the processing module 32 includes a translation correction submodule, which is used to correct the first translation text based on the difference between the second translation text and the speech recognition text to obtain a third translation text; the processing module 32 includes a translation self-core submodule, which is used to review and correct the third translation text with the consistency of the third translation text itself as the desired goal to obtain a final translation text.

[0082] In some disclosed embodiments, the processing module 32 includes an error recognition submodule for identifying potential error categories of each initial recognition text; the processing module 32 includes an error correction submodule for correcting potential error categories of the initial recognition text based on reference information to obtain a speech recognition text; wherein the reference information includes: context information and a knowledge base of the source language.

[0083] In some disclosed embodiments, the processing module 32 includes a text decomposition submodule, which is used to extract the initial recognition sentences of the same speech segment in the speech to be translated from each initial recognition text, and obtain the recognition sentence set of the corresponding speech segment; the processing module 32 includes a set correction submodule, which is used to perform error correction based on each recognition sentence set, and obtain the speech recognition sentence of the speech segment corresponding to the recognition sentence set; based on the speech recognition sentences of each speech segment in the speech to be translated, the speech recognition text is obtained by combining them.

[0084] In some disclosed embodiments, the processing module 32 includes a correction output submodule for outputting the correction content and correction reasons of each initial recognition text after performing error correction based on several initial recognition texts to obtain speech recognition texts, and before translating the speech recognition texts into a target language to obtain a first translation text.

[0085] In some disclosed embodiments, the processing module 32 includes a translation verification submodule for performing relevance verification and accuracy verification based on the first translation text after translating the speech recognition text into the target language to obtain a first translation text, and before translating the first translation text into the source language of the speech to be translated to obtain a second translation text; wherein the relevance verification is used to verify whether the first translation text contains the semantic information and contextual information of the speech recognition text, and the accuracy verification is used to verify whether the first translation text accurately reflects all elements in the speech recognition text.

[0086] In some disclosed embodiments, the acquisition module 31 includes a model acquisition submodule for acquiring several speech recognition models; wherein the configuration information of each speech recognition model includes the applicable scenario of the speech recognition model; the acquisition module 31 includes a model selection submodule for selecting a speech recognition model as a target recognition model based on the translation scenario in which the speech to be translated is located and the configuration information of each speech recognition model; the acquisition module 31 includes a speech recognition submodule for recognizing the speech to be translated based on the target recognition model to obtain several initial recognition texts.

[0087] In some disclosed embodiments, the thought chain describes the translation process in natural language text in combination with reference examples; and / or, each step in the translation process described by the thought chain is provided with a prompt instruction described in natural language, and the prompt instruction is used to instruct the large language model to execute the corresponding step.

[0088] See also Figure 4 , Figure 4 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other, the memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned speech translation method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include but is not limited to a server, a camera, a smart phone, a tablet computer, etc., and the specific type of the electronic device 40 is not limited here.

[0089] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned speech translation method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0090] In the above scheme, the electronic device 40 obtains several initial recognition texts of the speech to be translated and a thought chain for indicating translation, and uses the large language model to process the several initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated, and the thought chain indication includes: error correction based on the several initial recognition texts to obtain a speech recognition text, and translation into the target language based on the speech recognition text to obtain a first translation text, translation into the source language of the speech to be translated based on the first translation text to obtain a second translation text, and correction of the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text. Therefore, on the one hand, the large language model is guided by the thought chain for indicating translation during the translation process of the large language model, which can reduce the dependence of the large language model on data resources required for realizing speech translation as much as possible. On the other hand, since the large language model corrects the first translation text according to the difference between the speech recognition text of the source language and the second translation text of the first translation text translated into the target language through reverse translation (i.e., from the target language to the source language) as indicated by the thought chain, the bidirectional translation capability of the large language model can be used to achieve self-correction, which helps to improve the accuracy of speech translation as much as possible. Therefore, it is possible to improve the accuracy of speech translation achieved by large language models, especially when data resources are limited.

[0091] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned speech translation method embodiments.

[0092] In the above scheme, the computer-readable storage medium 50 obtains a number of initial recognition texts of the speech to be translated and a thought chain for indicating translation, and uses the large language model to process the number of initial recognition texts according to the thought chain to obtain the final translation text of the speech to be translated, and the thought chain indication includes: error correction based on the number of initial recognition texts to obtain a speech recognition text, and translation into the target language based on the speech recognition text to obtain a first translation text, translation into the source language of the speech to be translated based on the first translation text to obtain a second translation text, and correction of the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain a final translation text. Therefore, on the one hand, the large language model is guided by the thought chain for indicating translation during the translation process of the large language model, which can reduce the dependence of the large language model on data resources required for realizing speech translation as much as possible. On the other hand, because the large language model corrects the first translation text according to the difference between the speech recognition text of the source language and the second translation text of the first translation text translated into the target language through reverse translation (i.e., from the target language to the source language) as indicated by the thought chain, the bidirectional translation capability of the large language model can be used to achieve self-correction, which helps to improve the accuracy of speech translation as much as possible. Therefore, it is possible to improve the accuracy of speech translation achieved by large language models, especially when data resources are limited.

[0093] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0094] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0098] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0099] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A speech translation method, characterized in that: include: Acquire a number of initial recognition texts of speech to be translated and a thought chain for indicating translation; Processing the plurality of initial recognition texts according to the thought chain using a large language model to obtain a final translation text of the speech to be translated; The thought chain at least includes: performing error correction based on the several initial recognition texts to obtain speech recognition text, translating the speech recognition text into a target language to obtain a first translation text, translating the first translation text into a source language of the speech to be translated to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain the final translation text.

2. The method according to claim 1, characterized in that The step of correcting the first translated text at least based on the difference between the second translated text and the speech recognition text to obtain the final translated text includes: Correcting the first translated text based on the difference between the second translated text and the speech recognition text to obtain a third translated text; The third translation text is reviewed and corrected with the consistency of the third translation text itself as the desired goal to obtain the final translation text.

3. The method according to claim 1, characterized in that: The error correction is performed based on the plurality of initial recognition texts to obtain speech recognition texts, including: Identifying potential error categories of each of the initially recognized texts; The potential error categories of the initial recognition text are corrected based on reference information to obtain the speech recognition text; wherein the reference information includes: context information and a knowledge base of the source language.

4. The method according to claim 1, characterized in that: Before performing error correction based on the plurality of initial recognition texts to obtain speech recognition texts, the thought chain further includes: Extracting respectively the initial recognition sentences of the same speech segment in the speech to be translated from each of the initial recognition texts to obtain a recognition sentence set corresponding to the speech segment; The error correction is performed based on the plurality of initial recognition texts to obtain speech recognition texts, including: Perform error correction based on each of the recognition sentence sets to obtain a speech recognition sentence of the recognition sentence set corresponding to the speech segment; The speech recognition text is obtained by combining the speech recognition sentences of each speech segment in the speech to be translated.

5. The method according to claim 1, characterized in that After performing error correction based on the plurality of initial recognition texts to obtain speech recognition texts, and before translating the speech recognition texts into a target language to obtain a first translation text, the method further includes: Outputting the correction content and correction reason of each of the initially recognized texts; And / or, after translating the speech recognition text into the target language to obtain a first translated text, and before translating the first translated text into the source language of the speech to be translated to obtain a second translated text, the method further includes: Performing relevance verification and accuracy verification based on the first translation text; The relevance check is used to check whether the first translation text contains the semantic information and context information of the speech recognition text, and the accuracy check is used to check whether the first translation text accurately reflects all elements in the speech recognition text.

6. The method according to claim 1, characterized in that The step of obtaining a plurality of initial recognition texts of the speech to be translated includes: Acquire several speech recognition models; wherein the configuration information of each of the speech recognition models includes the applicable scenario of the speech recognition model; Based on the translation scenario of the speech to be translated and the configuration information of each of the speech recognition models, selecting the speech recognition model as the target recognition model; The speech to be translated is recognized based on the target recognition model to obtain the plurality of initial recognition texts.

7. The method according to any one of claims 1 to 6, characterized in that: The thought chain describes the translation process in natural language text combined with reference examples; And / or, each step in the translation process described by the thought chain is provided with a prompt instruction described in natural language, and the prompt instruction is used to instruct the large language model to execute the corresponding step.

8. A speech translation device, characterized in that: include: An acquisition module, used to acquire a number of initial recognition texts of the speech to be translated and a thought chain for indicating the translation; A processing module, used for processing the plurality of initial recognition texts according to the thought chain using a large language model to obtain a final translation text of the speech to be translated; The thought chain at least includes: performing error correction based on the several initial recognition texts to obtain speech recognition text, translating the speech recognition text into a target language to obtain a first translation text, translating the first translation text into a source language of the speech to be translated to obtain a second translation text, and correcting the first translation text based on at least the difference between the second translation text and the speech recognition text to obtain the final translation text.

9. An electronic device, characterized in that: The invention at least comprises a memory and a processor coupled to each other, wherein the memory at least stores program instructions, and the processor is used to execute the program instructions to implement the speech translation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech translation method according to any one of claims 1 to 7.

Images