Speech translation method, system and related device

Abstract

The invention discloses a speech translation method and system and a related device. The method comprises the following steps: acquiring an input audio of at least one end, determining a to-be-translated language of the input audio, taking the input audio of each end as the to-be-translated audio, and determining an initial recognition text corresponding to the to-be-translated audio; wherein the initial recognition text is matched with at least one recognition language; based on the to-be-translated language, the initial recognition text and the corresponding recognition language, obtaining a prompt text matched with the to-be-translated audio; wherein the prompt text comprises a conversion language matched with the recognition language; obtaining a reference vocabulary matched with the initial recognition text from a candidate word library; and based on the initial recognition text, the prompt text and the reference vocabulary, obtaining a translation text corresponding to the to-be-translated audio. In this way, the accuracy and efficiency of speech translation can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a speech translation method, system and related devices. Background Art

[0002] With the rapid development of artificial intelligence, its application in the field of translation is becoming more and more extensive. At present, there are many pre-trained translation models for audio translation in the prior art. However, in actual applications, due to factors such as the high degree of diversity in translation scenarios, rich terminology, and large cultural differences, it is impossible to accurately and effectively translate the audio input by the user alone, and translation errors and text nesting often occur.

[0003] Therefore, how to accurately and effectively translate the translation text is a problem that needs to be solved urgently in the prior art. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech translation method, system and related devices, which can improve the accuracy and efficiency of speech translation.

[0005] To solve the above technical problems, a technical solution adopted by the present application is: to provide a speech translation method, comprising: obtaining input audio from at least one end, determining the language to be translated of the input audio, taking the input audio from each end as the audio to be translated, and determining the initial recognition text corresponding to the audio to be translated; wherein the initial recognition text matches at least one recognition language; based on the language to be translated, the initial recognition text and the recognition language corresponding to it, obtaining a prompt text matching the audio to be translated; wherein the prompt text includes a conversion language matching the recognition language; obtaining a reference vocabulary matching the initial recognition text from a candidate vocabulary library; and obtaining a translation text corresponding to the audio to be translated based on the initial recognition text, the prompt text and the reference vocabulary.

[0006] To solve the above technical problems, another technical solution adopted by the present application is: to provide a speech translation system, comprising: a recognition module, used to obtain input audio from at least one end, determine the language to be translated of the input audio, use the input audio from each end as the audio to be translated, and determine the initial recognition text corresponding to the audio to be translated; wherein the initial recognition text matches at least one recognition language; a first processing module, used to obtain a prompt text matching the audio to be translated based on the language to be translated, the initial recognition text and the corresponding recognition language; wherein the prompt text includes a conversion language matching the recognition language; a second processing module, used to obtain a reference vocabulary matching the initial recognition text from a candidate vocabulary library; a translation module, used to obtain a translation text corresponding to the audio to be translated based on the initial recognition text, the prompt text and the reference vocabulary.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the method mentioned in the above technical solution is implemented.

[0009] The beneficial effects of the present application are as follows: Different from the prior art, the speech translation method proposed in the present application obtains input audio from at least one end of the scene and determines the language to be translated of the input audio. The audio to be translated is recognized to obtain the corresponding initial recognition text, and the prompt text is determined according to the audio to be translated, the initial recognition text and its matching recognition language. In addition, a reference vocabulary matching the initial recognition text is obtained from a candidate vocabulary library, and the translation text is obtained by combining the initial recognition text, the prompt text and the reference vocabulary, thereby improving the accuracy of speech translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an implementation method of speech translation of the present application; Figure 2 yes Figure 1Step S102 corresponds to a flow chart of another implementation method; Figure 3 yes Figure 1 Step S102 corresponds to a flowchart of another implementation method; Figure 4 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method; Figure 5 yes Figure 4 Step S404 corresponds to a flow chart of another implementation method; Figure 6 yes Figure 1 Step S104 corresponds to a flow chart of another implementation method; Figure 7 It is a structural diagram of an implementation method of the speech translation system of the present application; Figure 8 It is a structural schematic diagram of an embodiment of the electronic device of the present application; Fig. 9 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0012] See also Figure 1 , Figure 1 1 is a flow chart of an implementation method of speech translation of the present application, the method comprising: S101: Obtain input audio from at least one end, determine the language to be translated of the input audio, use the input audio from each end as the audio to be translated, and determine the initial recognition text corresponding to the audio to be translated; wherein the initial recognition text matches at least one recognition language.

[0013] In one embodiment, the input audio collected by at least one end is obtained, and the language to be translated of the input audio is determined. The input audio of each end is used as the audio to be translated, and preliminary text recognition is performed on the audio to be translated to obtain the initial recognition text corresponding to the audio to be translated.

[0014] Specifically, after taking the input audio of each end as the audio to be translated, the audio to be translated is encoded to obtain the corresponding target audio code, the target audio code is decoded to obtain the initial recognition text corresponding to the audio to be translated, and at least one recognition language corresponding to the initial recognition text is determined.

[0015] In a specific application scenario, the above-mentioned speech translation method is applied to a multi-person conference scenario, and multiple microphone collection terminals are used to collect input audio corresponding to the target object in the conference scenario, and multiple languages ​​to be translated corresponding to the input audio are determined. For example, the above-mentioned target object is a participant in the conference, and when the expression languages ​​of the multiple participants include Chinese, English and Korean, the multiple languages ​​to be translated are determined to be Chinese, English and Korean.

[0016] S102: Based on the language to be translated, the initial recognition text and its corresponding recognition language, obtain a prompt text matching the audio to be translated; wherein the prompt text includes a conversion language matching the recognition language.

[0017] In one embodiment, a prompt text is generated according to the language to be translated, the initial recognition text and its corresponding recognition language, wherein the prompt text includes a conversion language determined from the language to be translated and different from the recognition language, and the prompt text is used to prompt the audio to be translated to be translated into a translation text matching the corresponding conversion language.

[0018] In a specific application scenario, the corresponding conversion language is determined according to the expression language of the target object at either end. For example, when the target object speaks Chinese, the corresponding conversion language is determined to be Chinese, so that the translation text obtained by subsequent translation is a Chinese text that the target object can understand.

[0019] In another embodiment, a conversion language different from the recognition language is obtained from the language to be translated, and the conversion language may be specified by a relevant target object or a technician.

[0020] S103: Acquire reference words matching the initial recognition text from the candidate word library.

[0021] In one embodiment, in response to the initial recognition text including multiple recognition characters, each candidate word in the candidate word library includes multiple reference characters, and then according to the matching degree between the recognition characters and the corresponding reference characters, the reference words related to the initial recognition text are screened and obtained. By selecting reference words from the candidate word library to provide references for subsequent translation processes, the accuracy of text translation in different fields is improved.

[0022] Specifically, for the initial recognition text and each candidate word, the number of identical characters in the corresponding recognition characters and the corresponding reference characters is obtained, and the candidate word corresponding to the maximum number is used as the reference word.

[0023] S104: Based on the initial recognition text, the prompt text and the reference vocabulary, a translation text corresponding to the audio to be translated is obtained.

[0024] In one implementation, the audio to be translated is translated in combination with the acquired initial recognition text, prompt text, and reference vocabulary to obtain a corresponding translation text.

[0025] Specifically, the acquired initial recognition text, prompt text and reference vocabulary are input into the trained audio translation model, and the audio translation model is used to translate the audio to be translated to obtain the corresponding translation text. The specific structure of the above audio translation model can refer to the existing neural network model structure.

[0026] The speech translation method proposed in the present application obtains input audio from at least one end of the scene and determines the language to be translated of the input audio. The audio to be translated is recognized to obtain the corresponding initial recognition text, and the prompt text is determined according to the audio to be translated, the initial recognition text and the matching recognition language. In addition, a reference vocabulary matching the initial recognition text is obtained from a candidate word library, and the translation text is obtained by combining the initial recognition text, the prompt text and the reference vocabulary, thereby improving the accuracy of speech translation.

[0027] See also Figure 2 , Figure 2 yes Figure 1 Step S102 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S102 includes: S201: Based on the recognition language, divide the initial recognition text into multiple text segments; wherein each text segment is matched with the same recognition language, and adjacent text segments are matched with different recognition languages.

[0028] In one embodiment, the initial recognition text includes a plurality of continuous recognition characters, and the recognition characters in the initial recognition text are divided according to the recognition language to obtain a plurality of corresponding text segments, wherein the recognition characters in the same text segment are matched with the same recognition language, and adjacent text segments are matched with different recognition languages.

[0029] In a specific application scenario, the initial recognition text obtained is "Ladies and gentlemen, good morning everyone!", then it is determined that the initial recognition text matches the adjacent first text segment "Ladies and gentlemen, good morning everyone!" and the second text segment "Ladies and gentlemen, good morning everyone", and the recognition language matched by the first text segment is Chinese, and the recognition language matched by the second text segment is English.

[0030] S202: Based on all text segments and their matching recognition languages, determine the conversion language matched by each text segment.

[0031] In one embodiment, based on all text segments and their matched recognized languages, a conversion language different from the corresponding recognized language is determined from the languages ​​to be translated.

[0032] Specifically, the conversion language matching each text segment is determined according to the expression language of the target object at either end and the recognition language corresponding to each text segment. For example, when the target object speaks Chinese, the corresponding conversion language is determined to be Chinese for the text segment whose recognition language is English.

[0033] In another embodiment, a predetermined translation strategy is obtained, the translation strategy includes a conversion language determined from the languages ​​to be translated that is different from the corresponding recognition language, and the translation strategy is used to represent the translation between the recognition language and the conversion language. For example, the recognition language and the conversion language are Chinese and English, respectively. When the initial recognition text includes text segments corresponding to Chinese and English, respectively, it is determined that the conversion language corresponding to the Chinese text segment is English, and the conversion language corresponding to the English text segment is Chinese.

[0034] S203: Based on the text segments and the matching conversion languages, obtaining the prompt texts matching the respective audio segments in the audio to be translated; wherein the audio segments correspond to the text segments one by one.

[0035] In one embodiment, the audio to be translated is divided according to the obtained text segments to obtain audio segments corresponding to the text segments. For each audio segment, a prompt text matching the audio segment is generated according to the corresponding conversion language, and the prompt text is used to prompt the corresponding audio segment to be translated into a translation text matching the corresponding conversion language.

[0036] In a specific application scenario, when the recognition language of the text segment is Chinese and its corresponding conversion language is English, the generated prompt text is: "translate Chinese to English".

[0037] See also Figure 3 , Figure 3 yes Figure 1 Step S102 in FIG. 1 corresponds to a flowchart of another implementation method. Specifically, the implementation process of step S102 includes: S301: Acquire recognition characters matching different recognition languages ​​in the initial recognition text.

[0038] In one embodiment, the initial recognition text includes a plurality of continuous recognition characters, the recognition characters corresponding to each recognition language are determined, and the number of recognition characters corresponding to each recognition language is counted.

[0039] S302: Determine a translation strategy based on the number of matching recognition characters of each recognition language.

[0040] In one embodiment, a translation strategy is determined based on the number of recognition characters matched to each recognition language, wherein the translation strategy is used to determine the conversion language corresponding to the audio to be translated.

[0041] Specifically, according to the number of recognition characters corresponding to each recognition language, the proportion of the corresponding recognition language in the initial recognition text is determined. According to the proportion, the conversion text corresponding to the initial recognition text is determined.

[0042] In one implementation scenario, for the proportions corresponding to each recognized language, the maximum proportion is obtained. A preset ratio threshold is obtained, and when the maximum proportion is greater than the ratio threshold, the corresponding conversion language is determined according to the recognized language corresponding to the maximum proportion. Otherwise, the corresponding conversion language is determined for each recognized language. For example, when the recognized language corresponding to the maximum proportion is Chinese and the maximum proportion is greater than the ratio threshold, if the language spoken by the corresponding target object is English, English is used as the conversion language corresponding to the entire audio to be translated; or, when the recognized language corresponding to the maximum proportion is Chinese and the maximum proportion is less than the ratio threshold, if the language spoken by the corresponding target object is English, multiple audio segments corresponding to the audio to be translated are determined according to the recognized language, and the conversion language corresponding to each audio segment is determined to be English.

[0043] S303: Obtain prompt text based on the identified language and translation strategy.

[0044] In one implementation scenario, a prompt text is generated according to the recognized language and the translation strategy, wherein the prompt text includes a conversion language determined from the to-be-translated language and different from the recognized language, and the prompt text is used to prompt the to-be-translated audio to be translated into a translation text matching the corresponding conversion language.

[0045] The above solution helps to improve the efficiency of translation by determining the corresponding translation strategy according to the number of recognized characters.

[0046] In another embodiment, after the recognition characters matching different recognition languages ​​in the initial recognition text are obtained, the corresponding translation strategy is determined according to the recognition characters matching each recognition language.

[0047] Specifically, professional terms in the initial recognition text are recognized, and the recognition language corresponding to the professional terms is used as the corresponding conversion language.

[0048] In one implementation scenario, the corresponding professional noun is determined based on the special characters in the initial recognition text, and the recognition language corresponding to the professional noun is used as the corresponding conversion language so that the professional noun is not translated in the subsequent translation process, thereby ensuring the accuracy of the translation. For example, the quotation marks in the initial recognition text are recognized, and the part in the quotation marks is used as the professional noun.

[0049] See also Figure 4 , Figure 4 yes Figure 1 Step S103 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S103 includes: S401: Obtain target audio code corresponding to the audio to be translated.

[0050] In one implementation, a target audio code corresponding to the audio to be translated is obtained.

[0051] In one implementation scenario, in order to improve execution efficiency, the target audio code is obtained through the process of generating an initial recognition text. The specific acquisition process can refer to the corresponding implementation method above and will not be elaborated here.

[0052] S402: Obtain a reference code corresponding to each candidate word in the candidate word library.

[0053] In one implementation, the candidate word library includes a plurality of candidate words, and each candidate word is encoded respectively to obtain a corresponding reference code.

[0054] Specifically, a semantic coding model is obtained, and the candidate words are input into the semantic coding model to obtain corresponding reference coding. The specific structure of the reference coding model can refer to the existing neural network structure.

[0055] S403: Based on the target audio code and the reference code, obtain a similarity score between the audio to be translated and each candidate word.

[0056] In one embodiment, a similarity score between the audio to be translated and each candidate word is obtained based on the target audio code and the reference code corresponding to each candidate word.

[0057] Specifically, the cosine similarity between the target audio code and the reference code is calculated, and the cosine similarity is used as the similarity score between the audio to be translated and the corresponding candidate vocabulary.

[0058] S404: Determine a reference vocabulary from the candidate vocabulary based on the similarity score.

[0059] In one embodiment, at least one reference word is selected from all candidate words according to the similarity scores between the audio to be translated and each candidate word.

[0060] Specifically, the similarity score is compared with a preset first score threshold, and the candidate words corresponding to the similarity scores greater than the first score threshold are used as reference words. The first score threshold may be obtained by reverse deduction through multiple experiments, or may be obtained by estimation by relevant technical personnel.

[0061] In another embodiment, with respect to the similarity scores between the audio to be translated and each candidate word, the candidate word corresponding to the maximum similarity score is used as the reference word.

[0062] The above scheme determines reference vocabulary to provide a reference basis for the subsequent translation process of the audio to be translated, thereby improving the accuracy of the translation.

[0063] See also Figure 5 , Figure 5 yes Figure 4 Step S404 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S404 includes: S501: Determine at least one first screening word from all candidate words based on the similarity score.

[0064] In one embodiment, the candidate words corresponding to the similarity scores greater than the score threshold are used as the first screening words. Alternatively, for the similarity scores between the audio to be translated and each candidate word, the candidate word corresponding to the maximum similarity score is used as the first screening word.

[0065] S502: Obtaining a matching score between the initial recognition text and each candidate word, and determining at least one second screening word from all the candidate words based on the matching score.

[0066] In one embodiment, a corresponding matching score is obtained based on the initial recognition text and each candidate word, and at least one second screening word is determined from all the candidate words based on the matching score.

[0067] Specifically, the edit distance between the initial recognition text and each candidate word is obtained, and the corresponding matching score is determined based on the edit distance. The edit distance is inversely proportional to the corresponding matching score, that is, the larger the edit distance, the smaller the corresponding matching score. A pre-set second score threshold is obtained, and the candidate words corresponding to the matching scores greater than the second score threshold are used as the second screening words. Alternatively, for the matching scores between the audio to be translated and each candidate word, the candidate word corresponding to the maximum matching score is used as the second screening word. Among them, the above-mentioned second score threshold can be obtained by reverse deduction through multiple experiments, or it can also be obtained by relevant technical personnel through estimation.

[0068] In another embodiment, the initially recognized text is fuzzy matched with each candidate word to obtain a corresponding matching score. At least one second screening word is determined from all the candidate words according to the matching score.

[0069] In addition, it should be noted that the execution order of the above steps S501 and S502 may also be other, for example, executing step S502 first and then executing step S501; or, executing step S501 and step S502 at the same time.

[0070] S503: Determine a reference vocabulary based on the first screening vocabulary and the second screening vocabulary.

[0071] In one implementation, both the first screening vocabulary and the second screening vocabulary are used as reference vocabulary.

[0072] In another embodiment, in response to obtaining a plurality of first screening words and / or a plurality of second screening words, identical words in the plurality of first screening words and the plurality of second screening words are obtained, and the identical words are used as reference words.

[0073] The above scheme further improves the accuracy of determining the reference vocabulary by combining the similarity score and the matching score, and helps to improve the accuracy of subsequent translation.

[0074] See also Figure 6 , Figure 6 yes Figure 1 Step S104 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S104 includes: S601: The target audio code corresponding to the audio to be translated is used as the first prompt information.

[0075] In one implementation, the target audio code corresponding to the audio to be translated is used as the first prompt information. The specific process of obtaining the target audio code may refer to the above corresponding implementation.

[0076] S602: Obtain second prompt information based on encoding the prompt text.

[0077] In one implementation, the prompt text obtained in the above corresponding implementation is encoded to obtain the second prompt information.

[0078] Specifically, the prompt text is input into the semantic coding model to encode the prompt text using the semantic coding model, and the features obtained after encoding are used as the second prompt information. The specific implementation process can refer to the above corresponding implementation method.

[0079] S603: In response to the candidate word library including a reference translation word matching each candidate word, obtaining a reference translation word corresponding to the reference word, and obtaining third prompt information based on the reference word and the reference translation word matching the reference word; wherein the third prompt information is used to characterize a conversion relationship between the reference word and the reference translation word.

[0080] In one embodiment, the candidate word library includes candidate words matching multiple languages, and each candidate word is matched with a reference translation word corresponding to another language. For each reference word, a reference translation phrase is constructed based on the reference word and the corresponding reference translation word. A coding feature corresponding to the reference translation phrase is obtained, and the coding feature is used as the third prompt information.

[0081] Specifically, for each reference word obtained, a reference translation phrase is generated according to the reference word, the reference translation word corresponding to the reference word, and the conversion relationship between the two. The reference translation phrase is encoded using a semantic encoding model to obtain the third prompt information.

[0082] In a specific application scenario, when the reference word determined is "good morning", the corresponding reference translation word obtained is "good morning", then the corresponding reference translation phrase generated is: "good morning" translate to "good morning". By encoding it using the semantic encoding model, the third prompt information is obtained.

[0083] In another embodiment, in response to each candidate word in the candidate word library being matched with reference translation words corresponding to other multiple languages, a reference translation phrase is generated according to the corresponding conversion language, the reference word, the reference translation word corresponding to the reference word, and the conversion relationship between the two. The reference translation phrase is encoded using a semantic encoding model to obtain third prompt information.

[0084] In a specific application scenario, when the determined reference word is "good morning" and the conversion language matched by the audio to be translated is English, the corresponding reference translation word "good morning" is obtained, and the corresponding third prompt information is generated as: "Good morning" translate to "good morning".

[0085] S604: Inputting the first prompt information, the second prompt information, and the third prompt information into the intelligent analysis model to obtain a translation text output by the intelligent analysis model.

[0086] In a real-time manner, an intelligent analysis model is used to analyze the first prompt information, the second prompt information, and the third prompt information, thereby obtaining a translation text output by the intelligent analysis model.

[0087] Specifically, the first prompt information, the second prompt information, and the third prompt information are sequentially spliced ​​as a translation task text, and the translation task text is used to prompt the intelligent analysis model to translate the audio to be translated into a translation text matching the conversion language. The intelligent analysis model is used to analyze the above translation task text and output the corresponding translation text.

[0088] In one implementation scenario, the intelligent analysis model is a large language model with superior data analysis capabilities. The generated translation task text is input into the intelligent analysis model so that the intelligent analysis model interprets it in detail and outputs a translation text that matches the audio to be translated.

[0089] In a specific application scenario, the above-mentioned large language model may include but is not limited to deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory networks (LSTM), and generative pre-trained Transformer models, etc. No specific restrictions are made on the specific structure and specific deployment of the large language model. In addition, it should be noted that the specific structure and specific deployment of the intelligent analysis model mentioned in other implementations of this application can refer to this implementation.

[0090] In another embodiment, a preset translation task template is obtained, and the obtained first prompt information, second prompt information, and third prompt information are sequentially filled into corresponding positions in the translation task template to obtain a translation task text. The translation task text is input into the intelligent analysis model to obtain an output translation text.

[0091] In a specific application scenario, the pre-set translation task template is: "You are a professional translator. Please translate the audio to be translated according to the first prompt information, the second prompt information and the third prompt information, and output the corresponding translation text."

[0092] The above scheme uses an intelligent analysis model combined with a variety of prompt information to generate a translation text corresponding to the audio to be translated with high accuracy, thereby improving the effect of audio translation.

[0093] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of an embodiment of the speech translation system of the present application. Specifically, the speech translation system comprises a recognition module 10, a first processing module 20, a second processing module 30 and a translation module 40 which are coupled to each other.

[0094] Specifically, the recognition module 10 is used to obtain input audio from at least one end, determine the language to be translated of the input audio, use the input audio from each end as the audio to be translated, and determine the initial recognition text corresponding to the audio to be translated; wherein the initial recognition text matches at least one recognition language.

[0095] The first processing module 20 is used to obtain a prompt text matching the audio to be translated based on the language to be translated, the initial recognition text and its corresponding recognition language; wherein the prompt text includes a conversion language matching the recognition language.

[0096] The second processing module 30 is used to obtain reference words matching the initial recognition text from the candidate word library.

[0097] The translation module 40 is used to obtain a translation text corresponding to the audio to be translated based on the initial recognition text, the prompt text and the reference vocabulary.

[0098] In one embodiment, the initial recognition text matches multiple recognition languages. The first processing module 20 obtains prompt text matching the audio to be translated based on the language to be translated, the initial recognition text and its corresponding recognition language, including: dividing the initial recognition text into multiple text segments based on the recognition language; wherein each text segment matches the same recognition language, and adjacent text segments match different recognition languages; based on all text segments and their matching recognition languages, determining the conversion language matched by each text segment; based on the text segments and their matching conversion languages, obtaining prompt text matching each audio segment in the audio to be translated; wherein the audio segments correspond one to one to the text segments.

[0099] In one embodiment, the initial recognition text matches multiple recognition languages. The first processing module 20 obtains a prompt text that matches the audio to be translated based on the language to be translated, the initial recognition text and its corresponding recognition language, including: obtaining recognition characters that match different recognition languages ​​in the initial recognition text; determining a translation strategy based on the number of recognition characters that match each recognition language; and obtaining a prompt text based on the recognition language and the translation strategy.

[0100] In one embodiment, the candidate word library includes multiple candidate words, and the second processing module 30 obtains a reference word that matches the initial recognition text from the candidate word library, including: obtaining a target audio code corresponding to the audio to be translated; and obtaining a reference code corresponding to each candidate word in the candidate word library; based on the target audio code and the reference code, obtaining a similarity score between the audio to be translated and each candidate word; based on the similarity score, determining a reference word from all candidate words.

[0101] In one embodiment, the second processing module 30 determines a reference vocabulary from all candidate vocabulary based on the similarity score, including: determining at least one first screening vocabulary from all candidate vocabulary based on the similarity score; and obtaining a matching score between the initial recognition text and each candidate vocabulary, and determining at least one second screening vocabulary from all candidate vocabulary based on the matching score; determining a reference vocabulary based on the first screening vocabulary and the second screening vocabulary.

[0102] In one embodiment, the translation module 40 obtains the translation text corresponding to the audio to be translated based on the initial recognition text, the prompt text and the reference vocabulary, including: encoding the target audio corresponding to the audio to be translated as the first prompt information; and, based on encoding the prompt text, obtaining the second prompt information; and, in response to the reference translation vocabulary matching each candidate vocabulary included in the candidate vocabulary library, obtaining the reference translation vocabulary corresponding to the reference vocabulary, and obtaining the third prompt information based on the reference vocabulary and its matching reference translation vocabulary; wherein the third prompt information is used to characterize the conversion relationship between the reference vocabulary and the reference translation vocabulary; based on the first prompt information, the second prompt information and the third prompt information are input into the intelligent analysis model to obtain the translation text output by the intelligent analysis model.

[0103] In one embodiment, the translation module 40 obtains the third prompt information based on the reference vocabulary and its matching reference translation vocabulary, including: for each reference vocabulary, constructing a reference translation phrase based on the reference vocabulary and the corresponding reference translation vocabulary; obtaining the encoding feature corresponding to the reference translation phrase, and using the encoding feature as the third prompt information.

[0104] See also Figure 8 , Figure 8: is a structural diagram of an embodiment of an electronic device of the present application. The electronic device comprises: a memory 50 and a processor 60 coupled to each other. The memory 50 stores program instructions, and the processor 60 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 60 can also be called a CPU (Center Processing Unit). The processor 60 may be an integrated circuit chip with signal processing capabilities. The processor 60 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. In addition, the processor 60 may be implemented by an integrated circuit chip.

[0105] See also Fig. 9 , Fig. 9 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 70 stores program instructions 80 that can be executed by a processor. When the program instructions 80 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0106] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0110] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech translation method, characterized in that: include: Acquire input audio from at least one end, determine the language to be translated of the input audio, use the input audio from each end as the audio to be translated, and determine the initial recognition text corresponding to the audio to be translated; wherein the initial recognition text matches at least one recognition language; Based on the language to be translated, the initial recognition text and the corresponding recognition language, obtaining a prompt text matching the audio to be translated; wherein the prompt text includes a conversion language matching the recognition language; Acquire a reference vocabulary matching the initial recognition text from a candidate vocabulary library; Based on the initial recognition text, the prompt text and the reference vocabulary, a translation text corresponding to the audio to be translated is obtained.

2. The method according to claim 1, characterized in that The initial recognition text matches a plurality of the recognition languages, and obtaining a prompt text matching the audio to be translated based on the language to be translated, the initial recognition text and the corresponding recognition language, comprises: Based on the recognition language, the initial recognition text is divided into a plurality of text segments; wherein each of the text segments is matched with the same recognition language, and adjacent text segments are matched with different recognition languages; Based on all the text segments and the identified languages ​​matched thereto, determining the conversion language matched by each of the text segments; Based on the text segments and the matching conversion languages, the prompt texts matched by the respective audio segments in the audio to be translated are obtained; wherein the audio segments correspond one to one with the text segments.

3. The method according to claim 1, characterized in that The initial recognition text matches a plurality of the recognition languages, and obtaining a prompt text matching the audio to be translated based on the language to be translated, the initial recognition text and the corresponding recognition language, comprises: Acquire recognition characters matching different recognition languages ​​in the initial recognition text; Determining a translation strategy based on the number of the recognized characters matched by each of the recognized languages; The prompt text is obtained based on the recognized language and the translation strategy.

4. The method according to claim 1, characterized in that: The candidate word library includes a plurality of candidate words, and the step of obtaining a reference word matching the initial recognition text from the candidate word library includes: Obtaining a target audio code corresponding to the audio to be translated; and Obtaining a reference code corresponding to each of the candidate words in the candidate word library; Based on the target audio code and the reference code, obtaining a similarity score between the audio to be translated and each of the candidate words; The reference vocabulary is determined from all the candidate vocabulary based on the similarity scores.

5. The method according to claim 4, characterized in that The step of determining the reference vocabulary from all the candidate vocabulary based on the similarity score comprises: Based on the similarity score, determining at least one first screening vocabulary from all the candidate vocabulary; and, Obtaining a matching score between the initial recognition text and each of the candidate words, and determining at least one second screening word from all the candidate words based on the matching score; The reference vocabulary is determined based on the first screening vocabulary and the second screening vocabulary.

6. The method according to claim 1, characterized in that The step of obtaining a translation text corresponding to the audio to be translated based on the initial recognition text, the prompt text and the reference vocabulary includes: Using the target audio code corresponding to the audio to be translated as the first prompt information; and Based on encoding the prompt text, obtaining second prompt information; and, In response to the candidate word library including a reference translation word matching each candidate word, obtaining a reference translation word corresponding to the reference word, and obtaining third prompt information based on the reference word and the reference translation word matching the reference word; wherein the third prompt information is used to represent the conversion relationship between the reference word and the reference translation word; The first prompt information, the second prompt information and the third prompt information are input into an intelligent analysis model to obtain the translation text output by the intelligent analysis model.

7. The method according to claim 6, characterized in that The acquiring third prompt information based on the reference vocabulary and the reference translation vocabulary matched thereto includes: For each of the reference words, construct a reference translation phrase based on the reference word and the corresponding reference translation word; Obtain a coding feature corresponding to the reference translation phrase, and use the coding feature as the third prompt information.

8. A speech translation system, characterized in that: include: A recognition module, used to obtain input audio from at least one end, determine the language to be translated of the input audio, use the input audio from each end as the audio to be translated, and determine the initial recognition text corresponding to the audio to be translated; wherein the initial recognition text matches at least one recognition language; A first processing module is used to obtain a prompt text matching the audio to be translated based on the language to be translated, the initial recognition text and the corresponding recognition language; wherein the prompt text includes a conversion language matching the recognition language; A second processing module, configured to obtain a reference vocabulary matching the initial recognition text from a candidate vocabulary library; The translation module is used to obtain a translation text corresponding to the audio to be translated based on the initial recognition text, the prompt text and the reference vocabulary.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images