Simultaneous interpretation method and device, equipment, storage medium and program product

Abstract

The invention provides a simultaneous interpretation translation method and device, equipment, a storage medium and a program product, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a preset language pair containing a first language and a second language, and determining a target language judgment strategy from a plurality of language judgment strategies; and receiving the real-time audio stream to judge the source language and the target language according to the target language judgment strategy so as to execute translation operation on the real-time audio stream and output a translation result. According to the method, the optimal language judgment strategy is dynamically matched and adopted for the types of different language pairs, the defects that in the prior art, the accuracy rate is low and misjudgment is prone to occurring when different translation scenes are faced are overcome, the accuracy and robustness of automatic language judgment are remarkably improved, and therefore the correctness of the translation direction and the final translation quality are ensured, and the translation efficiency is improved. And the communication efficiency and experience of the user in bilingual real-time communication are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a simultaneous interpretation method, apparatus, device, storage medium, and program product. Background Technology

[0002] Simultaneous interpretation equipment is playing an increasingly important role in scenarios such as international conferences, business negotiations, and international travel. These scenarios often involve the alternating use of two or more languages.

[0003] To meet this need, existing technologies typically employ an artificial intelligence model-based approach to achieve end-to-end translation, including speech recognition, text translation, and speech synthesis. However, these methods generally require users to manually and unidirectionally specify the current source and target languages ​​before they begin speaking, such as "from Chinese to English" or "from English to Chinese".

[0004] However, this pre-set one-way translation path has significant limitations in real-world scenarios requiring frequent switching between two languages. Users must interrupt the conversation and reset their devices each time they change speakers or language, severely impacting the fluency and naturalness of communication. Some automatic language recognition schemes proposed to address this issue struggle to guarantee accuracy in complex real-world dialogue environments, frequently resulting in misjudgments. These misjudgments directly lead the system to activate the wrong translation direction, resulting in completely incorrect translations and compromising the effectiveness and reliability of communication. Summary of the Invention

[0005] This invention provides a simultaneous interpreting method, apparatus, device, storage medium, and program product to solve the problem in the prior art that simultaneous interpreting requires frequent manual switching between different languages, which seriously affects the fluency of communication; and the technical problem that in automatic recognition mode, due to the difficulty in ensuring the accuracy of language determination in the face of complex real dialogue environments, it is easy to lead to errors in translation direction and inaccurate output results.

[0006] This invention provides a simultaneous interpreting method, comprising: Obtain a preset language pair containing a first language and a second language, and determine the target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies; Receive a real-time audio stream and determine the source language of the real-time audio stream according to the target language determination strategy; when the source language is determined to be the first language, determine the second language as the target language; otherwise, when the source language is determined to be the second language, determine the first language as the target language. Based on the determined source language and target language, a translation operation is performed on the real-time audio stream, and the translation result is output.

[0007] According to the simultaneous interpretation method provided by the present invention, the language determination strategy includes at least a first language determination strategy and a second language determination strategy; The first language determination strategy includes: performing speech recognition on the audio stream to obtain recognized text, and determining the source language of the audio stream based on the text content of the recognized text; The second language determination strategy includes: extracting audio features of the audio stream to determine the source language of the audio stream based on the audio features of the audio stream.

[0008] According to a simultaneous interpretation method provided by the present invention, when the target language determination strategy corresponding to the translation type of the language pair is determined from at least two different language determination strategies to be the first language determination strategy, the step of determining the source language of the real-time audio stream according to the target language determination strategy includes: Determine whether the identified text is a mixed language text containing the first language and the second language; If so, the mixed language text is split according to the preset context information to obtain a first text part belonging to the first language and a second text part belonging to the second language; based on the first text part and the second text part, the source language of the time-audio stream is determined; If not, the language corresponding to the identified text is determined as the source language of the audio stream.

[0009] According to a simultaneous interpretation method provided by the present invention, determining the source language of the temporal audio stream based on the first text portion and the second text portion includes: The source language of the real-time audio stream is determined based on the proportion of the text length of the first text portion and the second text portion in the mixed language text; or, The matrix text constituting the backbone structure of the mixed language text is determined, and the language corresponding to the matrix text is determined as the source language of the real-time audio stream.

[0010] According to a simultaneous interpretation method provided by the present invention, before determining whether the identified text is a mixed language text containing the first language and the second language, or when it is impossible to split the mixed language text according to preset context information, it is determined whether a preset short sentence response condition is met. If the short sentence response conditions are met, the language that is the same as the target language of the previous sentence translation will be determined as the source language of the real-time audio stream; The preset short sentence response conditions include that the length of the recognized text is less than a preset length threshold, and the time interval between the start time of the speech corresponding to the recognized text and the end time of the previous sentence translation is less than a preset time threshold.

[0011] According to a simultaneous interpretation method provided by the present invention, when the target language determination strategy corresponding to the translation type of the language pair is determined to be the second language determination strategy from at least two different language determination strategies, the step of determining the source language of the real-time audio stream according to the target language determination strategy includes: Speech activity detection is performed on the real-time audio stream to obtain valid audio segments containing human voices; At the beginning of the valid audio segment, an initial language determination is performed, and an initial language determination result is generated; Monitor the duration of the valid audio segment; If the effective audio segment ends before the duration reaches a preset threshold, the effective audio segment is determined to be a short effective audio segment; at the end of the short effective audio segment, an end language determination is performed to determine the source language of the real-time audio stream; If the duration of the valid audio segment reaches the preset threshold, the valid audio segment is determined to be a long valid audio segment; when the long valid audio segment reaches the preset threshold, an intermediate language determination is performed to calibrate the initial language determination result; at the end of the long valid audio segment, an end language determination is performed to determine the source language of the real-time audio stream.

[0012] According to a simultaneous interpretation method provided by the present invention, the step of determining the source language of the real-time audio stream by performing an end-of-stream language determination includes: The cumulative first effective pronunciation duration of the first language and the cumulative second effective pronunciation duration of the second language are calculated within the entire effective audio segment at the end time. Based on the first effective pronunciation duration and the second effective pronunciation duration, determine the duration weights of the first language and the second language; The language corresponding to the maximum duration weight is determined as the source language of the real-time audio stream.

[0013] According to the simultaneous interpretation method provided by the present invention, after generating the initial language determination result, it further includes: The initial language determination result is sent to the user terminal in real time; The user terminal is controlled to display the language identifier corresponding to the initial language determination result on the interface.

[0014] According to a simultaneous interpretation method provided by the present invention, the receiving of real-time audio stream includes: Establish a long TCP connection with the client, and create a bidirectional stream on the long TCP connection for this simultaneous interpretation session; The real-time audio stream sent by the client is received through the bidirectional stream; Accordingly, the output translation result includes: The source language determination result and the translation result are pushed to the client in real time through the bidirectional stream.

[0015] The present invention also provides a simultaneous interpretation device, comprising the following modules: The strategy selection module is used to obtain a preset language pair containing a first language and a second language, and determine the target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies. The language determination module receives a real-time audio stream and determines the source language of the real-time audio stream according to the target language determination strategy; when the source language is determined to be the first language, the second language is determined to be the target language; otherwise, when the source language is determined to be the second language, the first language is determined to be the target language. The translation operation module is used to perform a translation operation on the real-time audio stream based on the determined source language and target language, and output the translation result.

[0016] The present invention also provides a simultaneous interpretation device, comprising: The audio acquisition module is used to acquire the user's real-time audio stream; The communication module is used to establish long-lived connection sessions with the cloud server; The processor is configured to perform the following operations: send a language pair containing a first language and a second language, along with the real-time audio stream, to the cloud server via the communication module; and receive the translation result of the real-time audio stream returned by the cloud server. The cloud server generates the translation result based on any of the simultaneous interpretation methods described above.

[0017] The simultaneous interpretation device provided by the present invention further includes: The output module is used to play the audio corresponding to the translation result and / or to update and display the source language determination result in real time on the display interface.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the simultaneous interpretation method as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the simultaneous interpretation method as described above.

[0020] The simultaneous interpretation method, apparatus, equipment, storage medium, and program products provided by this invention dynamically match and adopt the optimal language determination strategy for different language pairs, solving the shortcomings of existing technologies in terms of low accuracy and easy misjudgment when facing different translation scenarios. This significantly improves the accuracy and robustness of automatic language determination, thereby ensuring the correctness of the translation direction and the final translation quality, and greatly improving the communication efficiency and experience of users in real-time bilingual communication. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the system processing flow for simultaneous interpretation in existing technologies.

[0023] Figure 2 This is a flowchart illustrating the simultaneous interpretation method provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the system processing flow for simultaneous interpretation provided by the present invention.

[0025] Figure 4 This is one of the timing diagrams for simultaneous interpretation provided by the present invention.

[0026] Figure 5 This is one of the flowcharts for source language determination provided by the present invention.

[0027] Figure 6 This is the second flowchart illustrating the source language determination process provided by this invention.

[0028] Figure 7 This is the second timing diagram of simultaneous interpretation provided by the present invention.

[0029] Figure 8 This is the third flowchart of the source language determination provided by the present invention.

[0030] Figure 9 This is the fourth flowchart of the source language determination process provided by the present invention.

[0031] Figure 10 This is a schematic diagram of the simultaneous interpretation device provided by the present invention.

[0032] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0035] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0036] Currently, the general process for simultaneous interpretation based on AI capability cascading is to first select the language direction, then record the audio, and perform real-time speech recognition, text translation, and speech synthesis. The specific implementation method is to cascade AI capabilities such as speech recognition, text translation, and speech synthesis. For example, if the user selects the Chinese-to-Korean language direction, then Chinese recognition, Chinese-to-Korean translation, and Korean synthesized broadcasting will be performed.

[0037] Figure 1 This is a schematic diagram of the system processing flow for simultaneous interpretation in existing technologies, such as... Figure 1The diagram illustrates the information interaction between the client and server. The specific implementation process includes: First, the client sends the collected user voice as an audio stream to the server's recognition module. The recognition module processes the audio stream, converts the speech into text, and returns the recognition result to the client in real time. Simultaneously, when a complete text clause (i.e., a short paragraph) is recognized, it passes the recognition result to the next-level translation module. The translation module receives the recognized text clause, translates it according to a preset direction, and also returns the translation result to the client (e.g., displaying the translation on the screen). It then passes the translated text clause to the synthesis module. Finally, the synthesis module receives the translated text, converts it into speech, and sends the final synthesized result back to the client for playback. This entire process automatically loops to handle continuous voice input.

[0038] Existing technical solutions for simultaneous interpretation scenarios involving several hours of conversation require users to first set the language direction before recording can begin. In an international conference, if there is only one device and two people (one Chinese and one Korean) are communicating but do not want to frequently start conversations, they might set the language direction to Chinese-to-Korean, at which point they can only speak Chinese. If they want to speak Korean for recognition, they have to reset the language direction, which requires starting a new conversation. Similarly, if they want to speak Chinese again, they have to repeat the above setup process. In scenarios where automatic recognition is used between two languages, this technical solution causes great inconvenience and confusion for both users.

[0039] The simultaneous interpretation method provided by this invention only requires setting the language pair once at the beginning of a long simultaneous interpretation session, achieving seamless switching between the two languages, automatic recognition, and then performing translation and synthesis in the corresponding direction. For example, in a Chinese-Korean scenario, the user can speak either Chinese or Korean. When the recognition stage determines that it is Chinese, the Chinese to Korean translation is performed, and the Korean is synthesized; when the recognition stage determines that it is Korean, the Korean to Chinese translation is performed, and the Chinese is synthesized.

[0040] It is particularly noteworthy that this invention also dynamically matches and adopts the optimal language determination strategy based on the type of different language pairs, solving the shortcomings of existing technologies in terms of low accuracy and easy misjudgment when facing different translation scenarios. It can significantly improve the accuracy and robustness of automatic language determination, thereby ensuring the correctness of the translation direction and the final translation quality, and greatly improving the communication efficiency and experience of users in bilingual real-time communication.

[0041] The following is combined with Figures 2-10This invention describes the simultaneous interpretation method, apparatus, device, storage medium, and program product provided by the present invention. The execution entity of the simultaneous interpretation method can be a server, a terminal device, or a processor or control system in a system composed of a server and a terminal device; the server can be a cloud server. In the following embodiments, a control system deployed in an electronic device (such as a translator, translation headset, or smartphone) will be described in detail as the execution entity.

[0042] Figure 2 This is a flowchart illustrating the simultaneous interpretation method provided by the present invention, as shown below. Figure 2 As shown, including but not limited to the following steps: Step 11: Obtain a preset language pair containing a first language and a second language, and determine the target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies.

[0043] In a specific implementation of the present invention, a language pair refers to a set of languages ​​that the user expects to translate in two directions in real time. For example, the user can select Chinese and English as a language pair through the graphical user interface (GUI) of the translator for Chinese-English translation. The language pair then includes a first language (such as Chinese) and a second language (such as English).

[0044] After acquiring the language pairs, the control system first needs to determine the translation type of the language pairs. The translation type can be seen as a classification of the language pairs, and the classification can be based on various factors. For example, whether there is a mature mixed speech recognition model for the language pairs that can simultaneously recognize two languages, or whether the language pairs have a frequent code-switching (i.e., mixed speech) habit in daily communication. For example, since the Chinese-English mixed recognition technology is relatively mature, the control system can classify Chinese-English translation language pairs as the first type; while other language pairs such as "Chinese-Korean translation" may be classified as the second type.

[0045] Accordingly, the control system pre-stores at least two different language determination strategies. Each language determination strategy is an algorithm or rule set optimized for a specific type of translation, and essentially serves as a logical method for determining the language of the input audio. For example, the control system can load a strategy A that primarily relies on analyzing the text content obtained after audio recognition, and another strategy B that primarily relies on analyzing the streaming acoustic features of the audio.

[0046] As an alternative implementation, the process of determining the target language determination strategy can be a rule-based matching process. After determining the translation type of the current language pair (for example, determining Chinese-English translation as the first type), the control system will select the target language determination strategy that uniquely corresponds to the determination type from the pre-stored language determination strategy set (for example, selecting strategy A mentioned above).

[0047] Step 12: Receive the real-time audio stream and determine the source language of the real-time audio stream according to the target language determination strategy.

[0048] Specifically, when the source language is determined to be the first language, the second language is determined to be the target language; conversely, when the source language is determined to be the second language, the first language is determined to be the target language.

[0049] After the user begins to speak, the control system receives a continuous real-time audio stream through the device's audio acquisition unit, such as a microphone array. This real-time audio stream is the raw data stream containing the user's voice.

[0050] Next, the control system applies the target language determination strategy determined in the previous step to process the real-time audio stream to determine its source language. Assuming the target language determination strategy is strategy A, the determination process here executes the algorithm logic defined by strategy A, and its output clearly indicates which language in the preset language pair the current speech belongs to. For example, if the user is speaking Chinese, the determined source language is Chinese.

[0051] After determining the source language, the process of determining the target language is a simple logical complementarity. Taking the above Chinese-English translation as an example, when the source language is determined to be Chinese (the first language), the control system automatically determines English (the second language) as the target language; conversely, if the source language is determined to be English, then Chinese is automatically determined as the target language.

[0052] Step 13: Based on the determined source language and target language, perform a translation operation on the real-time audio stream and output the translation result.

[0053] Once the translation direction, consisting of the source language and the target language, is determined, the control system can execute subsequent translation operations.

[0054] The translation operation is a cascaded process consisting of multiple sub-steps, which may include but are not limited to: performing full automatic speech recognition (ASR) on the real-time audio stream to obtain the source language text, invoking a neural machine translation (NMT) engine to translate the source language text into the target language text, and then invoking a text-to-speech (TTS) engine to synthesize the target language text into the target language audio.

[0055] Finally, the control system outputs the translation result obtained from the translation operation to the client, such as a translator. The translation result can be a set containing various information, for example, it can include the translated target language text for display on the screen of the client, or it can include the synthesized target language audio for playback through the speaker of the client, and it can also include the direction information of this translation, such as Chinese to English, etc.

[0056] Figure 3 It is a schematic diagram of the system processing flow of simultaneous interpretation translation provided by the present invention. As Figure 3 shown, assume that the user has obtained a preset language pair on the client, such as a translator. For example, the first language A is Chinese and the second language B is English, and the control system has determined the target language determination strategy according to the type of this language pair.

[0057] Now, the user starts to speak. Figure 3 The system processing flow shown starts.

[0058] First, receive the audio stream and determine the source language. The client collects the user's speech to form a real-time audio stream and sends it to the server, where the LOOP part in Figure 3 is run. The recognition ability module of the server receives this real-time audio stream and processes this real-time audio stream by applying the determined target language determination strategy. It not only performs speech recognition but also corely executes language determination.

[0059] Assume that the user says "Hello" in Chinese. The recognition ability module determines that the source language is Chinese (language A). At the same time, it returns the recognized text result ("Hello") and the language determination result ("Chinese") to the client for display on the screen of the client.

[0060] Then, determine the translation direction and perform the translation. After determining that the source language is Language A, the process enters the judgment phase, that is, it is necessary to determine the translation direction. Since the source language is Language A, the control system determines the translation direction as from Language A to Language B (i.e., Chinese to English), and calls the language translation module with the translation ability from Language A to Language B. After receiving the text result "Hello", the language translation module starts to perform the translation operation and obtains the translation result "Hello". At the same time, it returns the translation result ("Hello") and the translation direction ("Chinese -> English") to the client.

[0061] Finally, the control system will determine the synthesized language and the output result. The translated text "Hello" enters the subsequent process. The translation direction is from Language A to Language B, and the target language is Language B (English), so the control system determines the synthesized language as English. At this time, the process will call the synthesis ability module to synthesize the text "Hello" into English speech.

[0062] Finally, the synthesis ability module returns the synthesis result (the audio of "Hello") and the language information ("English") to the client for playback. Thus, a complete "Chinese to English" translation operation is completed and the translation result is output.

[0063] In another embodiment, if the next user says "How are you?" in English, the entire process will be executed again, including: the recognition ability module will determine that the source language is English (Language B); when determining the translation direction, the process determines that the translation direction is from Language B to Language A (i.e., English to Chinese); call the translation ability module with the translation direction from Language B to Language A to obtain the translation result "How are you?". When determining the synthesized language, the synthesized language is determined as Chinese, and the synthesis ability module synthesizes Chinese speech and then returns it to the client for playback.

[0064] The simultaneous interpretation translation method provided by the present invention dynamically matches and adopts the optimal language determination strategy for different types of language pairs, solves the defects of low accuracy and easy misjudgment of the prior art in the face of different translation scenarios, significantly improves the accuracy and robustness of automatic language determination, thereby ensuring the correctness of the translation direction and the final translation quality, and greatly improving the communication efficiency and experience of users in real-time bilingual communication.

[0065] As an optional embodiment, the language determination strategy at least includes a first language determination strategy and a second language determination strategy.

[0066] Among them, the first language determination strategy may include: performing speech recognition on the audio stream to obtain the recognition text, and determining the source language corresponding to the audio stream based on the text content of the recognition text.

[0067] Specifically, the execution process of the first language determination strategy generally includes: First, perform real-time speech recognition on the audio stream to obtain preliminary or final recognition text. Then, conduct in-depth text content analysis on the generated recognition text. By analyzing features such as the vocabulary, grammar, and character composition of the text, determine which language the recognition text belongs to, thereby determining the source language corresponding to the real-time audio stream.

[0068] For example, if the recognized text content is mainly composed of Chinese characters, it can be determined that the source language of the audio stream is Chinese. If the text content is mainly composed of English words consisting of Latin letters, the source language is determined to be English. The advantage of the first language determination strategy is that it can use a powerful language model to perform refined analysis on text information, especially suitable for scenarios of language pairs with obvious language feature differences or existing mature hybrid recognition models. Figure 4 It is one of the timing schematic diagrams of simultaneous interpretation provided by the present invention. The first language determination strategy can be implemented by referring to Figure 4 the macroscopic processing mode shown, and its core is to first recognize the text and then analyze the text to determine the language.

[0069] First, the control system continuously receives the real-time audio stream, and the built-in voice activity detection module analyzes the audio stream in real time. When human voice is detected (ignoring silence and background noise), a valid audio segment is segmented from the continuous real-time audio stream, as shown by Figure 4 the first yellow rectangular block in. This valid audio segment has a clear start time and end time.

[0070] For this segmented valid audio segment, the control system performs speech recognition operations on it. In the context of the first language determination strategy, the first step of this task is to convert the valid audio segment into recognition text that the machine can understand. For example, if the user says "Hello, world" in Chinese, the text "你好,世界" will be obtained.

[0071] After obtaining the recognition text, the control system starts a text analysis algorithm to analyze the text content of "你好,世界". The analysis method adopted can be to check whether the recognition text is composed of Chinese characters, English letters, or other characters. In the above example, since the text "你好,世界" is completely composed of Chinese characters and Chinese punctuation marks, the text analysis algorithm will determine its language as Chinese. Therefore, the control system finally determines that the source language corresponding to this valid audio segment is Chinese. The above entire process repeats for each valid audio segment, achieving automatic language determination based on text content.

[0072] Furthermore, the second language determination strategy may include: extracting audio features from the audio stream to determine the source language corresponding to the audio stream based on these features. This second language determination strategy directly seeks judgment criteria from the sound signal itself.

[0073] When implementing the second language determination strategy, it does not rely entirely on the final recognized text, but rather extracts and analyzes the audio features of the real-time audio stream itself, such as phonemes, syllable structure, speech rate, pitch curve, Mel Frequency Cepstral Coefficients (MFCC), and other acoustic information to determine the language used by the speaker.

[0074] For example, different languages ​​have unique patterns in phoneme combinations and pronunciation rhythms. Second language identification strategies can use a pre-trained acoustic language classification model to match and score these patterns in real time, thereby determining the source language. The advantages of second language identification strategies are that they are less dependent on the accuracy of speech recognition results, have a faster response time, and provide a more robust identification method when dealing with languages ​​that have similar pronunciations but significant textual differences, or when the recognition model is incomplete.

[0075] It should be noted that this embodiment only demonstrates two different language determination strategies. Other language determination strategies can also be used in the simultaneous interpretation method provided by this invention, and these are not considered as specific limitations on the scope of protection of this invention.

[0076] This invention provides at least two language determination strategies that are fundamentally different in their technical principles: one based on text content and the other based on audio features. This greatly enriches the toolbox of the control system, enabling it to select a text content strategy that is better suited to handling mixed speech in scenarios such as Chinese-English translation, and a faster-responding and more robust audio feature strategy for other scenarios, depending on the type of translation between different language pairs. This differentiated strategy ensures that language determination can always be performed in the optimal way, regardless of the translation task, thereby significantly improving the accuracy and scenario adaptability of automatic language determination.

[0077] Figure 5 This is one of the flowcharts for source language determination provided by the present invention, such as... Figure 5 As shown, when the target language determination strategy corresponding to the translation type of the language pair is determined from at least two different language determination strategies to be the first language determination strategy, the step of determining the source language of the real-time audio stream according to the target language determination strategy includes: Determine whether the identified text is a mixed language text containing the first language and the second language; If so, the mixed language text is split according to the preset context information to obtain a first text part belonging to the first language and a second text part belonging to the second language; based on the first text part and the second text part, the source language of the time-audio stream is determined; If not, the language corresponding to the identified text is determined as the source language of the audio stream.

[0078] As an optional implementation, this embodiment elaborates in detail on the internal logic of the first language determination strategy, namely the text content-based determination strategy, especially providing a clear branch processing flow for how to handle the two cases of pure language text and mixed language text.

[0079] First, when implementing the first language determination strategy, the control system needs to perform a language purity analysis on the recognized text output by the speech recognition module, that is, to determine whether the recognized text is a mixed language text containing both the first and second languages. The recognized text here can be a complete sentence obtained in a single speech activity detection, or an intermediate text fragment generated during streaming recognition. The specific determination process can be implemented in various ways, for example: Method 1: Scan and recognize each character in the text, determining its Unicode character range. If the text contains both Chinese characters and Latin letters, it can be identified as a mixed Chinese-English language text.

[0080] Method 2 involves segmenting the identified text into words and matching them with dictionaries of the first language (e.g., Chinese) and the second language (e.g., English). If the matching results contain words from both languages, it is classified as mixed-language text.

[0081] Furthermore, if the determination result is negative, meaning the identified text is purely in another language, the process enters a simple and efficient processing branch. In this case, the control system directly determines the language of the identified text as the source language for this translation. For example, if the identified text is "Our product design is very innovative," the control system, through character set detection or dictionary analysis, determines that it is purely Chinese text, and then directly determines that the source language of the corresponding audio stream is Chinese.

[0082] Conversely, if the judgment result is yes, that is, the identified text is determined to be mixed language text, the process enters a specially designed intelligent processing branch, the specific execution steps of which are as follows: First, the mixed-language text is split according to the preset context information. The context information here is the key historical information guiding the splitting decision. In an optional embodiment, the context information may be the translation direction of the last successful translation in this session. For example, if the previous sentence is from English to Chinese, then the control system can infer that for a mixed language starting with Chinese in the current sentence (such as "好的, no problem"), the main part is likely to be a response to the previous English sentence. At this time, based on the context information and the language boundary of the text itself (such as the junction between Chinese characters and English words), a mixed-language text sentence is segmented into multiple pure-language text segments.

[0083] For example, for the recognized text "比如我们的UI design借鉴了a minimalist concept", the control system may split it to obtain the first text part belonging to the first language (Chinese) "比如我们的...借鉴了...", and the second text parts belonging to the second language (English) "UI design" and "a minimalist concept".

[0084] Furthermore, after obtaining the multiple text parts after splitting, the control system can make a final determination according to the preset rules to determine the main language of the whole sentence as the source language of the time audio stream. For example, it can be analyzed that the Chinese part constitutes the basic grammar framework of the sentence, thus determining the source language as Chinese.

[0085] The embodiments of the present invention can perform differential processing on different recognized texts. For simple pure-language inputs, it can pass quickly, while for complex mixed-language inputs, it starts an intelligent splitting and decision-making mechanism based on context information. This refined processing method greatly improves the accuracy of language determination of the system in dealing with real and complex dialogue scenarios, avoids semantic understanding errors and translation failures caused by roughly classifying a mixed-language whole sentence into a single language, and significantly enhances the intelligence and practicality of the translation system.

[0086] As an optional embodiment, determining the source language of the time audio stream based on the first text part and the second text part includes: Determining the source language of the real-time audio stream based on the text length ratios of the first text part and the second text part in the mixed-language text; Or, Determining the matrix text constituting the main structure of the mixed-language text, and determining the language corresponding to the matrix text as the source language of the real-time audio stream.

[0087] When a mixed-language text is split into multiple text parts belonging to the first language and the second language, the control system needs a clear rule to determine which language should be used as the source language for the whole sentence. For this purpose, the embodiments of the present invention provide two specific and alternative implementation methods to clarify how to perform language determination after the mixed-language text is split.

[0088] The first is the determination method based on the proportion of text length. In specific implementation, the control system will calculate the total length (which can be the number of characters or bytes) of all text parts belonging to the first language and the total length of all text parts belonging to the second language respectively.

[0089] Then, compare the sizes of these two total lengths. Generally, the language with a longer text length and a higher proportion can be considered as the main language corresponding to the text, so it is determined as the final source language. For example, for the recognized text "这个product的设计非常棒" (This design of the product is very good), the length of the Chinese part "这个...的设计非常棒" is significantly greater than the English part "product". Therefore, the control system will determine that the source language is Chinese.

[0090] The first is the determination method based on syntactic structure analysis. Among them, the matrix text refers to the language that provides the syntactic framework in the code-switching sentence. In the implementation of the present invention, this method no longer simply compares the lengths, but performs a deeper syntactic structure analysis on the mixed-language text.

[0091] In specific implementation, the control system will use natural language processing (NLP) technology to analyze the syntactic function words in the sentence, such as auxiliary words (的, 了, 着), prepositions (在, 从), conjunctions (和, 但是), etc. These syntactic function words usually determine the basic framework of a sentence. For example, in the recognized text "比如,我们的UIdesign借鉴了a minimalist concept" (For example, our UI design draws on a minimalist concept), although the English part has a considerable length, the key Chinese syntactic auxiliary words such as "的" and "了" clearly construct the subject-predicate-object structure of the whole sentence. Therefore, the control system will determine that Chinese is the matrix text of the main structure of this sentence, and thus determine the source language as Chinese.

[0092] In the preceding embodiments, the control system primarily processed long sentences in pure or mixed languages ​​with relatively clear structures. However, in real conversations, very short responses such as "OK," "good," "Right," and "um" frequently occur. These short sentences contain very little text and are difficult to determine through conventional segmentation or grammatical analysis, but their position and timing in the conversation exhibit strong regularity. Therefore, this embodiment of the invention adds a practical and important backup logic to the first language determination strategy, specifically designed to handle those short, immediate responses that are common in conversations. The execution logic of this scheme is as follows: First, a trigger condition is set: before determining whether the identified text is a mixed-language text containing both the first and second languages, or when it is impossible to segment the mixed-language text based on preset context information, a preset short-phrase response condition is determined. This trigger condition allows for rapid interception before entering the complex mixed-text analysis process, and also serves as a fallback if the mixed-text analysis process fails, ensuring the integrity of the processing flow.

[0093] The preset short-phrase response condition is key to triggering this backup logic, and it consists of two sub-conditions connected by an AND logic: The condition that the length of the identified text is less than a preset length threshold is used to filter out short sentences. The preset length threshold can be set based on experience, for example, 3 words or 5 Chinese characters. Only when the identified text is very short can it be a response.

[0094] The condition that the time interval between the start time of the recognized text corresponding to the speech and the end time of the previous sentence translation is less than a preset time threshold is used to filter out immediate feedback. This measure is the response delay between the current sentence and the previous sentence. In fluent dialogue, the response usually follows immediately after the previous sentence. The preset time threshold can be set to a short time, such as 1 second or 1.5 seconds.

[0095] In this embodiment, the control system has a high degree of confidence in recognizing a short response only when a recognized text simultaneously meets the conditions of short length and fast response, thereby triggering subsequent special processing logic.

[0096] Specifically, if the judgment result is yes, meaning the above-mentioned short sentence response conditions are met, then a special judgment rule based on contextual inertia is executed: the language that is the same as the target language of the previous sentence is determined as the source language of the real-time audio stream. This is because the language of a short, immediate response is usually consistent with the language of the previous sentence.

[0097] For example, User A speaks in English, and the device translates it into Chinese and presents it to User B. After hearing the Chinese translation, User B immediately says "OK". The control system recognizes the text "OK" and finds that it meets the conditions of short length and fast response. At this time, the control system finds that the target language of the previous translation is Chinese, so it can quickly determine that the source language of the current "OK" is also Chinese and perform the Chinese-to-English translation operation. This special processing logic avoids complex analysis of the word "OK" itself, but makes a fast and accurate judgment by utilizing its positional characteristics in the dialogue flow.

[0098] By adding a dedicated backup logic for handling short sentence responses, this invention greatly improves the robustness and fluency of the translation system in real-world dialogue flows. It can efficiently and accurately handle edge cases that are difficult to handle with conventional analysis methods, avoiding stuttering or misjudgment on these simple responses. This makes the entire bilingual communication process more coherent and natural, and significantly improves the user's actual experience.

[0099] Figure 6 This is the second schematic diagram of the source language determination process provided by the present invention, such as... Figure 6 As shown, this embodiment of the invention provides the internal implementation logic of a second language determination strategy, namely, a determination strategy based on audio features. This is particularly suitable for handling language pairs where mixed recognition models are immature or where mixed speech is infrequent (such as Chinese-Korean language pairs, Chinese-Thai language pairs, etc.). The main steps include, but are not limited to, the following: Step 121: Perform voice activity detection on the real-time audio stream to obtain valid audio segments containing human voices.

[0100] The continuous real-time audio stream contains user speech, silence, and ambient noise. To focus on relevant information, the control system uses voice activity detection technology to accurately identify and extract valid audio segments containing human speech from the real-time audio stream, and marks their start and end times. Subsequent language determination can then be based on these valid audio segments.

[0101] Step 122: Perform initial language determination at the start time of the valid audio segment and generate initial language determination result.

[0102] Figure 7 This is the second timing diagram of simultaneous interpretation provided by the present invention, as shown below. Figure 7As shown, to achieve rapid response and improve user experience, the control system immediately performs an initial language determination at the moment voice activity is detected (e.g., within the first few hundred milliseconds). This determination is based on extremely short audio information, and its purpose is not to pursue absolute accuracy, but to quickly generate an initial language determination result and feed it back to the user, allowing the user to immediately perceive that the system has started working, thereby alleviating waiting anxiety.

[0103] Step 123: Monitor the duration of the valid audio segment. This is a dynamic and continuous process. After the speech begins, the control system monitors in real time how long the current valid audio segment has lasted. There is a preset threshold (e.g., 2 seconds or 3 seconds) to distinguish between short and long sentences.

[0104] Step 124: If the effective audio segment ends before the duration reaches the preset threshold, then the effective audio segment is determined to be a short effective audio segment.

[0105] When the end of the voice is detected, if the total duration does not reach the preset threshold, the control system will determine the segment as a short valid audio segment.

[0106] Step 125: At the end of the short valid audio segment, perform an end language determination to determine the source language of the real-time audio stream.

[0107] For short, valid audio segments, the subsequent process is relatively simple. After the initial judgment is completed, the control system will directly call the end-of-speech language judgment at the end to determine the final source language. This end-of-speech language judgment is a decision with the highest authority, which will comprehensively analyze all audio information of the entire valid audio segment.

[0108] Figure 8 This is the third flowchart of the source language determination provided by the present invention, such as... Figure 8 As shown, this embodiment of the invention provides another method for determining the source language, which mainly includes, but is not limited to, the following steps: Step 131: Perform voice activity detection on the real-time audio stream to obtain valid audio segments containing human voices.

[0109] Step 132: Perform initial language determination at the start time of the valid audio segment and generate initial language determination result.

[0110] Step 133: Monitor the duration of the effective audio segment.

[0111] Step 134: If the duration of the valid audio segment reaches the preset threshold, then the valid audio segment is determined to be a long valid audio segment.

[0112] Step 135: When the long effective audio segment reaches the preset threshold, perform intermediate language determination to calibrate the initial language determination result.

[0113] Step 136: At the end of the long valid audio segment, perform an end language determination to determine the source language of the real-time audio stream.

[0114] Steps 131 to 133 are implemented in the same way as steps 121 to 123 in the previous embodiments, and will not be described again here. The difference from the previous embodiments is that, in this embodiment of the invention, when the duration is detected to have reached the preset threshold, the control system will determine the valid audio segment as a long valid audio segment.

[0115] refer to Figure 7 As shown, for long, valid audio segments, due to the larger amount of information, the control system adds an intermediate step: at the moment a preset threshold is reached, an intermediate language determination is performed. The purpose of this intermediate language determination is to calibrate the initial language determination result displayed on the interface based on the accumulated audio information over several seconds, in order to provide a more reliable temporary state.

[0116] Subsequently, when the end of this long valid audio segment is detected, the control system will also call the same end language judgment as the short valid audio segment processing to make the final and authoritative source language decision at the end moment.

[0117] This invention introduces an adaptive judgment process based on audio duration, which can adopt different complexity processing strategies for sentences of different lengths. By progressively refining the process, it improves the stability of judgment and the accuracy perceived by the user. This differentiated and progressive streaming judgment mechanism cleverly balances the two goals of rapid response and final accuracy, providing users with a simultaneous interpretation experience that is both sensitive and robust.

[0118] Figure 9 This is the fourth flowchart of the source language determination provided by the present invention, as follows: Figure 9 As shown, this invention provides a feasible implementation method for determining the end language, targeting the core step common to both the short and long valid audio segments in the above embodiments. This mainly includes, but is not limited to, the following steps: Step 211: Calculate the first effective pronunciation duration accumulated in the first language and the second effective pronunciation duration accumulated in the second language within the entire effective audio segment at the end time.

[0119] Step 212: Determine the duration weights of the first language and the second language based on the first effective pronunciation duration and the second effective pronunciation duration.

[0120] Step 213: Determine the language corresponding to the maximum duration weight as the source language of the real-time audio stream.

[0121] Specifically, upon detecting the end of a spoken segment, the control system reviews and analyzes the entire valid audio segment just received. The system's internal language recognition model performs fine-grained analysis of the valid audio segment frame-by-frame or segment-by-segment, identifying which parts belong to the first language (e.g., Chinese) and which parts belong to the second language (e.g., Korean). Generally, the durations of all audio segments identified as belonging to the first language are summed to obtain the first valid pronunciation duration; the durations of all audio segments identified as belonging to the second language are summed to obtain the second valid pronunciation duration.

[0122] For example, in a valid audio segment with a total length of 10 seconds, analysis reveals that the first 7 seconds are Chinese pronunciation and the last 3 seconds are Korean pronunciation. Therefore, the statistically determined first valid pronunciation duration is 7 seconds, and the second valid pronunciation duration is 3 seconds.

[0123] Furthermore, after obtaining the total pronunciation duration for each of the two languages, these physical durations can be transformed into decision-making criteria, namely, duration weighting.

[0124] In a simplified embodiment, the duration weight can be determined by directly using the effective pronunciation duration itself as the weight value. That is, weight value (Chinese) = 7, weight value (Korean) = 3.

[0125] In another alternative embodiment, normalization can also be performed. For example, the proportion of each language's effective pronunciation duration in the total duration can be calculated as a weight, resulting in: weight value (Chinese) = 7 / 10 = 0.7, weight value (Korean) = 3 / 10 = 0.3. Regardless of the method used, languages ​​with longer durations will necessarily have higher duration weights.

[0126] Finally, the control system compares the duration weights of each language calculated in the previous step. In the example above, since the duration weight of Chinese (whether 7 or 0.7) is greater than that of Korean (3 or 0.3), Chinese is the language with the highest duration weight. Therefore, the control system will ultimately determine Chinese as the source language of this real-time audio stream.

[0127] This invention significantly improves the robustness of language determination by introducing a weighted decision algorithm based on the entire segment's duration at the end of the judgment. This algorithm no longer relies solely on the last part of the audio or a single feature point, but comprehensively considers the weight of different linguistic components throughout the sentence, effectively avoiding misjudgments caused by noise such as foreign words, colloquialisms, or brand names at the end of sentences. This global perspective decision-making mechanism makes the final source language determination result more stable and reliable, and more in line with human communication habits, fundamentally ensuring the correctness of subsequent translation directions.

[0128] To achieve rapid response, the control system generates an initial language determination result at the very beginning of the speech. As an optional embodiment, a scheme is provided for using this rapidly generated initial language determination result to interact with the user. Specifically, after generating the initial language determination result, the system further includes: The initial language determination result is sent to the user terminal in real time.

[0129] The user terminal is controlled to display the language identifier corresponding to the initial language determination result on the interface.

[0130] The user end refers to the physical device that the user directly operates and perceives, such as a translator, a translation app interface on a smartphone, or a pairing application interface for translation headphones.

[0131] After the control system (usually located on a cloud server or a powerful processor on the device) determines the initial language at the start time, for example, if it initially determines that the language is Chinese, it will immediately execute a sending action. For example, the encoding or identifier of this initial language determination result is transmitted in real time to the module responsible for the user interface display via an internal communication bus or network connection.

[0132] After receiving the initial language determination result, the user interface display module on the user end will display a corresponding language identifier in a specific area of ​​the screen.

[0133] The language identifier can be a variety of visual elements, such as text identifiers, icon identifiers, dynamic status prompts, etc.

[0134] This invention, by converting a temporary language determination result into a user-visible interface feedback within a very short time after the start of speech, greatly improves the immediacy of human-computer interaction, effectively fills the waiting gap between when the user starts speaking and when they see the final translation result, significantly reduces user uncertainty and waiting anxiety, and makes the entire simultaneous interpretation product's interactive experience smoother, more user-friendly, and more intelligent.

[0135] This invention specifies the implementation methods for the two data transmission steps—receiving real-time audio streams and outputting translation results—at the system architecture and communication protocol levels. The aim is to provide a solid foundation for the entire simultaneous interpretation process that is low-latency, high-efficiency, and supports bidirectional real-time communication. This mainly includes: Establish a long TCP connection with the client and create a bidirectional stream on the long TCP connection for this simultaneous interpretation session; receive the real-time audio stream sent by the client through the bidirectional stream.

[0136] In a specific implementation of this invention, the server and client no longer use the traditional short-connection mode of one request corresponding to one response. Instead, a persistent communication channel is created by establishing a long TCP connection at the beginning of a complete simultaneous interpretation session. This long TCP connection avoids the network overhead and time delay caused by repeated TCP handshakes during the session.

[0137] In an optional embodiment, the persistent TCP connection is established based on the HTTP / 2 protocol. A core advantage of the HTTP / 2 protocol is its multiplexing capability, which allows multiple data streams to be transmitted in parallel and bidirectionally over a single TCP connection.

[0138] Therefore, after establishing a TCP long connection, the control system creates a dedicated bidirectional stream for this translation session. Once the client captures the user's voice through the audio acquisition device, it encodes the voice data and continuously sends it to the server in the form of data frames through the bidirectional stream created in the previous step.

[0139] In this embodiment, since the data is transmitted within a pre-established long-connection stream, true streaming transmission can be achieved, i.e., data is collected and transmitted simultaneously, which greatly reduces the transmission latency of the entire speech sentence and makes real-time processing on the server possible.

[0140] Accordingly, the output translation result includes: pushing the source language determination result and the translation result to the client in real time through the bidirectional stream.

[0141] After completing its internal processing logic, the server returns various results to the client. In a unified bidirectional stream, the server can proactively and in real time push these results to the client without waiting for the client's next request.

[0142] The data pushed to the client can include the source language determination results and translation results. The translation results mainly include the identified source language text, the translated target language text, and the synthesized target language audio stream. The determination results can be pushed to the client in real time during the initial, intermediate, and final determination stages to update the UI display.

[0143] This push mechanism, based on a single long connection and bidirectional stream, ensures that the results of each step from recognition to translation are presented to the user with minimal latency, which is a key technological guarantee for achieving a real-time experience comparable to simultaneous interpretation.

[0144] Figure 10 This is a schematic diagram of the simultaneous interpretation device provided by the present invention, as shown below. Figure 10 As shown, it mainly includes, but is not limited to: The strategy selection module 101 is used to obtain a preset language pair containing a first language and a second language, and determine the target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies. The language determination module 102 receives a real-time audio stream and determines the source language of the real-time audio stream according to the target language determination strategy; when the source language is determined to be the first language, the second language is determined to be the target language; otherwise, when the source language is determined to be the second language, the first language is determined to be the target language. The translation operation module 103 is used to perform a translation operation on the real-time audio stream based on the determined source language and target language, and output the translation result.

[0145] It should be noted that the simultaneous interpretation device provided by the present invention can execute the simultaneous interpretation method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0146] The simultaneous interpretation device provided by this invention dynamically matches and adopts the optimal language determination strategy for different language pairs, solving the shortcomings of existing technologies in terms of low accuracy and easy misjudgment when facing different translation scenarios. It significantly improves the accuracy and robustness of automatic language determination, thereby ensuring the correctness of the translation direction and the final translation quality, and greatly improving the communication efficiency and experience of users in bilingual real-time communication.

[0147] This invention also provides a simultaneous interpretation device, which mainly includes, but is not limited to: The audio acquisition module is used to acquire the user's real-time audio stream; The communication module is used to establish long-lived connection sessions with the cloud server; The processor is configured to perform the following operations: send a language pair containing a first language and a second language, along with the real-time audio stream, to the cloud server via the communication module; and receive the translation result of the real-time audio stream returned by the cloud server. The cloud server can generate the translation result based on the simultaneous interpretation method provided in any of the above embodiments.

[0148] In one specific embodiment, the simultaneous interpretation device mainly includes an audio acquisition module, a communication module, and a processor.

[0149] The audio acquisition module, in its physical implementation, can be one or more high-fidelity microphones, or a microphone array configured with beamforming and noise reduction algorithms. Its main function is to efficiently and clearly capture the user's voice and convert it into a digital real-time audio stream.

[0150] The communication module, in its physical implementation, can be a communication chip or module that integrates Wi-Fi, Bluetooth, or 5G / 4G cellular network functions. Its core function is to establish and maintain stable communication between the device and the cloud server.

[0151] In an optional embodiment, the communication module is used to establish a persistent TCP long connection with the cloud server based on the HTTP / 2 protocol and create a dedicated bidirectional stream for a complete simultaneous interpretation session.

[0152] The processor, as the main control unit of the device, can be a general-purpose CPU or a dedicated embedded processor. It is responsible for coordinating the work of other modules in the device and performing the following core operations: (1) When the user selects a language pair containing the first and second languages ​​(e.g., Chinese-English translation) through the device's interactive interface (such as a touch screen or buttons) and begins to speak, the processor first controls the audio acquisition module to work. Subsequently, the processor packages the preset language pair information and the real-time audio stream data generated by the audio acquisition module and sends it to the cloud server through the communication module along the established long connection session.

[0153] (2) While sending audio, the processor continuously monitors the data returned from the cloud server via the communication module. When it receives the translation result from the cloud server, it analyzes the translation result. It should be noted that the translation result is accurate and intelligent because the cloud server internally executes the simultaneous interpretation method provided in any of the aforementioned embodiments to generate it. For example, the cloud server will select a first language determination strategy based on the text content according to the translation type of the Chinese-English language pair, intelligently process the user's mixed speech, and finally determine the correct translation direction and complete the translation.

[0154] (3) After the processor parses the translation result, it will call the output module of the device to display the translated text on the screen or play the synthesized speech through the speaker.

[0155] Through this edge-cloud separation architecture, the simultaneous interpretation device provided by this invention can maintain lightweight hardware and low power consumption, while fully utilizing the powerful computing capabilities of cloud servers to execute complex AI algorithms. Users only need to operate a simple terminal device to enjoy high-quality, low-latency real-time simultaneous interpretation services provided by complex cloud methods. As an optional embodiment, the provided simultaneous interpretation device may further include: an output module for playing the audio corresponding to the translation result, and / or updating and displaying the source language determination result in real time on the display interface.

[0156] Specifically, the output module is the core interface through which users directly perceive the translation results and the device's operating status. Its design is tightly coupled with multiple aspects of the method of this invention, aiming to provide a complete and user-friendly interactive loop. In terms of physical implementation, the output module can integrate one or more components, such as a display unit (e.g., an LCD or OLED touchscreen) and an audio playback unit (e.g., a speaker, headphones, or an audio interface for connecting headphones).

[0157] Figure 11 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a simultaneous interpretation method. This method includes: acquiring a preset language pair containing a first language and a second language; determining a target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies; receiving a real-time audio stream and determining the source language of the real-time audio stream according to the target language determination strategy; when the source language is determined to be the first language, determining the second language as the target language; conversely, when the source language is determined to be the second language, determining the first language as the target language; and performing a translation operation on the real-time audio stream based on the determined source language and the target language, and outputting the translation result.

[0158] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the simultaneous interpretation method provided in the above embodiments, the method comprising: acquiring a preset language pair containing a first language and a second language; determining a target language determination strategy corresponding to the mutual translation type of the language pair from at least two different language determination strategies; receiving a real-time audio stream to determine the source language of the real-time audio stream according to the target language determination strategy; determining the second language as the target language when the source language is determined to be the first language; conversely, determining the first language as the target language when the source language is determined to be the second language; and performing a translation operation on the real-time audio stream based on the determined source language and the target language, and outputting a translation result.

[0160] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the simultaneous interpretation method provided in the above embodiments. The method includes: acquiring a preset language pair containing a first language and a second language; determining a target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies; receiving a real-time audio stream and determining the source language of the real-time audio stream according to the target language determination strategy; when the source language is determined to be the first language, determining the second language as the target language; otherwise, when the source language is determined to be the second language, determining the first language as the target language; and performing a translation operation on the real-time audio stream based on the determined source language and the target language, and outputting a translation result.

[0161] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A simultaneous interpreting method, characterized in that, include: Obtain a preset language pair containing a first language and a second language, and determine the target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies; Receive real-time audio streams and determine the source language of the real-time audio streams according to the target language determination strategy; When the source language is determined to be the first language, the second language is determined to be the target language; Conversely, when the source language is determined to be the second language, the first language is determined to be the target language; Based on the determined source language and target language, a translation operation is performed on the real-time audio stream, and the translation result is output.

2. The simultaneous interpretation method according to claim 1, characterized in that, The language determination strategy includes at least a first language determination strategy and a second language determination strategy. The first language determination strategy includes: performing speech recognition on the audio stream to obtain recognized text, and determining the source language of the audio stream based on the text content of the recognized text; The second language determination strategy includes: extracting audio features of the audio stream to determine the source language of the audio stream based on the audio features of the audio stream.

3. The simultaneous interpretation method according to claim 2, characterized in that, When the target language determination strategy corresponding to the translation type of the language pair is determined from at least two different language determination strategies to be the first language determination strategy, the step of determining the source language of the real-time audio stream according to the target language determination strategy includes: Determine whether the identified text is a mixed language text containing the first language and the second language; If so, the mixed language text is split according to the preset context information to obtain a first text part belonging to the first language and a second text part belonging to the second language; based on the first text part and the second text part, the source language of the time-audio stream is determined; If not, the language corresponding to the identified text is determined as the source language of the audio stream.

4. The simultaneous interpretation method according to claim 3, characterized in that, Determining the source language of the time-to-audio stream based on the first text portion and the second text portion includes: The source language of the real-time audio stream is determined based on the proportion of the text length of the first text portion and the second text portion in the mixed language text; or, The matrix text constituting the backbone structure of the mixed language text is determined, and the language corresponding to the matrix text is determined as the source language of the real-time audio stream.

5. The simultaneous interpretation method according to any one of claims 3-4, characterized in that, Before determining whether the identified text is a mixed language text containing the first language and the second language, or when it is impossible to split the mixed language text according to the preset context information, it is determined whether the preset short sentence response condition is met. If the short sentence response conditions are met, the language that is the same as the target language of the previous sentence translation will be determined as the source language of the real-time audio stream; The preset short sentence response conditions include that the length of the recognized text is less than a preset length threshold, and the time interval between the start time of the speech corresponding to the recognized text and the end time of the previous sentence translation is less than a preset time threshold.

6. The simultaneous interpretation method according to claim 2, characterized in that, When the target language determination strategy corresponding to the translation type of the language pair is determined to be the second language determination strategy from at least two different language determination strategies, the step of determining the source language of the real-time audio stream according to the target language determination strategy includes: Speech activity detection is performed on the real-time audio stream to obtain valid audio segments containing human voices; At the beginning of the valid audio segment, an initial language determination is performed, and an initial language determination result is generated; Monitor the duration of the valid audio segment; If the effective audio segment ends before the duration reaches a preset threshold, the effective audio segment is determined to be a short effective audio segment; at the end of the short effective audio segment, an end language determination is performed to determine the source language of the real-time audio stream; If the duration of the valid audio segment reaches the preset threshold, the valid audio segment is determined to be a long valid audio segment; when the long valid audio segment reaches the preset threshold, an intermediate language determination is performed to calibrate the initial language determination result; at the end of the long valid audio segment, an end language determination is performed to determine the source language of the real-time audio stream.

7. The simultaneous interpretation method according to claim 5, characterized in that, The process of determining the source language of the real-time audio stream by ending the execution includes: The cumulative first effective pronunciation duration of the first language and the cumulative second effective pronunciation duration of the second language are calculated within the entire effective audio segment at the end time. Based on the first effective pronunciation duration and the second effective pronunciation duration, determine the duration weights of the first language and the second language; The language corresponding to the maximum duration weight is determined as the source language of the real-time audio stream.

8. The simultaneous interpretation method according to any one of claims 5-6, characterized in that, After generating the initial language determination result, the following is also included: The initial language determination result is sent to the user terminal in real time; The user terminal is controlled to display the language identifier corresponding to the initial language determination result on the interface.

9. The simultaneous interpretation method according to claim 1, characterized in that, The receiving of real-time audio streams includes: Establish a long TCP connection with the client, and create a bidirectional stream on the long TCP connection for this simultaneous interpretation session; The real-time audio stream sent by the client is received through the bidirectional stream; Accordingly, the output translation result includes: The source language determination result and the translation result are pushed to the client in real time through the bidirectional stream.

10. A simultaneous interpretation device, characterized in that, include: The strategy selection module is used to obtain a preset language pair containing a first language and a second language, and determine the target language determination strategy corresponding to the translation type of the language pair from at least two different language determination strategies. The language determination module receives a real-time audio stream and determines the source language of the real-time audio stream according to the target language determination strategy. When the source language is determined to be the first language, the second language is determined to be the target language; Conversely, when the source language is determined to be the second language, the first language is determined to be the target language; The translation operation module is used to perform a translation operation on the real-time audio stream based on the determined source language and target language, and output the translation result.

11. A simultaneous interpretation device, characterized in that, include: The audio acquisition module is used to acquire the user's real-time audio stream; The communication module is used to establish long-lived connection sessions with the cloud server; The processor is configured to perform the following operations: send a language pair containing a first language and a second language, along with the real-time audio stream, to the cloud server via the communication module; and receive the translation result of the real-time audio stream returned by the cloud server. The cloud server generates the translation result based on the simultaneous interpretation method described in any one of claims 1-9.

12. The simultaneous interpretation equipment according to claim 11, characterized in that, Also includes: The output module is used to play the audio corresponding to the translation result and / or to update and display the source language determination result in real time on the display interface.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the simultaneous interpretation method as described in any one of claims 1 to 9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the simultaneous interpretation method as described in any one of claims 1 to 9.

Images