Visual session translation method and related equipment

Abstract

The invention discloses a visual session translation method and related equipment, and relates to the technical field of intelligent terminals and voice processing, and the method comprises the steps: responding to a session translation instruction, employing a microphone to collect a speech of a speaker, obtaining a to-be-translated first statement, employing a display of a mobile translation terminal to display the first statement, and obtaining a to-be-translated second statement; sending the first statement to a speaker to enable the speaker to determine whether the first statement has an error, if the speaker confirms that the first statement does not have an error, executing translation operation on the first statement based on a preset translation rule to obtain a second statement, and displaying the second statement to a listener. According to the translation mobile terminal, the display is integrated, when a speaker speaks, the collected content is displayed on the display, so that the speaker confirms whether the content collected by the mobile translation terminal is consistent with the content actually needing to be translated or not, the accuracy of the to-be-translated statement is improved, and then the accuracy of the translation result is improved.

Description

Technical Field

[0001] This application relates to the fields of smart terminals and voice processing technology, and in particular to a visual conversation translation method and related equipment. Background Technology

[0002] With the acceleration of globalization, the demand for mobile translation technology is growing rapidly. Existing solutions mainly include smartphone applications, translation headsets, and AI glasses. However, smartphone applications require both parties to share a single device, which is inconvenient to operate and results in a poor user experience. While translation headsets and AI glasses achieve device separation, they lack a visual confirmation mechanism for voice recognition, making it impossible for users to judge the accuracy of their input in real time, leading to insufficient interactivity and information transparency. Furthermore, these technologies struggle to support multi-person conversations, and their portability and adaptability to different scenarios are also limited. These shortcomings restrict the fluency and credibility of face-to-face cross-language communication, necessitating a new type of translation terminal that combines real-time feedback, multi-terminal collaboration, and a portable design to improve interaction efficiency and user experience.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is related technology. Summary of the Invention

[0004] The main purpose of this application is to provide a visual conversation translation method and related equipment, aiming to solve the technical problem of how to improve the accuracy of translation results.

[0005] To achieve the above objectives, this application proposes a visual conversation translation method applied to a mobile translation terminal. The visual conversation translation method includes: In response to a conversation translation command, the microphone is used to capture the speaker's speech, obtaining the first sentence to be translated; The first statement is displayed on the screen of the mobile translation terminal so that the speaker can determine whether there is an error in the first statement; If the speaker confirms that the first statement is error-free, based on preset translation rules, the first statement is translated to obtain the second statement, which is then presented to the listeners.

[0006] In one embodiment, before the step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement, the method further includes: If the speaker confirms that the first statement contains an error, obtain the first statement after the speaker has corrected the first statement; In one embodiment, the step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement further includes: Based on preset translation rules, a translation operation is performed on the modified first statement to obtain the second statement.

[0007] In one embodiment, the step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement further includes: Based on preset translation rules, the first statement is parsed using a preset speech parsing model to obtain the parsing result; Based on historical translation records and the analysis results, the current translation target is determined; Based on the translation target, a translation operation is performed on the first statement to obtain the second statement.

[0008] In one embodiment, before the step of parsing the first statement using a preset script parsing model to obtain the parsing result, the method further includes: Obtain sample data, and the parsing result corresponding to the sample data is the first parsing result; The sample data is processed using the current speech parsing model to obtain a second parsing result; Determine whether the first parsing result is consistent with the second parsing result; If there is a discrepancy, adjust the parameters of the current script parsing model. Based on the adjusted parameters, return to the step of processing the sample data using the current script parsing model to obtain the second parsing result. Continue until the first parsing result is consistent with the second parsing result to obtain the preset script parsing model.

[0009] In one embodiment, the step of determining the current translation target based on historical translation records and the parsing results further includes: Based on the analysis results, the first speech feature corresponding to the first statement is determined. The speech feature includes the speaker's speaking speed, the speaker's speaking emotion, and the semantics of the first statement. Determine the second speech features corresponding to each translation record in the historical translation record to obtain multiple sets of second speech features; Based on any one of the multiple sets of second speech features, a similarity score is calculated with the first speech feature to obtain multiple sets of similarity values; Based on the maximum similarity value among the multiple sets of similarity values, the target historical translation record that best matches the first statement is determined from the historical translation records; Based on the target's historical translation records, the current translation target is determined.

[0010] In one embodiment, the step of determining the translation target based on the target's historical translation records further includes: Determine the historical translation target corresponding to the historical translation record; Determine the target language corresponding to the conversation translation instruction; Based on the target language and the historical translation targets, the current translation target is generated.

[0011] Furthermore, to achieve the above objectives, this application also proposes a visual conversation translation device, which includes: The acquisition module is used to acquire the speaker's speech using a microphone in response to a conversation translation command, thereby obtaining the first sentence to be translated; The display module is used to display the first statement using the display of the mobile translation terminal, so that the speaker can determine whether there is an error in the first statement; The translation module is used to translate the first statement based on preset translation rules if the speaker confirms that there are no errors in the first statement, to obtain a second statement, and to display the second statement to the listener.

[0012] In one embodiment, the visualized conversation translation device further includes a correction module, which further includes: The first acquisition unit is configured to acquire the first statement after the speaker has corrected the first statement if the speaker confirms that the first statement contains an error. In one embodiment, the translation module further includes: Based on preset translation rules, a translation operation is performed on the modified first statement to obtain the second statement.

[0013] In one embodiment, the translation module further includes: The parsing unit is used to parse the first statement based on preset translation rules and a preset speech parsing model to obtain the parsing result; The first determining unit is used to determine the current translation target based on historical translation records and the analysis results; The translation unit is used to perform a translation operation on the first statement based on the translation target to obtain the second statement.

[0014] In one embodiment, the translation module further includes: The second acquisition unit is used to acquire sample data, wherein the parsing result corresponding to the sample data is the first parsing result; The data processing unit is used to process the sample data using the current speech parsing model to obtain a second parsing result; A judgment unit is used to determine whether the first parsing result is consistent with the second parsing result; The iterative training unit is used to adjust the parameters of the current dialogue parsing model if there is a discrepancy. Based on the current dialogue parsing model with adjusted parameters, it returns to the step of processing the sample data using the current dialogue parsing model to obtain a second parsing result, until the first parsing result is consistent with the second parsing result, thus obtaining a preset dialogue parsing model.

[0015] In one embodiment, the translation module further includes the following steps: The second determining unit is used to determine the first speech feature corresponding to the first statement based on the parsing result. The speech feature includes the speaker's speaking speed, the speaker's speaking emotion, and the semantics of the first statement. The third determining unit is used to determine the second speech feature corresponding to each translation record in the historical translation record, and obtain multiple sets of second speech features; The calculation unit is used to calculate the similarity between any one of the multiple sets of second speech features and the first speech features, and obtain multiple sets of similarity values. The fourth determining unit is used to determine the target historical translation record that best matches the first statement among the historical translation records based on the maximum similarity value among the multiple sets of similarity values; The fifth determining unit is used to determine the current translation target based on the target's historical translation records.

[0016] In one embodiment, the translation module further includes: The sixth determining unit is used to determine the historical translation target corresponding to the historical translation record; The seventh determining unit is used to determine the target language corresponding to the conversation translation instruction; The generation unit is used to generate the current translation target based on the target language and the historical translation targets.

[0017] Furthermore, to achieve the above objectives, this application also proposes a visual conversation translation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visual conversation translation method as described above.

[0018] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the visual conversation translation method described above.

[0019] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the visual conversation translation method described above.

[0020] One or more technical solutions proposed in this application have at least the following technical effects: This application proposes a visual conversation translation method and related equipment, relating to the fields of smart terminals and voice processing technology. In this context, with the acceleration of globalization, the demand for mobile translation technology is increasing. Existing solutions mainly include smartphone applications, translation headsets, and AI glasses. However, smartphone applications require both parties to share a single device, resulting in inconvenient operation and poor interactive experience. While translation headsets and AI glasses achieve device separation, they lack a visual confirmation mechanism for voice recognition, making it impossible for users to judge the accuracy of input in real time, leading to insufficient interactivity and information transparency. Furthermore, these technologies struggle to support multi-person conversation scenarios, and their portability and scenario adaptability are also limited. In contrast, this application firstly, in response to a conversation translation command, uses a microphone to capture the speaker's speech, obtaining a first sentence to be translated. Further, the first sentence is displayed on the screen of the mobile translation terminal, allowing the speaker to determine if there are any errors in the first sentence. Finally, if the speaker confirms that the first sentence is error-free, a translation operation is performed on the first sentence based on preset translation rules to obtain a second sentence, which is then displayed to the listener.

[0021] It is understood that the translation mobile terminal of this application integrates a display screen, which displays the collected content on the screen when the speaker speaks, so that the speaker can confirm whether the content collected by the mobile translation terminal is consistent with the actual content to be translated, thereby improving the accuracy of the sentence to be translated and thus improving the accuracy of the translation result. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating a visual embodiment of the conversation translation method of this application; Figure 2A flowchart illustrating the second embodiment of the visualized conversation translation method of this application; Figure 3 A flowchart illustrating the third embodiment of the visualized conversation translation method of this application; Figure 4 This is a schematic diagram of the module structure of a conversation translation device as described in an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the visualized conversation translation method in this application embodiment.

[0025] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0027] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0028] The main solution in this application embodiment is: In this embodiment, for ease of description, a visual conversation translation device will be used as the execution subject in the following description.

[0029] Due to the increasing globalization, the demand for mobile translation technology is growing. Existing solutions mainly include smartphone applications, translation headsets, and AI glasses. However, smartphone applications require both parties to share a single device, which is inconvenient to operate and has a poor interactive experience. Although translation headsets and AI glasses achieve device separation, they lack a visual confirmation mechanism for voice recognition, making it impossible for users to judge the accuracy of their input in real time. This results in insufficient interactivity and information transparency. In addition, the relevant technologies are difficult to support multi-person conversation scenarios, and their portability and adaptability to different scenarios are also limited.

[0030] This application provides a solution in which: first, in response to a conversation translation instruction, a microphone is used to capture the speaker's speech to obtain a first statement to be translated; further, the first statement is displayed on the display of a mobile translation terminal so that the speaker can determine whether there are any errors in the first statement; finally, if the speaker confirms that there are no errors in the first statement, a translation operation is performed on the first statement based on preset translation rules to obtain a second statement, and the second statement is displayed to the listener.

[0031] It is understood that the translation mobile terminal of this application integrates a display screen, which displays the collected content on the screen when the speaker speaks, so that the speaker can confirm whether the content collected by the mobile translation terminal is consistent with the actual content to be translated, thereby improving the accuracy of the sentence to be translated and thus improving the accuracy of the translation result.

[0032] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or a visual conversation translation device capable of performing the above functions. The following description uses a visual conversation translation device as an example to illustrate this embodiment and the subsequent embodiments.

[0033] Based on this, embodiments of this application provide a visual conversation translation method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the conversation translation method visualized in this application.

[0034] In this embodiment, the visualized conversation translation method includes steps S100~S300: Step S100: In response to the conversation translation instruction, the speaker's speech is captured using a microphone to obtain the first sentence to be translated; It should be noted that "responding to conversation translation commands" refers to activating the translation mode of the smart terminal through physical buttons, touch operation, or voice wake-up words, and the trigger signal is synchronized to the translation software on the mobile phone via Bluetooth / Wi-Fi or other connection methods.

[0035] It should be noted that "using a microphone for sound acquisition" specifically refers to the MEMS microphone array built into the smart terminal performing sound pickup. This array uses a beamforming algorithm to directionally acquire speech signals corresponding to the speaking direction (within a 180° range), while simultaneously suppressing oncoming interference to ensure that there is no crosstalk when two parties are talking at close range. The acquired analog signal is then enhanced by an independent speech preprocessing chip within the terminal.

[0036] It should be noted that "receiving the first sentence to be translated" refers to the preprocessed voice data being transmitted to the mobile device via a low-latency protocol (<50ms), where the translation engine transcribes it into text in real time. This text is then sent back to the terminal screen for the speaker's confirmation. This process uses edge computing units to perform VAD (Voice Activity Detection) and voiceprint pre-recognition, ensuring that only valid voice segments are transmitted and processed, reducing power consumption and the false recognition rate.

[0037] Step S200: Display the first statement on the display of the mobile translation terminal so that the speaker can determine whether there is an error in the first statement; It should be noted that the smart terminal (mobile translation terminal) is equipped with a double-sided foldable e-ink screen + touch LCD composite module. This design aims to provide optimal display effects and interactive experience in different scenarios. It should be noted that the outer e-ink screen is used to continuously display the translated text (the second sentence), presenting it to the listener in a power-saving and clear manner.

[0038] It should be noted that the inner touch-screen LCD is used to display the original text (first sentence) after speech recognition, for the speaker to view and proofread. The touch-screen LCD supports user interaction, allowing the speaker to make text corrections through touch operations.

[0039] In this embodiment, after the speech recognition module converts the speaker's speech into text, the resulting "first statement" is immediately displayed on the inner touchscreen LCD of the smart terminal. The displayed content includes: Speech recognition result: the text content of the "first statement".

[0040] Confidence level indicator: The system will mark the text based on the confidence level of speech recognition. For example, text with a confidence level below 85% may be highlighted in red to indicate that the speaker may have made a recognition error.

[0041] Contextual correction prompts: If the system detects polysemous words or ambiguous semantics, it may display correction suggestions or prompts next to the text.

[0042] Furthermore, the speaker can check the accuracy of the speech recognition by viewing the text on the inner touchscreen LCD. If the speaker believes the displayed text is accurate, they can directly click the "Confirm" button on the screen, and the system will proceed to the next translation step. If the speaker finds errors in the text (such as speech recognition errors, unclear meanings, etc.), they can correct them in the following ways: Manual editing: Speakers can directly modify the text on the touch LCD screen, such as deleting, inserting, or replacing text.

[0043] Speech Repetition: If the system detects text with low confidence, it will automatically prompt the speaker to repeat the text, and the system will perform speech recognition again and update the text.

[0044] Context selection: For polysemous words or ambiguous meanings, the speaker can choose from the correction suggestions provided by the system, and the system will update the text according to the user's selection.

[0045] Understandably, the dual-screen design allows speakers and listeners to view different content simultaneously, avoiding information confusion and enhancing the transparency of the interaction. Highlighting low-confidence text with color or highlighting visually alerts users to potential errors, increasing their sense of control over the speech recognition results. Combined with a context engine, the system provides correction suggestions based on the dialogue context, further optimizing translation accuracy. Speakers can easily correct text using the touchscreen LCD, lowering the operational threshold and improving the user experience.

[0046] Understandably, this visual confirmation mechanism solves the uncertainty problem of speech recognition and enhances the reliability and user experience of the translation system through user interaction.

[0047] Step S300: If the speaker confirms that there are no errors in the first statement, a translation operation is performed on the first statement based on preset translation rules to obtain the second statement, and the second statement is displayed to the listener.

[0048] It should be noted that in the previous step, the speaker viewed the first sentence after speech recognition via the inner touchscreen LCD of the smart terminal and confirmed its accuracy. Further, after confirming the first sentence is correct, the system will translate it based on preset translation rules. These "preset translation rules" include: Translation engine: The system's built-in translation algorithm or a third-party translation service (such as Google Translate, Baidu Translate, etc.) is used to translate the first sentence from the source language to the target language.

[0049] Corpus: The language dataset used by the system to support the accuracy and adaptability of the translation engine. The corpus may contain vocabulary, phrases, grammatical structures, etc., of multiple languages.

[0050] Context engine: An intelligent module that combines the context of the dialogue to handle issues such as polysemous words and ambiguous semantics. For example, based on the content of the first 30 seconds of the dialogue, the system can more accurately select the appropriate translation option.

[0051] Personalized thesaurus: The system dynamically updates the personalized thesaurus based on the user's historical translation records and correction data to adapt to the user's specific expression habits and professional terminology.

[0052] After the translation is complete, the system generates a second sentence in the target language. The generation process of the second sentence includes the following key steps: Text translation: Translate the text content of the first sentence into text in the target language.

[0053] Speech synthesis (optional): If needed, the system can also convert the second statement into speech and play it to the listener through the smart terminal's speaker.

[0054] Formatting: Ensure the second statement is clearly formatted and easy to read, such as by adding punctuation marks and adjusting paragraph formatting.

[0055] Finally, the generated second sentence will be displayed to the listener via the external e-ink screen of the smart terminal. The display process includes: Text display: The second statement is displayed in clear text on the e-ink screen, ensuring easy readability under various lighting conditions.

[0056] Voice playback (optional): If the system is configured with voice output function, the second statement can also be played to the listener through the smart terminal's speaker, providing a richer interactive experience.

[0057] Multi-terminal support: In multi-person communication scenarios, the system can connect multiple smart terminals to simultaneously display the second statement to multiple listeners.

[0058] Understandably, in this embodiment, the speaker confirmation mechanism ensures the accuracy of the original text being translated, thereby improving the reliability of the translation. Combined with a context engine, the system can dynamically adjust the translation results based on the dialogue context, avoiding misunderstandings caused by polysemous words or ambiguous semantics. Utilizing a personalized thesaurus, the system can adapt to users' specific expression habits and professional terminology, improving the accuracy and adaptability of the translation. It supports both text and voice output modes to meet the needs of different scenarios and users. It supports connection to multiple smart terminals, making it suitable for multi-person communication scenarios and improving communication efficiency.

[0059] Understandably, this design enables efficient cross-language translation and significantly improves translation accuracy and user experience through user interaction and context adaptation mechanisms.

[0060] Specifically, before the step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement, step S310 is further included: Step S310: If the speaker confirms that the first statement has an error, obtain the first statement after the speaker has corrected the first statement; In this embodiment, when the speaker confirms that the first sentence obtained from speech recognition contains an error, the system enters the correction phase. The speaker can correct the first sentence in the following ways: Manual editing: Speakers can directly edit text using the on-screen LCD of the smart terminal, including deleting, inserting, replacing, or modifying words and phrases. The system provides a virtual keyboard or handwriting input function for convenient text correction.

[0061] Voice Re-input: The speaker can choose to rephrase the incorrect part or the entire sentence. The system will again capture the speech through the microphone, perform speech recognition, and generate a new first sentence. The system will then replace the previous incorrect text with the newly recognized text and display it again on the inner screen for the speaker to confirm.

[0062] Semantic correction suggestions: If the system detects polysemous words or ambiguous semantics, it will automatically provide correction suggestions. Speakers can correct their text by selecting the system's recommended options.

[0063] It should be noted that the system saves the corrected first sentence in real time to ensure that subsequent translation operations are based on the correct text. The corrected first sentence will be displayed again on the inner screen for the speaker to confirm whether further corrections are still needed.

[0064] Specifically, the step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement further includes step S320: In this embodiment, after the speaker completes and confirms the correction of the first statement, the system will translate the corrected first statement based on preset translation rules. This process is consistent with normal translation operations, but the input text is the version corrected by the user.

[0065] It should be noted that translation rules refer to the translation operations based on preset translation rules, including translation engines, corpora, context engines, and personalized lexicons.

[0066] In this embodiment, the system dynamically adjusts the translation strategy based on the context and historical correction records to ensure the accuracy and adaptability of the translation results.

[0067] Finally, based on the adjusted translation strategy, the system translates the corrected first sentence into a second sentence in the target language. The translation result will be displayed as text on the external e-ink screen of the smart terminal for the listener to view. If needed, the system can also convert the second sentence into speech and play it to the listener through the smart terminal's speaker.

[0068] It should be noted that the corrected first sentence and its translation will be recorded in the system to optimize future translation and speech recognition models. The system will dynamically update the personalized lexicon based on the user's correction behavior to adapt to the user's specific expression habits.

[0069] Understandably, this design solves the problem of speech recognition errors and significantly improves translation accuracy and user experience through user interaction and context adaptation mechanisms.

[0070] This application proposes a visual conversation translation method and related equipment, relating to the fields of smart terminals and voice processing technology. In this context, with the acceleration of globalization, the demand for mobile translation technology is increasing. Existing solutions mainly include smartphone applications, translation headsets, and AI glasses. However, smartphone applications require both parties to share a single device, resulting in inconvenient operation and poor interactive experience. While translation headsets and AI glasses achieve device separation, they lack a visual confirmation mechanism for voice recognition, making it impossible for users to judge the accuracy of input in real time, leading to insufficient interactivity and information transparency. Furthermore, these technologies struggle to support multi-person conversation scenarios, and their portability and scenario adaptability are also limited. In contrast, this application firstly, in response to a conversation translation command, uses a microphone to capture the speaker's speech, obtaining a first sentence to be translated. Further, the first sentence is displayed on the screen of the mobile translation terminal, allowing the speaker to determine if there are any errors in the first sentence. Finally, if the speaker confirms that the first sentence is error-free, a translation operation is performed on the first sentence based on preset translation rules to obtain a second sentence, which is then displayed to the listener.

[0071] It is understood that the translation mobile terminal of this application integrates a display screen, which displays the collected content on the screen when the speaker speaks, so that the speaker can confirm whether the content collected by the mobile translation terminal is consistent with the actual content to be translated, thereby improving the accuracy of the sentence to be translated and thus improving the accuracy of the translation result.

[0072] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 The step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement further includes steps A100 to A300: Step A100: Based on preset translation rules, use a preset speech parsing model to parse the first statement and obtain the parsing result; It should be noted that a pre-defined speech parsing model is an algorithm or tool used to analyze and understand natural language text. It is typically based on Natural Language Processing (NLP) technology and is capable of performing syntactic, semantic, and pragmatic analysis on the input text (the first sentence).

[0073] It should be noted that the pre-defined speech analysis model can identify the structure of sentences, including components such as subject, predicate, and object; it can understand the meaning of sentences, including the polysemy of words and contextual relationships; and it can also analyze the actual use of sentences in specific contexts, such as whether it is a statement, question, or command.

[0074] In this embodiment, the parsing model performs word-by-word analysis on the first sentence, extracts key information, and combines the context information of the sentence to understand its specific meaning in the dialogue. Finally, it outputs the parsing results, which are usually presented in a structured form.

[0075] Step A200: Based on historical translation records and the analysis results, determine the current translation target; It should be noted that historical translation records refer to data accumulated by the system in past translation processes, including user input text, translation results, and user correction actions.

[0076] It should be noted that by analyzing historical translation records, we can obtain users' past translation habits and preferences, and then adjust our translation strategies accordingly. Furthermore, by combining this historical contextual information, we can better understand the semantics of the current sentence.

[0077] Furthermore, the parsed results are matched with similar sentences in the historical translation records to find the closest semantic pattern. Combined with contextual information from the historical records, the specific purpose of the current sentence in the dialogue and its expression in the target language are determined. Based on the user's revision history, the translation strategy is adjusted to ensure that the translation results conform to the user's habits and preferences.

[0078] Finally, the system outputs the current translation goal, which is the translation direction and strategy determined by the system, including: the semantic structure of the target language (determining how the translated sentence should be expressed), context-adaptive adjustment (selecting appropriate vocabulary and expressions based on the context), and personalized preferences (adjusting the translation results based on the user's historical correction behavior).

[0079] Specifically, the step of determining the current translation target based on historical translation records and the analysis results further includes steps A210 to A250: Step A210: Based on the parsing results, determine the first speech feature corresponding to the first statement, wherein the speech feature includes the speaker's speaking speed, the speaker's speaking emotion, and the semantics of the first statement; It should be noted that the first speaker features refer to a series of features related to the current speaker, which help the system better understand the speaker's intentions and emotional state. Specifically, these include: Speech rate: The speed at which a speaker speaks, usually measured in words per minute (WPM) or syllables per second.

[0080] Speech emotion: The emotional state of the speaker (such as happy, angry, calm, etc.) is obtained by analyzing the characteristics of the tone, intensity and rhythm of the voice.

[0081] The semantics of the first statement: the semantic information of the statement extracted from the parsing results, including keywords, semantic roles, and contextual relationships.

[0082] Step A220: Determine the second speech feature corresponding to each translation record in the historical translation record to obtain multiple sets of second speech features; It should be noted that the second speech feature refers to the speech features recorded during each translation in the historical translation record. Similar to the first speech feature, it includes: speech rate (the speaker's speech rate in the historical record), speech emotion (the speaker's emotional state in the historical record), and semantics (the semantic information of the sentences in the historical record).

[0083] Step A230: Based on any one of the multiple sets of second speech features, calculate the similarity with the first speech feature to obtain multiple sets of similarity values; It should be noted that similarity refers to the degree of matching between the features of the first statement and the features of the second statement in the historical record. Specific similarity includes: Speech rate similarity: This calculates the difference between speech rates, usually using absolute values ​​or relative errors.

[0084] Emotional similarity: Use similarity measures for emotion classification, such as cosine similarity or edit distance.

[0085] Semantic similarity: Calculated using the similarity of semantic vectors, such as cosine similarity or Jaccard similarity.

[0086] Furthermore, by combining the similarity of speech rate, emotion, and semantics, an overall similarity value is obtained.

[0087] Step A240: Based on the maximum similarity value among the multiple sets of similarity values, determine the target historical translation record that best matches the first statement in the historical translation records; It should be noted that the maximum similarity value refers to finding the maximum value among all similarity values. Then, the historical translation record corresponding to the maximum similarity value is determined, that is, the target historical translation record, which is the historical record that best matches the current first sentence.

[0088] Step A250: Determine the current translation target based on the target's historical translation records.

[0089] In this embodiment, semantic and pragmatic information in the target's historical translation records is analyzed. Based on the translation strategies and user preferences in the historical translation records, the current translation target is adjusted. Finally, the current translation target is output, including the semantic structure, pragmatic style, and personalized preferences of the target language.

[0090] Understandably, this design not only enables efficient identification of translation targets but also significantly improves translation accuracy and user experience through multi-dimensional feature analysis and the utilization of historical data.

[0091] Specifically, the step of determining the translation target based on the target's historical translation records further includes steps A251 to A253: Step A251: Determine the historical translation target corresponding to the historical translation record; It should be noted that historical translation objectives refer to the specific translation strategies and goals set by the system for each translation task based on past translation records. These objectives typically include: The semantic structure of the target language: how the translated sentence should be expressed, including grammar, vocabulary selection, etc.

[0092] Pragmatic style: the tone and formality of the translation (e.g., formal, colloquial, polite, etc.).

[0093] Contextual relevance: How the translation fits into a specific dialogue scenario or topic.

[0094] Personalized preferences: Specific preferences exhibited by users in their historical translations, such as frequently used vocabulary and expression habits.

[0095] Step A252: Determine the target language corresponding to the conversation translation instruction; It should be noted that the target language refers to the target language specified by the user in the current translation instruction. For example, a user may want to translate a piece of Chinese into English, Japanese, or another language.

[0096] Step A253: Generate the current translation target based on the target language and the historical translation targets.

[0097] It should be noted that the current translation objective refers to the specific translation strategy and goals generated by the system for the current translation task based on the target language and historical translation objectives. It combines the user's historical preferences with current needs, ensuring that the translation results are both consistent with the user's habits and adaptable to the current context.

[0098] In this embodiment, the translation strategy is adjusted according to the target language to ensure that the translation conforms to the grammar and expression habits of the target language. The tone and vocabulary selection are adjusted by combining the pragmatic style and personalized preferences in the historical translation targets. Based on the contextual information in the historical translation targets, the translation is ensured to be natural and fluent in the current dialogue scenario. If the target language differs from the language in the historical translation targets, the system needs to dynamically adjust the translation strategy to adapt to the new linguistic environment.

[0099] Understandably, this design not only enables efficient generation of translation targets but also significantly improves translation accuracy and user experience through personalized adaptation and contextual association.

[0100] Step A300: Based on the translation target, perform a translation operation on the first statement to obtain the second statement.

[0101] In this embodiment, firstly, according to the translation objective, the first sentence is converted from the source language to the target language, ensuring that the translated sentence is semantically accurate, natural, and fluent in the target language. Further, the punctuation, grammatical structure, etc., of the sentence are adjusted to conform to the writing conventions of the target language. Finally, the second sentence, i.e., the translated text, is output.

[0102] In this embodiment, the second statement is displayed on the external e-ink screen of the smart terminal for the listener to view. If the system is configured with a voice output function, the second statement can also be played to the listener through the smart terminal's speaker.

[0103] Understandably, by combining historical translation records, the system can adjust translation strategies based on users' habits and preferences, thereby improving the accuracy and adaptability of translations.

[0104] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 Before the step of parsing the first statement using a preset speech parsing model to obtain the parsing result, steps B100~B400 are further included: Step B100: Obtain sample data, and the parsing result corresponding to the sample data is the first parsing result; It should be noted that sample data refers to the labeled dataset used to train and validate the speech parsing model. This data typically contains natural language text and its corresponding correct parsing results.

[0105] It should be noted that the first parsing result is the correct parsing result corresponding to the sample data, which is usually output by manual annotation or a verified system.

[0106] Step B200: Process the sample data using the current speech parsing model to obtain a second parsing result; It should be noted that the current speech parsing model refers to a natural language processing model that is being trained and optimized, used to parse input text.

[0107] It should be noted that the second parsing result is the parsing result obtained by the current speech parsing model after processing the sample data.

[0108] Step B300: Determine whether the first parsing result is consistent with the second parsing result; It should be noted that in this embodiment, by comparing the first parsing result (correct result) and the second parsing result (model output result), the degree of matching between the two is determined, and then the performance of the current speech parsing model is determined to meet the expected standard.

[0109] Step B400: If there is a discrepancy, adjust the parameters of the current dialogue parsing model. Based on the adjusted parameters of the current dialogue parsing model, return to the step of using the current dialogue parsing model to process the sample data and obtain the second parsing result, until the first parsing result is consistent with the second parsing result, and obtain the preset dialogue parsing model.

[0110] Understandably, if the first and second parsing results are inconsistent, it indicates that the performance of the current speech parsing model needs optimization. In this case, the model parameters need to be adjusted. Furthermore, a supervised learning algorithm is used to adjust the model parameters based on the labeled information of the sample data; based on the inconsistent results, the model's weights, thresholds, and other parameters are adjusted; through multiple iterative adjustments, the model's performance is gradually improved.

[0111] Furthermore, after adjusting the parameters, the sample data is processed again using the current speech parsing model to obtain a new second parsing result. The adjusted model is then used to verify whether it can generate an output consistent with the first parsing result.

[0112] Repeat the above process until the first and second parsing results are consistent. This ensures that the script parsing model can accurately parse the sample data, thereby obtaining a reliable preset script parsing model.

[0113] Understandably, by following the steps above, we can efficiently train and optimize the speech parsing model, and ensure the accuracy and reliability of the model in practical applications, thereby improving the overall performance of the system and the user experience.

[0114] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the visualization of the conversation translation method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0115] It should be noted that all user-related data involved in this application (such as user attribute data, user behavior data, and user geographical location, etc., the data types here should be modified according to the adaptability of the solution content) were obtained with the user's permission or consent; that is to say, when this application is applied to specific products or technologies, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.

[0116] For example, when it is necessary to obtain a user's current geographical location, a location acquisition prompt can be displayed on the user's terminal. After receiving confirmation from the user regarding the location acquisition prompt, the terminal can obtain the user's current geographical location.

[0117] This application also provides a visual conversation translation device, please refer to... Figure 4 The visualized conversation translation device includes: Acquisition module 10 is used to acquire the speaker's speech using a microphone in response to a conversation translation command, thereby obtaining the first sentence to be translated; Display module 20, the display module is used to display the first statement using the display of the mobile translation terminal, so that the speaker can determine whether there is an error in the first statement; Translation module 30 is used to translate the first statement based on preset translation rules if the speaker confirms that there are no errors in the first statement, to obtain the second statement, and to display the second statement to the listener.

[0118] In one embodiment, the visualized conversation translation device further includes a correction module, which further includes: The first acquisition unit is configured to acquire the first statement after the speaker has corrected the first statement if the speaker confirms that the first statement contains an error. In one embodiment, the translation module further includes: Based on preset translation rules, a translation operation is performed on the modified first statement to obtain the second statement.

[0119] In one embodiment, the translation module further includes: The parsing unit is used to parse the first statement based on preset translation rules and a preset speech parsing model to obtain the parsing result; The first determining unit is used to determine the current translation target based on historical translation records and the analysis results; The translation unit is used to perform a translation operation on the first statement based on the translation target to obtain the second statement.

[0120] In one embodiment, the translation module further includes: The second acquisition unit is used to acquire sample data, wherein the parsing result corresponding to the sample data is the first parsing result; The data processing unit is used to process the sample data using the current speech parsing model to obtain a second parsing result; A judgment unit is used to determine whether the first parsing result is consistent with the second parsing result; The iterative training unit is used to adjust the parameters of the current dialogue parsing model if there is a discrepancy. Based on the current dialogue parsing model with adjusted parameters, it returns to the step of processing the sample data using the current dialogue parsing model to obtain a second parsing result, until the first parsing result is consistent with the second parsing result, thus obtaining a preset dialogue parsing model.

[0121] In one embodiment, the translation module further includes the following steps: The second determining unit is used to determine the first speech feature corresponding to the first statement based on the parsing result. The speech feature includes the speaker's speaking speed, the speaker's speaking emotion, and the semantics of the first statement. The third determining unit is used to determine the second speech feature corresponding to each translation record in the historical translation record, and obtain multiple sets of second speech features; The calculation unit is used to calculate the similarity between any one of the multiple sets of second speech features and the first speech features, and obtain multiple sets of similarity values. The fourth determining unit is used to determine the target historical translation record that best matches the first statement among the historical translation records based on the maximum similarity value among the multiple sets of similarity values; The fifth determining unit is used to determine the current translation target based on the target's historical translation records.

[0122] In one embodiment, the translation module further includes: The sixth determining unit is used to determine the historical translation target corresponding to the historical translation record; The seventh determining unit is used to determine the target language corresponding to the conversation translation instruction; The generation unit is used to generate the current translation target based on the target language and the historical translation targets.

[0123] The visual conversation translation device provided in this application, employing the visual conversation translation method in the above embodiments, can solve the technical problems of visual conversation translation. Compared with related technologies, the beneficial effects of the visual conversation translation device provided in this application are the same as those of the visual conversation translation method provided in the above embodiments, and other technical features in the visual conversation translation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0124] This application provides a visual conversation translation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the visual conversation translation method in Embodiment 1 above.

[0125] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a visual conversation translation device suitable for implementing embodiments of this application. The visual conversation translation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The visualized conversation translation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 5 As shown, the visual conversation translation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the visual conversation translation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the visual session translation device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows a visual session translation device with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0127] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0128] The visual conversation translation device provided in this application, employing the visual conversation translation method described in the above embodiments, can solve the technical problems. Compared with related technologies, the beneficial effects of the visual conversation translation device provided in this application are the same as those of the visual conversation translation method provided in the above embodiments, and other technical features of this visual conversation translation device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0129] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0131] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the visualized conversation translation method in the above embodiments.

[0132] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0133] The aforementioned computer-readable storage medium may be included in the visual conversation translation device; or it may exist independently and not be assembled into the visual conversation translation device.

[0134] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a visual session translation device, cause the visual session translation device to: In response to a conversation translation command, the microphone is used to capture the speaker's speech, obtaining the first sentence to be translated; The first statement is displayed on the screen of the mobile translation terminal so that the speaker can determine whether there is an error in the first statement; If the speaker confirms that the first statement is error-free, based on preset translation rules, the first statement is translated to obtain the second statement, which is then presented to the listeners.

[0135] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0138] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described visualized conversation translation method, thereby solving the technical problem of visualized conversation translation. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the visualized conversation translation method provided in the above embodiments, and will not be repeated here.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the visual conversation translation method described above.

[0140] The computer program product provided in this application can solve the technical problem of visual conversation translation. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the visual conversation translation method provided in the above embodiments, and will not be repeated here.

[0141] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A visual conversation translation method, applied to a mobile translation terminal, characterized in that, The visualized conversation translation method includes: In response to a conversation translation command, the microphone is used to capture the speaker's speech, obtaining the first sentence to be translated; The first statement is displayed on the screen of the mobile translation terminal so that the speaker can determine whether there is an error in the first statement; If the speaker confirms that the first statement is error-free, based on preset translation rules, the first statement is translated to obtain the second statement, which is then presented to the listeners.

2. The visualized conversation translation method as described in claim 1, characterized in that, Before the step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement, the method further includes: If the speaker confirms that the first statement contains an error, obtain the first statement after the speaker has corrected the first statement; The step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement further includes: Based on preset translation rules, a translation operation is performed on the modified first statement to obtain the second statement.

3. The visualized conversation translation method as described in claim 1, characterized in that, The step of performing a translation operation on the first statement based on preset translation rules to obtain the second statement further includes: Based on preset translation rules, the first statement is parsed using a preset speech parsing model to obtain the parsing result; Based on historical translation records and the analysis results, the current translation target is determined; Based on the translation target, a translation operation is performed on the first statement to obtain the second statement.

4. The visualized conversation translation method as described in claim 3, characterized in that, Before the step of parsing the first statement using a preset speech parsing model to obtain the parsing result, the method further includes: Obtain sample data, and the parsing result corresponding to the sample data is the first parsing result; The sample data is processed using the current speech parsing model to obtain a second parsing result; Determine whether the first parsing result is consistent with the second parsing result; If there is a discrepancy, adjust the parameters of the current script parsing model. Based on the adjusted parameters, return to the step of processing the sample data using the current script parsing model to obtain the second parsing result. Continue until the first parsing result is consistent with the second parsing result to obtain the preset script parsing model.

5. The visualized conversation translation method as described in claim 3, characterized in that, The step of determining the current translation target based on historical translation records and the analysis results further includes: Based on the analysis results, the first speech feature corresponding to the first statement is determined. The speech feature includes the speaker's speaking speed, the speaker's speaking emotion, and the semantics of the first statement. Determine the second speech features corresponding to each translation record in the historical translation record to obtain multiple sets of second speech features; Based on any one of the multiple sets of second speech features, a similarity score is calculated with the first speech feature to obtain multiple sets of similarity values; Based on the maximum similarity value among the multiple sets of similarity values, the target historical translation record that best matches the first statement is determined from the historical translation records; Based on the target's historical translation records, the current translation target is determined.

6. The visualized conversation translation method as described in claim 5, characterized in that, The step of determining the translation target based on the target's historical translation records further includes: Determine the historical translation target corresponding to the historical translation record; Determine the target language corresponding to the conversation translation instruction; Based on the target language and the historical translation targets, the current translation target is generated.

7. A visual conversation translation device, characterized in that, The visualized conversation translation device includes: The acquisition module is used to acquire the speaker's speech using a microphone in response to a conversation translation command, thereby obtaining the first sentence to be translated; The display module is used to display the first statement using the display of the mobile translation terminal, so that the speaker can determine whether there is an error in the first statement; The translation module is used to translate the first statement based on preset translation rules if the speaker confirms that there are no errors in the first statement, to obtain a second statement, and to display the second statement to the listener.

8. A visual conversation translation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the visualized conversation translation method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the visual conversation translation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the visual conversation translation method as described in any one of claims 1 to 6.

Images