Audio translation method, electronic equipment, storage medium and product

Abstract

The invention relates to an audio translation method, electronic equipment, a storage medium and a product, and relates to the technical field of computers. The audio translation method comprises the following steps: translating original audio acquired by user equipment; determining whether the original audio is from outside or inside the user equipment; in response to the original audio coming from the outside of the user equipment, playing translated audio of the original audio by default; and in response to the original audio coming from the interior of the user equipment, defaulting not to play the translated audio of the original audio, and displaying the translated text of the original audio.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an audio translation method, electronic device, storage medium and product. Background Art

[0002] With the development of machine learning technology, users can use various translation tools to translate text, documents, and voice. For example, an application can collect the user's first language voice and translate it into the second language. The translation result is played to the user or displayed on the screen of the user's device. Summary of the invention

[0003] According to some embodiments of the present disclosure, an audio translation method is provided, including: translating original audio acquired by a user device; determining whether the original audio comes from outside or inside the user device; in response to the original audio coming from outside the user device, playing a translated audio of the original audio by default; in response to the original audio coming from inside the user device, not playing the translated audio of the original audio by default, and displaying a translated text of the original audio.

[0004] According to some embodiments of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the audio translation method of any embodiment described in the present disclosure based on instructions stored in the memory.

[0005] According to some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the audio translation method of any embodiment described in the present disclosure is performed.

[0006] According to some embodiments of the present disclosure, a computer program product is provided. When the computer program product is executed on a computer, the computer implements the audio translation method of any embodiment described in the present disclosure.

[0007] Other features, aspects and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The following describes embodiments of the present disclosure with reference to the accompanying drawings. It should be understood that the accompanying drawings described below only relate to some embodiments of the present disclosure and do not constitute a limitation to the present disclosure.

[0009] Figure 1 A flowchart of an audio translation method according to some embodiments of the present disclosure is shown.

[0010] Figure 2(a) shows a schematic diagram of the interactive interface of an intelligent agent with translation function.

[0011] Figure 2 (b) shows a schematic diagram of the communication interface with the intelligent agent.

[0012] Figure 3 (a) to 3(c) show schematic diagrams of an interactive interface of a translation function according to some embodiments of the present disclosure.

[0013] Figure 4 (a) and 4(b) show schematic diagrams of the user interface when the agent is running in the background.

[0014] Figure 5 A schematic diagram of an interface showing a translated text display method according to some embodiments of the present disclosure is shown.

[0015] Figure 6 A schematic structural diagram of an audio translation device according to some embodiments of the present disclosure is shown.

[0016] Figure 7 A block diagram of an electronic device according to some embodiments of the present disclosure is shown.

[0017] Figure 8 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0018] It should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not necessarily drawn according to the actual proportional relationship. The same or similar reference numerals are used in the various drawings to represent the same or similar parts. Therefore, once an item is defined in one drawing, it may not be further discussed in subsequent drawings. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present disclosure. It should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein.

[0020] It should be understood that the various steps recorded in the method embodiments of the present disclosure can be performed in different orders, and / or performed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown in the execution. The scope of the present disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments should be interpreted as being merely exemplary and do not limit the scope of the present disclosure.

[0021] The term “including” and its variations used in the present disclosure are intended to be open terms that include at least the following elements / features but do not exclude other elements / features, that is, “including but not limited to.” The term “based on” means “at least partly based on.”

[0022] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units. Unless otherwise specified, the concepts of "first", "second", etc. are not intended to imply that the objects described in this way must be in a given order in time, space, ranking, or any other manner.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings, but the present disclosure is not limited to these specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. In addition, in one or more embodiments, specific features, structures or characteristics can be combined in any suitable manner that will be clear from the present disclosure by a person of ordinary skill in the art.

[0027] After analysis, it was found that in real-time translation scenarios, users may have multiple translation needs. For example, when a user is talking face to face with others, he hopes that the application can help translate the conversation content; or when the user listens to or watches foreign content on the user device, he hopes to obtain translation results in real time. Some translation applications have relatively simple functions. Some translation applications have more functions, but users are required to manually set them or manually switch to the corresponding functions. Since the translation process needs to be started as soon as possible in scenarios such as real-time translation, cumbersome operations will lead to slow start of translation, and thus the translation efficiency of related technologies is relatively low.

[0028] The present disclosure provides an audio translation method, electronic device, storage medium and product. The embodiments of the present disclosure automatically determine the presentation mode of the translation result according to the source of the original audio, so as to improve the translation efficiency and enhance the user experience.

[0029] Figure 1 FIG. 2 shows a flow chart of an audio translation method according to some embodiments of the present disclosure. Figure 1 As shown, the audio translation method of this embodiment includes steps S102 to S108.

[0030] In step S102, the original audio acquired by the user device is translated.

[0031] The user device can monitor the acquired audio as the original audio with the user's authorization, for example, after the user enables the real-time translation function.

[0032] After obtaining the original audio, the translation function can be used for translation. For example, the content in the audio can be identified to obtain the original text in the original language used by the original audio, and then the original text is translated into a text (which can be called a translated text) in another language (i.e., a language different from the original language). In the process of translation, a translation model can be used for processing, and the present disclosure will not go into details about the specific translation process. When determining the language used by the translated text, it can be determined based on a default setting or according to the original language.

[0033] The translation process can be performed locally on the user device. Alternatively, the original audio or original text can be sent to other devices (such as a server, etc.) for processing, and the user device then receives the translation result returned by the other device, such as a translated text or translated audio.

[0034] The user device can monitor the acquired audio stream and translate the audio stream in real time. That is, the user device can translate the content of each original audio segment acquired in the audio stream, and then acquire the next original audio segment and continue translating. Thus, the effect of real-time translation can be achieved.

[0035] In step S104, it is determined whether the original audio comes from the outside or the inside of the user device.

[0036] The ways in which a user device obtains audio include the outside of the user device and the inside of the user device. Audio outside the user device refers to the sound in the environment collected by an external audio input device of the user device (such as an external microphone), such as the sound of a face-to-face conversation between a user and other people, the sound of an offline meeting attended by the user, and so on. Audio inside the user device refers to the sound emitted by the user device, such as the sound played by an application running in the user device, including the sound of played audio and video, the sound from other users in an online meeting attended by the user, and so on. The audio inside the user device can be collected through an internal audio input device (such as an internal microphone). It can be determined whether the collected original audio comes from the outside or inside of the user device based on the type of device or interface that collects the audio.

[0037] Sometimes, audio can be obtained from both the outside and the inside of the user device. In this case, the user can be actively asked to determine which way to translate the original audio obtained. Alternatively, the user can also pre-set the currently used translation scene or the monitoring channel to determine in advance whether to translate the external audio or the internal audio.

[0038] In step S106, in response to the original audio coming from outside the user device, the translated audio of the original audio is played by default.

[0039] The translated audio is audio obtained by converting the translated text into speech, that is, the translated audio includes the translation result of the content of the original audio.

[0040] Playing the translated audio by default means that after the translation function is started, the translated audio will be played to the user if the user does not change the playback settings. If needed, if the user does not want to listen to the audio during use, the playback function can also be turned off manually. In addition, if needed, the translated text can also be displayed on the user's device for the user to read and refer to.

[0041] In the case where the original audio comes from outside the user device, it means that the user is currently paying attention to the sound outside the user device. Therefore, the translated audio can be played out so that the user can quickly obtain the translation result without looking at the user device screen. For example, if the user does not understand language A, when the user hears a foreigner speaking language A on site, the user device obtains the foreigner's speech audio, translates it, and plays it to the user. In this way, the user can quickly obtain the foreigner's speech content and respond.

[0042] The translated audio can be played through the user device or an external audio device of the user device (such as headphones, speakers, wearable devices, etc.).

[0043] In step S108, in response to the original audio coming from inside the user device, the translated audio of the original audio is not played by default, and the translated text of the original audio is displayed.

[0044] By default, the translation audio is not played. This means that after the translation function is started, the translation audio will not be played to the user unless the user changes the playback settings. If the user still wants to listen to the audio during use, the playback function can be manually turned on as needed.

[0045] When the original audio comes from the user device, it means that the user is currently paying attention to the sound emitted by the user device itself. Since the user device is playing sound and occupies at least one channel, the translation audio is not played by default, but the translation result is displayed. Therefore, the user can obtain the translation result through the screen of the user device.

[0046] For example, if a user does not understand language A, when the user uses a user device to listen to or watch content in language A, the user device can display the translated text to the user instead of playing the translated audio. This can maintain the user's immersion in the content being listened to or watched, and convey the translation result in the form of text.

[0047] The above embodiment uses whether the audio collected by the user device is from the inside or outside of the user device to determine the default presentation mode for the translation result. Specifically, for the external original audio, the translation result is played; for the internal translated audio, the translation result is not played but displayed. Thus, the embodiment of the present disclosure can use a translation result presentation mode adapted to the current translation scenario based on the source of the audio, so that the user can quickly obtain the translation result adapted to the current scenario. Therefore, the efficiency of translation can be improved and the user experience can be enhanced.

[0048] As described above, the original audio collection process can be triggered by the translation function. For example, in response to the audio translation function being triggered, the external and internal audio of the user device are monitored. The audio translation function can be triggered in a variety of ways, such as starting a translation application, or a translation function in an application, etc.

[0049] With the development of artificial intelligence technology, users can quickly obtain information by interacting with intelligent agents. An intelligent agent refers to an intelligent virtual object based on a machine learning model, which can automatically determine a response to a user's input based on the user's input. For example, for an intelligent agent with a translation function, a user can interact with the intelligent agent. After receiving the input content (such as text, voice, audio, etc.), the intelligent agent can obtain a translation result with the help of a machine learning model and return the translation result to the user. The following exemplarily describes several embodiments of obtaining translation results by triggering an intelligent agent.

[0050] In some embodiments, an agent is invoked through a designated external audio device to trigger the translation function. The invoked agent is provided by an application in the user device and is an agent with a translation function. The agent then runs in the foreground or background of the user device to obtain the original audio monitored by the user device, translate it, and then return the translation result to the user.

[0051] The external audio device of the user device includes, for example, headphones, speakers, wearable devices, etc., which can be connected to the user device by wire or wirelessly. The agent can be run in response to the startup of the external audio device. For example, after the Bluetooth headset is taken out of the headset box and the Bluetooth headset is successfully connected to the mobile phone, the agent can run automatically. In addition, the agent can also be awakened according to the user's instructions after the external audio device starts working. After the audio device is started, the audio device or the user device listens to the user's instructions, which can be voice instructions, gesture instructions, trigger instructions for the physical or virtual controls of the user device, etc. Taking the headset as an example, after the user wears the headset and the headset is connected to the user device, the user can say a specified voice instruction, such as "Hello, agent A". After recognizing the content of the voice, agent A and the translation function provided by agent A are started.

[0052] When the user wakes up the agent through a designated external audio device, if the feedback mode of the translation result is set to play the translated audio, the translated audio can be played through the external audio device by default. Of course, the translated audio can also be played through other channels, and the playback channel and playback mode can be modified according to the user's operation.

[0053] In some embodiments, the external and internal audio of the user device is monitored by triggering the translation function of the agent. For example, the user can trigger the translation function of the agent by triggering a control corresponding to the translation function of the agent in the application. Figure 2 (a) shows a schematic diagram of the interactive interface of agent B with translation function. Figure 2As shown in (a), in the interface 21, the user can have a text conversation with the agent B. In response to the user triggering the control 211 in the interface 21, the real-time translation function can be turned on. After the real-time translation function is turned on, the agent B will obtain the audio monitored by the user device and translate it.

[0054] In some embodiments, when the agent is in a real-time call with the user and in translation mode, the external and internal audio of the user device is monitored. If the agent only has a translation mode, when a real-time call (such as a voice call or a video call) with the agent is started, it is assumed to be in translation mode. If the agent has multiple modes, during a call between the user and the agent, in response to the user selecting the call mode as translation mode, the translation function is turned on. Figure 2 (b) shows a schematic diagram of the communication interface with agent C. Figure 2 As shown in (b), in the call interface 22, in response to the selection control "scenario selection" 221 that triggers the call mode, the user can determine the current scenario as the real-time translation scenario, that is, select the real-time translation mode.

[0055] The above methods are merely exemplary of several translation function triggering methods. Those skilled in the art may also select other methods as needed.

[0056] In a translation scenario, sometimes multiple parties involved in a conversation need to obtain translation results, and sometimes only the current user wants to obtain translation results in a language that he or she does not master. The embodiments of the present disclosure can provide multiple translation methods to meet the needs in different scenarios.

[0057] Some embodiments of the present disclosure can achieve mutual translation between multiple languages. In some embodiments, the language of the spoken content in the original audio is identified; the spoken content in the first language is translated into the second language; and the spoken content in the second language is translated into the first language. That is, there are multiple languages ​​in the original audio, and in this embodiment, each language can be translated. However, although this embodiment is illustrated by taking the first language and the second language as examples, it is not limited to only two languages ​​in the speaking scene. For example, in the case of a third language, the spoken content in the third language can be further translated into the first language.

[0058] Identifying the language to which each content belongs can be achieved by using a speech recognition model or a classification model, which will not be elaborated here.

[0059] In the case of multi-language translation, the translation results of the content in multiple languages ​​can be played. The multiple languages ​​can be all the languages ​​involved in the original audio, or multiple languages ​​supported by the translation function. For example, for the spoken content in the first language, the translated audio in the second language is played; for the spoken content in the second language, the translated audio in the first language is played. Accordingly, in the case where there are more languages ​​in the original audio, the audio of the translation results in other languages ​​can also be played. For example, in the case where there is a third language, the spoken content in the third language can be further translated into the first language, and the translated audio in the first language can be played for the spoken content in the third language.

[0060] This method of playing the translation results of content in multiple languages ​​can be applied to scenarios such as face-to-face translation. For example, two people who use different languages ​​to chat face to face do not understand each other's language. Then the content of language 1 in the original audio can be translated into language 2 for playback, so that the person using language 2 can understand what the other person is saying; the content of language 2 can be translated into language 1 for playback, so that the person using language 1 can understand what the other person is saying.

[0061] In some embodiments, when the user device is not connected to headphones, that is, when the translated audio is played through a speaker or a sound box, the translation results of the content in multiple languages ​​can be played by default. Thus, when multiple people involved in the original audio can hear the translation results, everyone can hear the translation results in the language they use. Of course, this setting can be modified according to the needs of the user.

[0062] In the case of multi-language translation, only the content of the language selected by the user in the translation result can be played. In some embodiments, the user's selection of the target language is received, and the target language includes one or more; the audio corresponding to the content of the target language in the translation result is played. For example, the language used by the user is language 1, and the user also participates in the conversation, the user's speech content is translated into language 2, and the speech content of other users is translated into language 1, then the translation results played only include the translation results of the speech content of other users. Thus, in some translation scenarios, when only the user himself needs to listen to the translation result and other speakers do not need it, only the audio of the target language selected by the user in the translation result can be played.

[0063] In some embodiments, when the user device is connected to headphones, that is, when the translated audio is played through the headphones, the translated audio including the target language selected by the user can be played by default. Because when the user listens through headphones, it is mostly unnecessary to play the audio to other users who are not wearing headphones, so only the translation results that the user can understand are played, reducing the amount of audio playback, and improving the user experience in this scenario. Of course, this setting can be modified according to the needs of the user.

[0064] Some embodiments of the present disclosure may also translate only the content in the language specified by the user, that is, it is not necessary to translate all the content. In some embodiments, the user's selection of a target language may be received; the content in the original audio that is not in the target language may be translated into the target language. For example, if the original audio includes content in language 1 and language 2, and the user selects language 1 as the target language, then only the content in language 1 in the original audio needs to be translated, and the content in language 2 may not need to be translated. Thus, the amount of content that needs to be translated can be saved according to the user's needs, which saves computing resources and improves processing efficiency.

[0065] In the case where the original audio comes from inside the user device, the translated text of the original audio is displayed. In some embodiments, in response to the original audio coming from outside the user device, the translated text of the original audio may also be displayed. The translated text may be displayed in real time as the translation proceeds, for example, when the user device is in the case of a bright screen. Of course, in the case where the original audio comes from outside the user device, the user can obtain the translation result by listening to the audio, and the user device may also be in a state of turning off the screen, or the translation function may be in a state of running in the background. At this time, the translated text may be recorded in the background, and the translated text may be displayed in response to the user's viewing operation of the translated text.

[0066] There may be multiple ways to display the translated text, which are described below in conjunction with several embodiments.

[0067] In some embodiments, in response to the agent running in the foreground, the translated text of the original audio is displayed in the interactive interface of the agent. For example, for an agent with a translation function, such as a conversational agent, the translation result can be displayed in an interactive interface such as an interface for a message conversation with the agent or an interface for a real-time call.

[0068] In the real-time translation scenario, the user device will continuously monitor multiple original audios, that is, monitor video streams. During translation, multiple original audios can be segmented and the playback results of each segment can be displayed or played one by one. Different segments can be divided according to duration, content or speaker.

[0069] Taking the division by duration as an example, the duration of the original audio or the translated audio corresponding to each translated text does not exceed the first threshold. Taking the division by content volume as an example, the content volume of each translated text is less than the threshold.

[0070] Taking dividing paragraphs according to speakers as an example, one or more speakers in the original audio and the translated text of the original audio corresponding to each speaker can be displayed. Figure 3(a) shows a schematic diagram of an interactive interface of a translation function according to some embodiments of the present disclosure. Figure 3 As shown in (a), in the interactive interface 30, the translation results can be displayed. For example, a text including multiple paragraphs is displayed, paragraph 301 corresponds to speaker 1, and paragraph 302 corresponds to speaker 2. Speaker 1 uses Chinese, and paragraph 301 includes original Chinese speech content 3011 and English translation result 3012 of 3011. Speaker 2 uses English, and paragraph 302 includes original English speech content 3021 and Chinese translation result 3022 of 3021.

[0071] In some embodiments, in the interactive interface of the agent, in response to the original audio coming from the inside of the user device and the user triggering a certain paragraph, the translated audio corresponding to the paragraph is played. Since the translated audio is not played by default when the original audio comes from the inside of the user device. Therefore, the translated audio that the user wants to listen to can be played by triggering a certain paragraph. For example, in response to the triggering of the play mark 3023 of paragraph 302, the translated audio of the paragraph can be played. During the playback of the translated audio, the mark 3023 can be switched to other styles to indicate that the paragraph associated with it is playing.

[0072] When the intelligent agent is running in the background, the translated text may or may not be displayed.

[0073] In some embodiments, in response to the agent running in the background, when the original audio comes from outside the user device, the translation mark is displayed in the interface of the user device. When the original audio comes from outside the user device, the user's translation needs can be met by playing the translated audio, and displaying the translated text is not necessary. Therefore, when the user needs to use the user device to view other functions besides the translation function, only the translation mark can be displayed.

[0074] Figure 4 (a) and 4(b) show schematic diagrams of the user interface when the agent is running in the background. Figure 4 As shown in (a), interface 41 displays other interfaces except the interface of the translation function, and the interface may include a translation mark 411 to prompt the user that the real-time translation is currently in progress. As needed, in response to triggering the translation mark 411, the interface of the translation function may be displayed, such as displaying the translated text. Figure 4 The interface 42 shown in (b) is the lock screen interface of the user device. On the lock screen interface 42, a logo 421 may be displayed to remind the user that the real-time translation is currently in progress.

[0075] In some embodiments, when the agent is running in the background, a floating layer is displayed on the upper layer of the currently displayed interface; the translation text of the latest paragraph of the original audio is displayed in the floating layer. Thus, the translation result can be displayed to the user while reducing the impact on the user's browsing of other application content. In particular, when the currently displayed interface is the interface of the audio source, the translation text displayed on the floating layer can play a role similar to subtitles, helping the user to quickly understand the content currently being listened to or watched.

[0076] Figure 5 A schematic diagram of an interface showing a translated text display method according to some embodiments of the present disclosure is shown. Figure 5 The interface shown is a playback interface 50 of a podcast, and the playback content of the podcast interface is the original audio, and the content is the technology news (Tech News) broadcast by host X (HostX). The language used in the content of the podcast is not the user's language. Thus, the user can turn on the real-time translation function. A floating layer 501 is displayed on the upper layer of the podcast interface, which includes the translated text of the podcast content so that the user can keep abreast of the content of the podcast. At the same time, the translation function can run in the background, and the user can continue to operate the playback of the podcast. Therefore, when the user efficiently obtains the translation result, it can not affect the operating efficiency of the application currently concerned.

[0077] The above-mentioned embodiments exemplarily describe the playing of the translated audio and the display of the translated text. The following exemplarily describes several application scenarios to which the embodiments of the present disclosure are applicable.

[0078] In an exemplary application scenario, the user device obtains audio from outside the user device, and the user device is not currently connected to headphones. It can be determined that the current scenario is a face-to-face translation scenario. Each language in the original audio is identified, and multiple languages ​​are translated, and the translation results of each language are played. Multiple people in the face-to-face translation scenario can understand each other's speech.

[0079] In an exemplary application scenario, the user device obtains audio from outside the user device, and the user device is currently connected to headphones. It can be determined that the current scenario is that the user needs to obtain the translation results of the sound in the environment, such as participating in an offline foreign language meeting, speech, etc., or when the user is talking to others, the user does not understand the other party's language, but the other party understands the user's language. Identify the content in the original audio that the user does not understand, and play the corresponding translation results of these contents to the user through headphones, and the translation results are all in the user's language.

[0080] For the above two scenarios, the translated text can be recorded in the background and displayed to the user when the user views it.

[0081] In an exemplary application scenario, the user device obtains audio from the user device. It can be determined that the current user needs to translate the content of other foreign language applications in the user device. At this time, the translation result of the audio can be displayed in the interface through a floating layer.

[0082] The above application scenarios exemplarily provide some default configurations for feeding back translation results to users, however, those skilled in the art may also adjust these configurations as needed. Furthermore, users may also make manual changes.

[0083] For example, in Figure 3 In the interactive interface 30 shown in (a), the control 303 is used to set the translation mode. The currently used translation mode is Chinese-English translation, that is, the Chinese content in the original audio is translated into English, and the English content is translated into Chinese. In response to the triggering of the control 303, the following may be displayed: Figure 3 The floating layer 3030 in (b) includes multiple translation modes for the user to choose from, so that the user can easily determine the target language for translation.

[0084] For example, in Figure 3 In the interactive interface 30 shown in (a), the control 304 is used to set the play mode. The currently used play mode is not to play the translated audio. In response to the triggering of the control 304, the following may be displayed: Figure 3 The floating layer 3040 in (c) includes multiple play modes for the user to choose from, so that the user can easily determine which language the translation results are played by default.

[0085] The above describes the methods of various embodiments of the present disclosure. The following describes the apparatus for executing the methods of various embodiments.

[0086] Figure 6 FIG. 4 shows a schematic diagram of the structure of an audio translation device according to some embodiments of the present disclosure. Figure 6 As shown, the audio translation device 60 of this embodiment includes: a translation module 601, configured to translate the original audio obtained by the user device; a determination module 602, configured to determine whether the original audio comes from outside or inside the user device; an interaction module 603, configured to play the translated audio of the original audio by default in response to the original audio coming from outside the user device; in response to the original audio coming from inside the user device, not play the translated audio of the original audio by default, and display the translated text of the original audio.

[0087] In some embodiments, the audio translation device 60 also includes: a monitoring module 604, which is configured to monitor external and internal audio of the user device in response to the audio translation function being triggered, and the triggering method of the audio translation function includes at least one of the following: waking up the intelligent agent through a designated external audio device, the translation function of the intelligent agent is triggered, and the intelligent agent is in a real-time call with the user and is in translation mode.

[0088] In some embodiments, the translation module 601 is further configured to: identify the language of the spoken content in the original audio; translate the spoken content in the first language into the second language; and translate the spoken content in the second language into the first language.

[0089] In some embodiments, the interaction module 603 is further configured to: play the translated audio in the second language for the spoken content in the first language; and play the translated audio in the first language for the spoken content in the second language.

[0090] In some embodiments, the interaction module 603 is further configured to: receive a user's selection of a target language, the target language including one or more; and play audio corresponding to the content of the target language in the translation result.

[0091] In some embodiments, the user translation module 601 is further configured to: receive a user's selection of a target language; and translate the non-target language content in the original audio into the target language.

[0092] In some embodiments, the interaction module 603 is further configured to: in response to the original audio coming from outside the user device, display the translated text of the original audio.

[0093] In some embodiments, the interaction module 603 is further configured to: in response to the intelligent agent running in the foreground, display the translated text of the original audio in the interactive interface of the intelligent agent; in response to the intelligent agent running in the background, display the translation mark in the interface of the user device when the original audio comes from outside the user device.

[0094] In some embodiments, the interaction module 603 is further configured to: display a floating layer on the upper layer of the currently displayed interface when the intelligent agent is running in the background; and display the translated text of the latest paragraph of the original audio in the floating layer.

[0095] In some embodiments, the translated text includes multiple paragraphs, the content volume of each paragraph is less than a threshold, and the interaction module 603 is further configured to: in the interactive interface of the intelligent agent, in response to the original audio coming from inside the user device and the user triggering a paragraph, play the translated audio corresponding to the paragraph.

[0096] In some embodiments, the interaction module 603 is further configured to: display one or more speakers in the original audio, and the translated text of the original audio corresponding to each speaker.

[0097] Figure 7 A block diagram of an electronic device according to some embodiments of the present disclosure is shown. The memory 71 is used to store one or more computer-readable instructions. The memory 71 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), flash memory. The memory 71 can, for example, store an operating system, an application, a boot loader (Boot Loader), a database, and other programs, and can also store various applications and various data.

[0098] The processor 72 is used to run computer-readable instructions to implement the method described in any of the above embodiments. The specific implementation of each step of the method can be referred to the above embodiments, and the repeated parts will not be repeated here.

[0099] The processor 72 may be configured to execute Figures 1 to 5 The processor 72 may be embodied as various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) may be an X87 or ARM architecture, etc.

[0100] The processor 72 and the memory 71 may communicate with each other directly or indirectly. For example, the processor 72 and the memory 71 may communicate with each other via a network. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The processor 72 and the memory 71 may also communicate with each other via a system bus, which is not limited in the present disclosure.

[0101] It should be noted that Figure 7 The components of the electronic device 7 shown are only exemplary and non-restrictive. The electronic device 7 may also have other components according to actual application requirements. The processor 72 may control other components in the electronic device 7 to perform desired functions.

[0102] The electronic device 7 may be implemented by software, firmware and / or hardware, and may be integrated into a device installed with relevant application programs.

[0103] Figure 8A block diagram of an electronic device according to some other embodiments of the present disclosure is shown. Figure 8 The electronic device 8 shown may be a computer system with a dedicated hardware structure, which can execute corresponding functions when a relevant application program is installed.

[0104] Electronic devices include, but are not limited to, mobile terminals such as smart phones, laptops, personal digital assistants (PDA), tablet computers (Tablet Personal Computer, Tablet PC), PMP (portable multimedia player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., as well as fixed terminals such as digital televisions, desktop computers, etc.

[0105] like Figure 8 As shown, the central processing unit (CPU) 81 performs various processes according to the program stored in the read-only memory (ROM) 82 or the program loaded from the storage part 88 to the random access memory (RAM) 83. In the RAM 83, data required when the CPU 81 performs various processes, etc. is stored as needed. The central processing unit is only exemplary, and it can also be other types of processors, such as the various processors described above. The ROM 82, RAM 83 and the storage part 88 can be various forms of computer-readable storage media. It should be noted that although Figure 8 ROM 82, RAM 83 and storage section 88 are shown separately in FIG. 8 , but one or more of them may be combined or located in the same or different memory or storage modules.

[0106] The CPU 81, the ROM 82, and the RAM 83 are connected to one another via a bus 84. To the bus 84, an input / output interface 85 is also connected.

[0107] The following components are connected to the input / output interface 85: an input section 86, such as a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output section 87, including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage section 88, including a hard disk, a magnetic tape, etc.; and a communication section 89, including a network interface card such as a LAN card, a modem, etc. The communication section 89 allows communication processing to be performed via a network such as the Internet. It is easy to understand that although Figure 8 Some of the electronic devices 8 are shown to communicate via a bus 84, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless networks and wired networks.

[0108] A drive 810 is also connected to the input / output interface 85 as needed. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 810 as needed so that a computer program read therefrom is installed into the storage section 88 as needed.

[0109] When the above-described series of processing is implemented by software, a program constituting the software can be installed from a network such as the Internet or a storage medium such as the removable medium 811 .

[0110] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which, when the computer program product is run on a computer, enables the computer to implement the method described in any of the aforementioned embodiments. The computer program product includes a computer instruction carried on a computer-readable medium, containing a program code for executing the method shown in the flowchart. In such an embodiment, the computer instruction can be downloaded and installed from the network through the communication part 89, or installed from the storage part 88, or installed from the ROM 82. When the computer program is executed by the CPU 81, the method of the embodiment of the present disclosure is executed.

[0111] It should be noted that, in the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by an instruction execution system, apparatus, or device or for use in conjunction with an instruction execution system, apparatus, or device.

[0112] The computer readable medium may be a computer readable storage medium, or a computer readable signal medium, or any combination of the two.

[0113] Computer-readable storage media include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device. Computer instructions are stored on a computer-readable storage medium, and when the instructions are executed by a processor, the method described in any of the foregoing embodiments is implemented.

[0114] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0115] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0116] In some embodiments, a computer program is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method described in any of the above embodiments. For example, the instructions may be embodied as computer program codes.

[0117] In embodiments of the present disclosure, computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In situations involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The functions described above may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0120] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. An audio translation method, comprising: Translate the original audio obtained by the user's device; Determining whether the original audio comes from outside or inside the user device; In response to the original audio being from outside the user device, playing the translated audio of the original audio by default; In response to the original audio coming from inside the user device, the translated audio of the original audio is not played by default, and the translated text of the original audio is displayed.

2. The audio translation method according to claim 1, further comprising: In response to the audio translation function being triggered, external and internal audio of the user device are monitored, and the triggering method of the audio translation function includes at least one of the following: the intelligent agent is awakened through a designated external audio device, the translation function of the intelligent agent is triggered, and the intelligent agent is in a real-time call with the user and is in translation mode.

3. The audio translation method according to claim 1, wherein: The translating of the original audio obtained by the user device includes: Identifying a language of spoken content in the original audio; Translate speech from the first language into the second language; The utterance in the second language is translated into the first language.

4. The audio translation method according to claim 3, wherein: Playing the translated audio of the original audio includes: Playing the translated audio in the second language for the spoken content in the first language; The translated audio in the first language is played for the spoken content in the second language.

5. The audio translation method according to claim 3, wherein: Playing the translated audio of the original audio includes: receiving a target language selection by the user, wherein the target language includes one or more; The audio corresponding to the content in the target language in the translation result is played.

6. The audio translation method according to claim 1, wherein: The translating of the original audio obtained by the user device includes: receiving a target language selection by the user; The content in the original audio which is not in the target language is translated into the target language.

7. The audio translation method according to claim 1, further comprising: In response to the original audio being from outside the user device, displaying a translated text of the original audio.

8. The audio translation method according to claim 1 or 7, wherein: The display of the translated text of the original audio comprises: In response to the intelligent agent running in the foreground, displaying the translated text of the original audio in the interactive interface of the intelligent agent; In response to the agent running in the background, a translation mark is displayed in the interface of the user device when the original audio comes from outside the user device.

9. The audio translation method according to claim 1 or 7, wherein: The display of the translated text of the original audio comprises: When the agent is running in the background, a floating layer is displayed on the upper layer of the currently displayed interface; The translated text of the latest segment of the original audio is displayed in the floating layer.

10. The audio translation method according to any one of claims 1 to 7, wherein: The translation text includes a plurality of paragraphs, the content amount of each paragraph is less than a threshold, and the translation method further includes: In the interactive interface of the intelligent agent, in response to the original audio coming from inside the user device and the user triggering a certain paragraph, the translated audio corresponding to the paragraph is played.

11. The audio translation method according to claim 1 or 7, wherein displaying the translated text of the original audio comprises: One or more speakers in the original audio and a translation text of the original audio corresponding to each speaker are displayed.

12. An electronic device comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the audio translation method according to any one of claims 1 to 11 based on instructions stored in the memory.

13. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the audio translation method according to any one of claims 1 to 11 is implemented.

14. A computer program product, when the computer program product is run on a computer, enables the computer to implement the audio translation method according to any one of claims 1 to 11.

Images