Method, device and equipment for real-time audio translation

Abstract

The invention provides a method, a device and equipment for real-time audio translation. The method comprises the following steps: deploying a target application in a virtual machine; setting a hook anchor point at the entrance of the audio recording service so as to intercept an original sound audio stream input through the audio recording service, and collecting the original sound audio stream; setting a hook anchor point at the entrance of the audio playing service so as to intercept the original sound audio stream output to the audio playing service for playing, and collecting the original sound audio stream; calling a simultaneous interpretation service of a preset target language, and translating the collected original sound audio stream in real time to obtain a translated sound audio stream of the target language; and transmitting the target language translated audio stream to a hook anchor point, so that the target application obtains the target language translated audio stream through an audio recording service, and / or transmitting the target language translated audio stream to the hook anchor point, so that the target language translated audio stream is played through an audio playing service. According to the method, the original sound can be shielded, only the translation sound of the target language is heard, and the user experience is greatly improved.

Description

Technical Field

[0001] This application relates to the field of computer software technology, and in particular to a technology for real-time audio translation. Background Technology

[0002] Existing multimedia players or browsers with multimedia playback capabilities that support audio output, as well as social applications, video conferencing, and other scenarios with audio input and output, do not support the function of simultaneously muting the original audio and playing the translated audio. When listening to or watching audio and video in other languages ​​on devices such as computers or smartphones, language barriers can be overcome through simultaneous interpretation. However, users will hear both the original audio and the simultaneously interpreted target audio at the same time, and the target audio will be interfered with by the original audio, resulting in a poor user experience. Summary of the Invention

[0003] In order to overcome the shortcomings of the existing technology, the purpose of this application is to provide a method, apparatus and device for real-time audio translation.

[0004] According to one aspect of this application, a method for real-time audio translation is provided, applicable to an Android or iOS platform, wherein the method includes: Deploy the target application in a virtual machine; A first hook anchor point is set at the entry point of the audio recording service called by the target application to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream; A second hook anchor is set at the entry point of the audio playback service called by the target application to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application. Simultaneous interpretation services in a preset target language are invoked to translate the first original audio stream in real time to obtain the translated audio stream in the first target language, and / or the second original audio stream is translated in real time to obtain the translated audio stream in the second target language; The first target language translated audio stream is sent to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or the second target language translated audio stream is sent to the second hook anchor point so that the second target language translated audio stream can be played through the audio playback service.

[0005] Optionally, the virtual machine includes a sandbox.

[0006] Optionally, if the target application records audio and video synchronously, the method further includes: A third hook anchor is set at the entry point of the video recording service called by the target application to intercept the first video stream that is synchronized with the first original audio stream and input into the target application through the video recording service, and to collect the first video stream. The first original audio stream and the first video stream are cached, and the first original audio stream in the cache is sliced ​​into original audio segments of a preset duration, and speech recognition is performed to obtain original audio subtitles including timestamps. The original audio subtitles are translated into subtitles in the first target language, and the corresponding translated audio stream in the first target language is obtained based on the subtitles in the first target language, as well as the timestamp of synchronizing the translated audio stream in the first target language to the first original audio stream in the cache. Specifically, when the first target language translated audio stream is sent to the first hook anchor point, the first video stream in the cache is simultaneously sent to the third hook anchor point, so that the target application can simultaneously obtain the first target language translated audio stream through the audio recording service and obtain the first video stream through the video recording service.

[0007] Optionally, if the target application is playing audio and video synchronously, the method further includes: A fourth hook anchor is set at the entry point of the video playback service called by the target application to intercept the second video stream that is synchronized with the second original audio stream and output by the target application to the video playback service, and to collect the second video stream output by the target application. The second original audio stream and the second video stream are cached, and the second original audio stream in the cache is sliced ​​into original audio segments of a preset duration, and speech recognition is performed to obtain original audio subtitles including timestamps. The original audio subtitles are translated into subtitles in the second target language, and the corresponding translated audio stream in the second target language is obtained based on the translated audio stream in the second target language, as well as the timestamp of synchronizing the translated audio stream in the second target language to the second original audio stream in the cache. Specifically, when the second target language translated audio stream is sent to the second hook anchor point, the second video stream in the cache is simultaneously sent to the fourth hook anchor point so that the second video stream is played simultaneously through the video playback service while the second target language translated audio stream is played through the audio playback service.

[0008] According to another aspect of this application, an apparatus for real-time audio translation is provided, deployed on an Android or iOS platform, characterized in that the apparatus comprises: The first module is used to deploy the target application in a virtual machine; The second module is used to set a first hook anchor point at the entry point of the audio recording service called by the target application, so as to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream. The third module is used to set a second hook anchor point at the entry point of the audio playback service called by the target application, so as to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application. The fourth module is used to call the simultaneous interpretation service of the preset target language to translate the first original audio stream in real time to obtain the translated audio stream of the first target language, and / or to translate the second original audio stream in real time to obtain the translated audio stream of the second target language; The fifth module is used to send the first target language translated audio stream to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or send the second target language translated audio stream to the second hook anchor point so that the second target language translated audio stream can be obtained through the audio playback service and played.

[0009] Compared with existing technologies, this application provides a method, apparatus, and device for real-time audio translation, deployed on an Android or iOS platform. The method includes: deploying a target application in a virtual machine; setting a first hook anchor at the entry point of an audio recording service called by the target application to intercept and collect a first original audio stream input to the target application through the audio recording service; and setting a second hook anchor at the entry point of an audio playback service called by the target application to intercept and collect a second original audio stream output by the target application to the audio playback service. The application uses the output second original audio stream; it calls a simultaneous interpretation service in a preset target language to translate the collected first original audio stream in real time to obtain a translated audio stream in the first target language, and / or translates the collected second original audio stream in real time to obtain a translated audio stream in the second target language; it then sends the translated audio stream in the first target language to the first hook anchor point so that the target application can obtain the translated audio stream in the first target language through the audio recording service, and / or sends the translated audio stream in the second target language to the second hook anchor point so that the translated audio stream in the second target language can be played through the audio playback service. This application sets hook anchor points at the relevant interfaces of the target application deployed in a virtual machine that call internal or system libraries, blocking the target application's processing of the original audio, and translating the original audio into the target language in real time as the processing result of the target application. This achieves the goal of blocking the original audio and only hearing the translated audio in the target language, greatly improving the user experience. Attached Figure Description

[0010] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of a method for real-time audio translation according to one aspect of this application is shown; Figure 2 A schematic diagram of an apparatus for real-time audio translation according to another aspect of this application is shown; The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0011] The present application will now be described in further detail with reference to the accompanying drawings.

[0012] In a typical configuration of various embodiments of this application, the method execution entity, each trusted party of the system, and / or each module of the device may include one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0013] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0014] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0015] To further illustrate the technical means adopted and the effects achieved in this application, the technical solution of this application will be clearly and completely described below in conjunction with the accompanying drawings and preferred embodiments.

[0016] Figure 1 The illustration shows a method for real-time audio translation according to one aspect of this application, applicable to Android or iOS platforms, wherein one embodiment of the method includes: S101 deploys the target application in a virtual machine; S102 sets a first hook anchor at the entry point of the audio recording service called by the target application to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream; S103 sets a second hook anchor at the entry point of the audio playback service called by the target application to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application; S104 calls the simultaneous interpretation service of the preset target language to translate the first original audio stream in real time to obtain the translated audio stream of the first target language, and / or translates the second original audio stream in real time to obtain the translated audio stream of the second target language. S105 The first target language translated audio stream is sent to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or the second target language translated audio stream is sent to the second hook anchor point so that the second target language translated audio stream can be played through the audio playback service.

[0017] In the application scenarios of this application, the target application is a multimedia application that can record original audio (e.g., a live recording application), play original audio (e.g., a live playback application), or support both (e.g., multimedia social media applications).

[0018] In this embodiment, a virtual machine is installed in the smart device 100 that has deployed the Android or iOS platform. In step S101, the target application can be deployed in the virtual machine without modifying the target application, so as to achieve rapid deployment and provide overall security.

[0019] The normal processing flow of the target application includes: obtaining the original audio stream as audio input by calling its internal or system library's audio recording service; processing the obtained original audio stream; playing it locally by calling its internal or system library's audio playback service; and / or transmitting the processed audio stream over the network. If real-time translation from the original audio to the translated audio is required, simultaneous interpretation or other related equipment is needed. However, the original audio and the translated audio will play simultaneously, and the original audio cannot be muted, leading to mutual interference and a poor user experience.

[0020] The smart device 100 includes, but is not limited to, smartphones, tablets, multimedia wearable devices, etc. This is merely an example; other existing or future devices and / or resource platforms applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0021] Continuing in this embodiment, in step S102, a first hook anchor is set at the entry point of the audio recording service called by the target application to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream.

[0022] If the target application supports audio stream recording, a hook anchor (first hook anchor) is set at the entry point of the audio recording service called by the target application to intercept the original audio stream (first original audio stream) input to the target application through the audio recording service, so as to block the channel through which the target application originally obtains the first original audio stream through the audio recording service and collect the first original audio stream.

[0023] Continuing in this embodiment, in step S103, a second hook anchor is set at the entry point of the audio playback service called by the target application to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application.

[0024] If the target application supports audio streaming, a hook anchor (second hook anchor) is set at the entry point of the audio playback service called by the target application. When the target application outputs the native audio stream (second original audio stream) to the audio playback service, the hook anchor is intercepted to block the channel through which the target application originally played the second original audio stream through the audio playback service, and the second original audio stream output by the target application is collected.

[0025] Continuing in this embodiment, in step S104, a simultaneous interpretation service for a preset target language is invoked to translate the first original audio stream in real time to obtain a translated audio stream in the first target language, and / or the second original audio stream is translated in real time to obtain a translated audio stream in the second target language.

[0026] After acquiring the original audio stream (first original audio stream and / or second original audio stream), the simultaneous interpretation service of the preset target language can be invoked to translate the acquired original audio stream in real time, thereby obtaining the target language translated audio stream (first target language translated audio stream and / or second target language translated audio stream).

[0027] Continuing in this embodiment, in step S105, the first target language translated audio stream is sent to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or the second target language translated audio stream is sent to the second hook anchor point so that the second target language translated audio stream can be played through the audio playback service.

[0028] Specifically, if the target application supports audio stream recording, the target language translated audio stream (first target language translated audio stream) obtained through simultaneous interpretation can be sent to the first hook anchor point, replacing the first original audio stream, allowing the target application to obtain and process the first target language translated audio stream. If the target application supports audio stream playback, the target language translated audio stream (second target language translated audio stream) obtained through simultaneous interpretation can be sent to the second hook anchor point, replacing the second original audio stream, and the second target language translated audio stream can be played through the audio playback service called by the target application.

[0029] Optionally, in step S101, the virtual machine includes a sandbox.

[0030] In order to provide greater security and convenience, target applications can also be deployed in a sandbox.

[0031] Optionally, if the target application records audio and video synchronously, the method further includes: A third hook anchor is set at the entry point of the video recording service called by the target application to intercept the first video stream that is synchronized with the first original audio stream and input into the target application through the video recording service, and to collect the first video stream. The first original audio stream and the first video stream are cached, and the first original audio stream in the cache is sliced ​​into original audio segments of a preset duration, and speech recognition is performed to obtain original audio subtitles including timestamps. The original audio subtitles are translated into subtitles in the first target language, and the corresponding translated audio stream in the first target language is obtained based on the subtitles in the first target language, as well as the timestamp of synchronizing the translated audio stream in the first target language to the first original audio stream in the cache. In step S105, when the first target language translated audio stream is sent to the first hook anchor, the first video stream in the cache is simultaneously sent to the third hook anchor, so that the target application can simultaneously obtain the first target language translated audio stream through the audio recording service and obtain the first video stream through the video recording service.

[0032] In this optional embodiment, if the target application synchronously records audio and video, then in step S102, a hook anchor (third hook anchor) is also set at the entry point of the video recording service called by the target application to intercept the video stream (first video stream) input to the target application through the video recording service that is synchronized with the first original audio stream, thereby blocking the channel through which the target application originally obtained the first video stream and acquiring the first video stream. Then, the acquired first original audio stream and first video stream are cached, and the cached first original audio stream is sliced ​​into original audio segments of a preset duration for speech recognition to obtain original audio subtitles including timestamps. Then, the obtained original audio subtitles including timestamps are translated into target language subtitles (first target language subtitles), and the first target language subtitles are processed by text-to-audio conversion to obtain the corresponding first target language translated audio stream, and the timestamp of synchronizing the first target language translated audio stream to the cached first original audio stream is also included. In step S105, when the first target language translated audio stream is sent to the first hook anchor, the first video stream in the cache is also sent to the third hook anchor, so that the target application can obtain the first target language translated audio stream through the audio recording service it calls and obtain the first video stream through the video recording service it calls, thereby achieving synchronization between the first video stream and the first target language translated audio stream.

[0033] Optionally, if the target application is playing audio and video synchronously, the method further includes: A fourth hook anchor is set at the entry point of the video playback service called by the target application to intercept the second video stream that is synchronized with the second original audio stream and output by the target application to the video playback service, and to collect the second video stream output by the target application. The second original audio stream and the second video stream are cached, and the second original audio stream in the cache is sliced ​​into original audio segments of a preset duration, and speech recognition is performed to obtain original audio subtitles including timestamps. The original audio subtitles are translated into subtitles in the second target language, and the corresponding translated audio stream in the second target language is obtained based on the translated audio stream in the second target language, as well as the timestamp of synchronizing the translated audio stream in the second target language to the second original audio stream in the cache. In step S105, when the second target language translated audio stream is sent to the second hook anchor point, the second video stream in the cache is simultaneously sent to the fourth hook anchor point so that the second video stream is played simultaneously through the video playback service while the second target language translated audio stream is played through the audio playback service.

[0034] In this optional embodiment, if the target application synchronously plays audio and video, in step S103, a hook anchor (fourth hook anchor) is also set at the video playback service entry point called by the target application to intercept the video stream (second video stream) output by the target application to the video recording service for playback synchronized with the second original audio stream, thereby blocking the channel through which the target application originally played the second video stream via the video playback service, and capturing the second video stream. Then, the captured second original audio stream and second video stream are cached, and the cached second original audio stream is sliced ​​into original audio segments of a preset duration for speech recognition to obtain original audio subtitles including timestamps. Then, the obtained original audio subtitles including timestamps are translated into target language subtitles (second target language subtitles), and the second target language subtitles are processed by text-to-audio conversion to obtain the corresponding second target language translated audio stream, and the timestamp of synchronizing the second target language translated audio stream to the cached second original audio stream is also recorded. In step S105, when the second target language audio stream is sent to the second hook anchor, the second video stream in the cache is also sent to the fourth hook anchor. This is so that when the second target language audio stream is played through the audio playback service called by the target application, the second video stream is played through the video playback service called by the application, thereby achieving synchronization between the first video stream and the first target language audio stream.

[0035] In the above optional embodiments, if the last sentence of the obtained original audio subtitles including timestamps is incomplete, the original audio segment corresponding to the last sentence should be merged into the next set of original audio segments for processing. Correspondingly, a video of the same duration should also be synchronously cut from the cached video stream and processed synchronously with the next set of original audio segments to ensure the synchronization of the target language translated audio stream and the video stream.

[0036] The method for real-time audio translation provided by the above embodiments and / or optional embodiments enables the target application to process the target language translated audio stream obtained by real-time translation of the original audio stream while blocking the original audio stream, thus avoiding the user hearing the original audio and the target language translated audio at the same time and greatly improving the user experience.

[0037] Figure 2 The diagram illustrates an apparatus for real-time audio translation according to another aspect of this application, deployed on an Android or iOS platform, wherein, in one embodiment, the apparatus includes: The first module 210 is used to deploy the target application in a virtual machine; The second module 220 is used to set a first hook anchor at the entry point of the audio recording service called by the target application, so as to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream. The third module 230 is used to set a second hook anchor at the entry point of the audio playback service called by the target application, so as to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application. The fourth module 240 is used to call the simultaneous interpretation service of the preset target language to translate the first original audio stream in real time to obtain the translated audio stream of the first target language, and / or to translate the second original audio stream in real time to obtain the translated audio stream of the second target language. The fifth module 250 is used to send the first target language translated audio stream to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or send the second target language translated audio stream to the second hook anchor point so that the second target language translated audio stream can be obtained through the audio playback service and played.

[0038] In this embodiment, the device is deployed on a smart device 100 running an Android or iOS platform. The first module 210 of the device deploys the target application in a virtual machine for rapid deployment and overall security. If the target application supports audio stream recording, the second module 220 of the device sets a hook anchor (first hook anchor) at the entry point of the audio recording service called by the target application to intercept the original audio stream (first original audio stream) input to the target application through the audio recording service, thus blocking the target application's original channel for obtaining the first original audio stream through the audio recording service and capturing the first original audio stream. If the target application supports audio stream playback, the third module 230 of the device sets a hook anchor (second hook anchor) at the entry point of the audio playback service called by the target application to intercept the original audio stream (second original audio stream) output by the target application to the audio playback service for playback, thus blocking the target application's original channel for playing the second original audio stream through the audio playback service and capturing the second original audio stream output by the target application. Through the fourth module 240 of the device, a simultaneous interpretation service in a preset target language can be invoked to translate the acquired original audio stream in real time, resulting in a translated audio stream in the target language (a translated audio stream in the first target language and / or a translated audio stream in the second target language). Through the fifth module 250 of the device, if the target application supports audio stream recording, the translated audio stream in the target language (the translated audio stream in the first target language) obtained through simultaneous interpretation can be sent to the first hook anchor point, replacing the first original audio stream, allowing the target application to obtain and process the translated audio stream in the first target language; if the target application supports audio stream playback, the translated audio stream in the target language (the translated audio stream in the second target language) obtained through simultaneous interpretation can be sent to the second hook anchor point, replacing the second original audio stream, and the translated audio stream in the second target language can be played through the audio playback service invoked by the target application.

[0039] Optionally, if the target application supports simultaneous audio and video recording, the second module 220 of the device also sets a hook anchor (third hook anchor) at the entry point of the video recording service called by the target application. This intercepts the video stream (first video stream) input to the target application through the video recording service, which is synchronized with the first original audio stream, thereby blocking the target application's original channel for obtaining the first video stream through the video recording service and capturing the first video stream. Then, the captured first original audio stream and first video stream are cached, and the cached first original audio stream is sliced ​​into original audio segments of a preset duration for speech recognition to obtain original audio subtitles including timestamps. The obtained original audio subtitles including timestamps are then translated into target language subtitles (first target language subtitles), and the first target language subtitles are processed by text-to-audio conversion to obtain the corresponding first target language translated audio stream, and the timestamp of synchronizing the first target language translated audio stream to the cached first original audio stream. Specifically, through the fifth module 250 of the device, when the first target language translated audio stream is sent to the first hook anchor point, the first video stream in the cache is also sent to the third hook anchor point simultaneously, so that the target application can obtain the first target language translated audio stream through the audio recording service it calls and obtain the first video stream through the video recording service it calls, thereby realizing the synchronization of the first video stream and the first target language translated audio stream.

[0040] Optionally, if the target application supports synchronized audio and video playback, the third module 230 of the device also sets a hook anchor (fourth hook anchor) at the video playback service entry point called by the target application. This intercepts the video stream (second video stream) synchronized with the second original audio stream output by the target application to the video recording service, blocking the channel through which the target application originally played the second video stream via the video playback service, and then captures the second video stream. Next, the captured second original audio stream and second video stream are cached, and the cached second original audio stream is sliced ​​into original audio segments of a preset duration for speech recognition to obtain original audio subtitles including timestamps. Then, the obtained original audio subtitles including timestamps are translated into target language subtitles (second target language subtitles), and the second target language subtitles are processed by text-to-audio conversion to obtain the corresponding second target language translated audio stream, and the timestamp of synchronizing the second target language translated audio stream to the cached second original audio stream. Specifically, through the fifth module 250 of the device, when the second target language translated audio stream is sent to the second hook anchor point, the second video stream in the cache is also sent to the fourth hook anchor point simultaneously, so that when the second target language translated audio stream is played through the audio playback service called by the target application, the second video stream is played through the video playback service called by the target application, thereby achieving synchronization between the first video stream and the first target language translated audio stream.

[0041] In the above-described apparatus embodiments and / or optional embodiments, any components of the apparatus not mentioned herein are the same as those in the aforementioned related method embodiments and / or optional embodiments, and will not be repeated here.

[0042] According to another aspect of this application, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement some or all of the foregoing method embodiments and / or optional embodiments.

[0043] It should be noted that the method embodiments and / or optional embodiments in this application do not strictly limit the order of execution of each step, as long as the method embodiments and / or optional embodiments can solve the defects existing in the prior art, achieve the inventive purpose of this application, and obtain beneficial effects. The method embodiments and / or optional embodiments in this application can be implemented in software and / or combinations of software and hardware. The software program involved in this application can be executed by a processor to implement the steps or functions of the above embodiments. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium.

[0044] Furthermore, part or all of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. The program instructions invoking the methods of this application may be stored in a fixed or removable recording medium, and / or transmitted via data streams in broadcast or other signal carrying media, and / or stored in the working memory of a computer device operating according to the program instructions.

[0045] According to another aspect of this application, a device for real-time audio translation is also provided. The device includes: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to run part or all of the methods and / or technical solutions of the foregoing embodiments.

[0046] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0047] In this application, when terms such as "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" are used, the indicated orientation and / or positional relationship is based on the orientation and / or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation. Furthermore, some of the above terms, in addition to indicating orientation or positional relationship, can also be used to indicate other meanings; for example, the term "upper" can also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application according to the specific circumstances. Furthermore, the terms "installation," "setup," "equipped with," "connection," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection via an intermediate medium; and they can refer to an internal connection between two devices, components, or constituent parts. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0048] Furthermore, the terms "first," "second," etc., are primarily used to distinguish different devices, units, modules, elements, circuits, or components (which may be the same or different in specific type and construction), and are not intended to indicate or imply the relative importance, order, and / or quantity of the indicated devices, units, modules, elements, circuits, or components. Unless otherwise stated, "a plurality of" means two or more.

[0049] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device through software and / or hardware.

Claims

1. A method for real-time audio translation, applied to Android or iOS platforms, characterized in that, The method includes: Deploy the target application in a virtual machine; A first hook anchor point is set at the entry point of the audio recording service called by the target application to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream; A second hook anchor is set at the entry point of the audio playback service called by the target application to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application. Simultaneous interpretation services in a preset target language are invoked to translate the first original audio stream in real time to obtain the translated audio stream in the first target language, and / or the second original audio stream is translated in real time to obtain the translated audio stream in the second target language; The first target language translated audio stream is sent to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or the second target language translated audio stream is sent to the second hook anchor point so that the second target language translated audio stream can be played through the audio playback service.

2. The method according to claim 1, characterized in that, The virtual machine includes a sandbox.

3. The method according to claim 1, characterized in that, If the target application records audio and video synchronously, the method further includes: A third hook anchor is set at the entry point of the video recording service called by the target application to intercept the first video stream that is synchronized with the first original audio stream and input into the target application through the video recording service, and to collect the first video stream. The first original audio stream and the first video stream are cached, and the first original audio stream in the cache is sliced ​​into original audio segments of a preset duration, and speech recognition is performed to obtain original audio subtitles including timestamps. The original audio subtitles are translated into subtitles in the first target language, and the corresponding translated audio stream in the first target language is obtained based on the subtitles in the first target language, as well as the timestamp of synchronizing the translated audio stream in the first target language to the first original audio stream in the cache. Specifically, when the first target language translated audio stream is sent to the first hook anchor point, the first video stream in the cache is simultaneously sent to the third hook anchor point, so that the target application can simultaneously obtain the first target language translated audio stream through the audio recording service and obtain the first video stream through the video recording service.

4. The method according to claim 1, characterized in that, If the target application plays audio and video synchronously, the method further includes: A fourth hook anchor is set at the entry point of the video playback service called by the target application to intercept the second video stream that is synchronized with the second original audio stream and output by the target application to the video playback service, and to collect the second video stream output by the target application. The second original audio stream and the second video stream are cached, and the second original audio stream in the cache is sliced ​​into original audio segments of a preset duration, and speech recognition is performed to obtain original audio subtitles including timestamps. The original audio subtitles are translated into subtitles in the second target language, and the corresponding translated audio stream in the second target language is obtained based on the translated audio stream in the second target language, as well as the timestamp of synchronizing the translated audio stream in the second target language to the second original audio stream in the cache. Specifically, when the second target language translated audio stream is sent to the second hook anchor point, the second video stream in the cache is simultaneously sent to the fourth hook anchor point so that the second video stream is played simultaneously through the video playback service while the second target language translated audio stream is played through the audio playback service.

5. A device for real-time audio translation, deployed on an Android or iOS platform, characterized in that, The device includes: The first module is used to deploy the target application in a virtual machine; The second module is used to set a first hook anchor point at the entry point of the audio recording service called by the target application, so as to intercept the first original audio stream input to the target application through the audio recording service and collect the first original audio stream. The third module is used to set a second hook anchor point at the entry point of the audio playback service called by the target application, so as to intercept the second original audio stream output by the target application to the audio playback service and collect the second original audio stream output by the target application. The fourth module is used to call the simultaneous interpretation service of the preset target language to translate the first original audio stream in real time to obtain the translated audio stream of the first target language, and / or to translate the second original audio stream in real time to obtain the translated audio stream of the second target language; The fifth module is used to send the first target language translated audio stream to the first hook anchor point so that the target application can obtain the first target language translated audio stream through the audio recording service, and / or send the second target language translated audio stream to the second hook anchor point so that the second target language translated audio stream can be obtained through the audio playback service and played.

6. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement part or all of the method as described in any one of claims 1 to 4.

7. A device for real-time audio translation, characterized in that, The device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform some or all of the operations of the method as described in any one of claims 1 to 4.

Images