Translation method and device and electronic equipment

Abstract

The embodiment of the invention provides a translation method and device and electronic equipment, in the translation method, after the electronic equipment receives first content input by a user and the tone set by the user, the first content is translated into second content, the first content uses a first language, and the second content uses a second language; the second language is different from the first language, and the second content is related to the setting information of the user, so that a corresponding translation result can be obtained by combining the first content input by the user and the tone set by the user, and the translated second content can be changed along with the change of the tone set by the user.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent terminal technology, and in particular to a translation method, device and electronic device. Background Art

[0002] In recent years, with the advancement of speech translation technology, some devices and applications based on this technology have gradually developed, such as face-to-face translation, simultaneous interpretation and / or call translation, etc. In these application scenarios, the user inputs a speech stream from the source language and sets the target language to be translated, and the electronic device can translate it end-to-end into a speech stream in the target language or text in the target language.

[0003] However, the existing voice translation only mechanically translates the content of the voice stream input by the user into a voice stream or text of the target language, so the translation result is relatively rigid and the translation quality is poor. Summary of the invention

[0004] The embodiments of the present application provide a translation method, apparatus, and electronic device. The embodiments of the present application also provide a computer-readable storage medium, so that the electronic device can obtain a corresponding translation result by combining the first content input by the user and the tone set by the user, and can realize that the translated second content changes with the tone set by the user.

[0005] In a first aspect, an embodiment of the present application provides a translation method, including: receiving first content input by a user, the first content using a first language; receiving setting information of the user, the setting information of the user including a tone set by the user; translating the first content into second content, the second content using a second language, the second language being different from the first language, and the second content being related to the setting information of the user.

[0006] In the above translation method, after the electronic device receives the first content input by the user and the tone set by the user, it translates the first content into the second content, wherein the first content uses a first language, the second content uses a second language, the second language is different from the first language, and the second content is related to the user's setting information, so that the corresponding translation result can be obtained by combining the first content input by the user and the tone set by the user, and the translated second content can change with the change of the tone set by the user.

[0007] In one possible implementation manner, the user setting information also includes a timbre set by the user.

[0008] In one possible implementation, the translating the first content into the second content includes: obtaining a sequence of the first content according to the first content; and obtaining a feature vector corresponding to the setting information according to the setting information of the user; encoding the sequence of the first content to obtain a first latent variable vector; fusing the first latent variable vector and the feature vector corresponding to the setting information to obtain a sequence of the second content; and obtaining a translated second content according to the sequence of the second content.

[0009] In one possible implementation, when the user's setting information also includes the timbre set by the user, obtaining the feature vector corresponding to the setting information according to the user's setting information includes: performing feature conversion on the tone and timbre set by the user to obtain the feature vector corresponding to the setting information. This implementation can achieve that in the process of translating the first content into the second content, the translation is performed in combination with the tone and timbre set by the user, so that the second content obtained by translation changes with the change of the tone set by the user, and when the second content is voice content, the timbre of the voice content obtained by translation is the timbre set by the user.

[0010] In one possible implementation, the first content includes first speech content; obtaining the sequence of the first content according to the first content includes: converting the first speech content into a first spectrogram feature sequence according to its length; encoding the first spectrogram feature sequence to obtain a second latent variable vector; decoding the second latent variable vector, and clustering the sequence obtained by decoding to obtain a speech semantic unit sequence of the first content. This implementation can realize the conversion of speech into discrete speech semantic units (units), that is, converting the original speech input by the user into a unit sequence of the first content that retains the corresponding speech semantic information.

[0011] In one possible implementation, the first content includes a first text content; obtaining the sequence of the first content according to the first content includes: encoding the first text content to obtain a third latent variable vector; decoding the third latent variable vector to obtain the sequence of the first content; wherein the sequence of the first content includes a text sequence of the first content or a sequence of speech semantic units of the first content. This implementation can realize the conversion of the first text content input by the user into a text sequence of the first content or a sequence of speech semantic units of the first content, thereby preparing for translating the first text content into a second text content or a second speech content.

[0012] In one possible implementation, the second content includes second text content, and the sequence of the second content includes: a text sequence of the second content; and obtaining the translated second content according to the sequence of the second content includes: decoding the text sequence of the second content to obtain the translated second text content.

[0013] In one possible implementation, the second content includes a second voice content, and the sequence of the second content includes a sequence of voice semantic units of the second content; before obtaining the translated second content according to the sequence of the second content, the method further includes: obtaining a second spectrogram feature sequence corresponding to the timbre set by the user; obtaining the translated second content according to the sequence of the second content includes: voice encoding the second spectrogram feature sequence to obtain a voice feature matrix; and voice semantic encoding the voice semantic unit sequence of the second content to obtain a feature matrix of the second content; fusing the voice feature matrix with the feature matrix of the second content; and decoding the fused feature matrix to obtain the second voice content. This implementation can synthesize the second content audio with the voice semantic unit sequence of the second content according to the required audio features. At the same time, the audio features of the reference input audio are maintained, that is, the voice semantic information of the second content expressed by the user is combined with the reference audio set by the user to synthesize the corresponding voice translation audio.

[0014] In one possible implementation, the second content includes a second text content, and the sequence of the second content includes a sequence of speech and semantic units of the second content; the step of obtaining the translated second content according to the sequence of the second content includes: performing speech and semantic encoding on the sequence of speech and semantic units of the second content to obtain a feature matrix of the second content; and decoding the feature matrix of the second content to obtain the second text content. This implementation can obtain the second text content according to the sequence of speech and semantic units of the second content, thereby translating the speech content into text content.

[0015] In one possible implementation manner, after translating the first content into the second content, the method further includes: displaying the first content and the second content.

[0016] In one possible implementation, after displaying the first content and the second content, the process further includes: obtaining the first content modified by the user and / or the setting information modified by the user; and re-executing the step of receiving the first content input by the user and its subsequent steps. That is, after the electronic device displays the first content and the second content, if the user wants to modify the first content and / or the user wants to change the tone, the user can modify the first content and / or the setting information, the electronic device obtains the first content modified by the user and / or the setting information modified by the user, and then re-executing the step of receiving the first content input by the user and its subsequent steps, and re-obtaining the translated second content.

[0017] In one possible implementation, after displaying the first content and the second content, the further step includes: obtaining and displaying the second content modified by the user. That is, after the electronic device displays the first content and the second content, if the user is not satisfied with the translated second content, or the user wants to modify the second content, the user can modify the second content, and the electronic device obtains the second content modified by the user and displays it.

[0018] In a second aspect, an embodiment of the present application provides a translation device, which is included in an electronic device, and has the function of implementing the electronic device behavior in the first aspect and the possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving unit and a translation unit, etc.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the method provided in the first aspect.

[0020] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0023] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flow chart of speech translation provided for the existing related technology;

[0025] Figure 2 Another flow chart of speech translation provided for the existing related technology;

[0026] Figure 3 A schematic diagram of a text editing module provided by the prior art;

[0027] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0028] Figure 5(a) to Figure 5(c) A schematic diagram of a call translation scenario on a mobile phone provided by an embodiment of the present application;

[0029] Figure 6 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of a speech translation module provided in one embodiment of the present application;

[0031] Figure 8 A schematic diagram of an implementation of an S2U module provided in one embodiment of the present application;

[0032] Fig. 9 A schematic diagram of an implementation of a U2U module provided in one embodiment of the present application;

[0033] Fig.10 A schematic diagram of an implementation of a U2S module provided in one embodiment of the present application;

[0034] Fig.11 A flowchart of a translation method provided for one embodiment of the present application;

[0035] Fig.12 A flowchart of a translation method provided for another embodiment of the present application;

[0036] Fig.13 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application;

[0037] Fig.14 A schematic diagram of the structure of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0038] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0039] Existing voice translation only mechanically translates the content of the voice stream input by the user into a voice stream or text in the target language, and the user's translation experience is poor. However, unlike ordinary text translation, the input of voice translation contains richer language information, such as the user's timbre and / or tone, etc. Integrating this information into the translation process will greatly enhance the user's translation experience. In particular, the user's tone can also play a strong optimization role in the translated text content. Taking Chinese-English translation as an example, assuming that the user enters a voice with a text content of "go away", when the user enters it in a calm tone, the electronic device should translate it into a calmer English text "Leave me alone.", but when the user enters it in an angry tone, the electronic device should translate it into a more angry English text "Get out!". To give another example, suppose a user records a voice message with the text content "You should come with me too." When the user records it in a declarative tone, the electronic device should translate it as "You should come with me." When the user records it in an interrogative tone, the electronic device should translate it as "Do you want to come with me?".

[0040] At a higher level, users have the need to control the tone of the translated text. Taking the above “go away” as an example, the user may enter this sentence in an angry tone, but hope to translate it to the recipient in a calmer way. At this time, the user can set his tone to “calm”, and the electronic device will translate “go away” into “Leave me alone” in a “calm” tone.

[0041] Figure 1 A flow chart of speech translation provided for the existing related technology, such as Figure 1 As shown, after receiving the source language speech, the source language tone is first identified by the tone recognition module, the source language text is identified by the text recognition module, and then the source language text is translated into the target language text by the text translation module. The tone corresponding to each target language text is obtained by the tone text matching module, and it is input into the speech synthesis module together with the target language text to obtain the target language speech. In the specific implementation, there may be multiple source language speech, multiple tone information obtained by recognition, and multiple target language texts obtained by translation, and the role of the tone text matching module is to determine which tone information each target language text has. In summary, the tone information in this scheme is only used for the synthesis of the target language speech and does not participate in the translation process.

[0042] In the above scheme, since the tone information is not involved in the translation process, the target language texts translated under different tones are the same, which does not provide a good user experience. In addition, since the tone information comes from the recognition of the source language speech, this scheme can only maintain the tone of the user when recording the speech, and the user cannot specify the tone of the translation result by himself.

[0043] Figure 2 Another flow chart of speech translation provided for the existing related technology, Figure 2 The solution shown supports the user to input text and select the tone of the text, that is, the user edits the text into text under the corresponding tone by selecting the tone, and synthesizes it into speech under the corresponding tone. Figure 2 The main function of the text editing module in is to display the text entered by the user in different shapes according to the tone selected by the user, but it does not involve changes in the content, such as Figure 3 As shown, Figure 3 A schematic diagram of a text editing module provided by the prior art. Figure 3 The top table shows the tone types that users can choose. The text in row (a) means "This is a really good idea." Row (b) shows the text display effect under the user's normal tone. At this time, the stress is evenly distributed, so the size and shape of each font are relatively uniform, and the color is relatively positive. Row (c) shows the text display result under the yin-yang tone. At this time, the stress is concentrated on the two words "this" and "really", so the shape of the first two words is elongated, and the color of the font is also relatively negative.

[0044] However, this solution only supports one language and does not include a translation process; moreover, this solution only supports text input, does not support user voice input and automatically extract text from user voice, and in this solution, the text editing method only involves editing of text display effects, such as color, shape and / or size, etc., and does not involve editing of text content.

[0045] In summary, integrating the user's tone into the speech translation process in a controllable manner will bring a better translation experience. However, in the existing related technologies, during the translation process, the user cannot ensure that the translated text meets the tone they need, that is, the user cannot specify different text translation outputs under different tones; in addition, during the user's speech translation process, in the solutions provided by the existing related technologies, the tone only affects the speech synthesis or text display effect, but cannot change the content of the translated text.

[0046] Based on the above problems, the embodiment of the present application proposes a translation method, which is a translation scheme with controllable voice features, and can accurately give translation results in different tones according to user settings. Specifically, the translation result can include two parts: translated text and translated voice, in which the text content can be adaptively changed according to the tone set by the user, and the corresponding voice also has the voice features (timbre and / or tone, etc.) set by the user.

[0047] The translation method provided in the embodiments of the present application can be applied to electronic devices, wherein the above-mentioned electronic devices can be smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks or personal digital assistants (PDA) and other devices; the embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0048] For example, Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown as follows: Figure 4 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0049] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0050] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0051] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0052] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0053] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0054] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0055] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device 100 through the power management module 141.

[0056] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0057] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0058] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0059] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0060] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0061] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0062] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0063] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0064] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0065] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0066] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0067] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0068] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0069] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0070] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0071] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0072] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0073] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0074] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0075] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0076] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0077] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0078] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0079] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0080] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0081] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0082] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0083] For ease of understanding, the following embodiments of the present application will be described in detail with Figure 4 Taking the electronic device with the structure shown as an example, the translation method provided in the embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0084] The embodiment of the present application provides an end-to-end voice translation method with controllable voice features, which can combine the user's setting information and voice input information to generate a controllable personalized voice translation result - combining the first content input by the user and the user's setting information to obtain the corresponding translation result. Specifically, the translated second content changes with the change of tone. Among them, the first content uses the first language, and the second content uses the second language; the second language is different from the first language. Specifically, the first language and the second language can be different types of languages ​​and / or languages ​​with different accents, etc. For example, the first language can be Chinese and the second language can be English.

[0085] The translation method provided in the embodiment of the present application can be applied to all electronic devices 100 with translation functions. Specifically, the translation method provided in the embodiment of the present application can be applied to call translation in mobile phones, PCs or headset devices (after each person in the call completes a paragraph, the paragraph is translated), face-to-face translation (when the two parties in the conversation communicate, the translation is performed at the same time) or simultaneous interpretation (translation and distribution while listening to the conversation in real time) and other scenes. In addition, the translation method provided in the embodiment of the present application can also be embedded in chat software to meet the user's entertainment and / or creation needs, for example, the user says a sentence in a normal tone, and then selects a spoof tone to translate it into other languages, and creates video content or audio content. The translation method provided in the embodiment of the present application can also be used in artificial intelligence (AI) subtitle-related applications, for example, according to the tone of the actor's lines in the movie, it is translated into a text under a specific tone. More advanced, the user can edit the AI ​​subtitles by himself, set the tone according to his understanding of the actor's tone, and then the electronic device 100 translates the user-edited subtitles into the text of the second language the user wants according to the set tone and the user-edited subtitles.

[0086] Take call translation as an example. Figure 5(a) to Figure 5(c) A schematic diagram of a call translation scenario on a mobile phone provided in an embodiment of the present application, which can be used based on the following interaction logic during user use:

[0087] 1. The user enters the call translation interface, which may be shown in FIG. 5( a ).

[0088] 2. The user can enter the user setting interface by clicking the "Go to Settings" icon 51 in the interface shown in FIG5(a) to set the tone, timbre and / or second language and other related information of the call translation. Of course, the user can also choose not to set and use the default configuration.

[0089] 3. After setting up, turn on the call translation service.

[0090] 4. In the call translation service, the user makes a normal call, inputs the audio in the first language into the microphone, and receives the audio in the first language through the audio device.

[0091] 5. After the user inputs the voice, the call translation interface displays the bilingual text results (first language and second language) of the input voice, and sends the user's voice input to the receiving user, that is, the other party of the call translation, after voice translation.

[0092] 6. After the user device receives the voice reply from the other party, the call translation interface displays the bilingual text broadcast of the voice (first language and second language), and broadcasts the audio after voice translation to the user.

[0093] 7. After multiple rounds of conversation, the call ends and the user terminates the call translation service.

[0094] FIG5(b) shows the translation result of the user's input voice when the user does not set the tone. As can be seen from FIG5(b), the first voice content input by the user is "Come and see me right away!", and after voice translation, the translation result of the second content is "Get your ass over here and meet me"; after the second content is sent to the other party of the call translation, the voice input by the other party of the call translation is "I quit!", and the translation result is "I quit!"

[0095] FIG5(c) shows the translation result of the user's input voice when the user sets the tone to "peaceful". As can be seen from FIG5(c), the first voice content input by the user is "Come to see me right away!". When the tone is set to "peaceful", the translation result of the second content is "Come and see me now" after voice translation; and after the second content is sent to the other party of the call translation, the voice input by the other party of the call translation is "Sorry, I'll be quick.". After voice translation, the translation result is "I'm very sorry, I'll be quick."

[0096] By comparing FIG. 5( b ) and FIG. 5( c ), it can be clearly seen that when the tone is different, the result of speech translation is different, so that the translation results under different tones can be accurately given according to the user settings.

[0097] Figure 6 A schematic diagram of a system architecture provided for an embodiment of the present application, Figure 6 The user setting module and the voice translation module are two key components. The above system architecture is explained below.

[0098] 1. The speech translation module takes the user's speech as input and can perform speech translation according to the user's settings. The speech translation module can combine the text information and the tone information set by the user to obtain the corresponding translation result. Specifically, the translated text content changes with the tone. The "speech translation module" in the embodiment of the present application can include any system, module, deep model and / or statistical model that can achieve similar functions as above.

[0099] 2. The user settings module means that the system architecture can select the user's real-time personalized settings or default settings as appropriate before processing the user's input voice. When the user chooses to enter the personalized settings, the system architecture can interact with the user and update the user settings; if the user chooses not to enter the personalized settings, the user input voice will be used by default for user settings.

[0100] 3. The user setting module may include three sub-modules: a second language selection module, a tone selection module and a timbre selection module.

[0101] Figure 7 A schematic diagram of the structure of a speech translation module provided in one embodiment of the present application, Figure 7 In the example, the speech translation module is constructed based on a deep learning model and trained by stochastic gradient descent. Specifically, the speech translation module is an end-to-end model of a cascade of multiple encoder-decoder structure models.

[0102] Figure 7 The speech translation module shown is a unit-based speech translation solution, involving data in three modes: speech, text, and discrete speech semantic units. Unit is a sequence of time length, such as [10 5 77...999], and the numbers inside represent the features at that time (for ease of understanding, the speech semantic information can be classified and put into different "baskets", and this number is the number of the frame, that is, the features at the first frame are in the "basket" of 10). Figure 7 The speech translation module shown mainly involves five main modules, namely speech to speech semantic unit converter (speech to unit, S2U), text to speech semantic unit converter (text to unit, T2U), speech semantic unit translator (unit to unit, U2U), speech semantic unit audio synthesizer (unit to speech, U2S) and speech semantic unit text synthesizer (unit to text, U2T).

[0103] Among them, S2U is used to convert the input speech into a discrete sequence of speech semantic units, T2U is used to convert the input text into a discrete sequence of speech semantic units, U2U is used to convert the input speech semantic unit sequence of the first content into a speech semantic unit sequence of the second content, U2S is used to synthesize audio using the speech semantic unit sequence, and U2T is used to synthesize text using the speech semantic unit sequence.

[0104] Figure 7 The speech translation module shown is mainly composed of three parts: T2U / S2U, U2U and U2T / U2S. During training, the three parts of the model are trained independently using the stochastic gradient descent method. The details are as follows:

[0105] (1) First, the S2U model is trained. The pre-trained Hubert model of the automatic speech recognition (ASR) task is used as the initialization, and the audio divided into time frames is input (40ms is divided into one frame), the feature number in the audio is extracted, and the feature map on the time series is output. Then, the K-means clustering algorithm is used to cluster the feature map on the time series obtained previously with k=1000 (it can be understood as selecting one of the 1000 categories as its own feature for the feature map obtained in each frame before), and the discretized unit sequence on the time series corresponding to the speech is obtained. Then, the speech is used as input, and the corresponding discretized unit sequence on the time series is used as output. The encoder-decoder model with the encoder parameters initialized by the pre-trained Hubert model is supervised and trained (the loss between the model generation and the label is compared, and the model parameters are updated by back propagation, which will not be described in detail later), and the S2U model is obtained, so that the conversion from speech to unit can be completed.

[0106] (2) Based on the trained S2U model, a <speech-text-unit> triple dataset is established (the S2U training data contains audio and its original text label). The training is performed using a high-fidelity generative adversarial network (HiFiGAN) model architecture based on generative adversarial learning, with speech and the corresponding unit as input and the corresponding speech as output. The S2U model is obtained, so that the corresponding speech can be synthesized from the unit.

[0107] (3) Based on the trained S2U model, a <speech-text-Unit> triple dataset is established (the S2U training data contains audio and its original text label). A transformer model with an encoder-decoder architecture is used, with text as input and the corresponding unit as output, and the model is trained using supervised learning. The T2U model is obtained, which can complete the conversion from text to unit.

[0108] (4) Based on the T2U and S2U models trained above, a large amount of data is used to construct aligned unit data, that is, the unit representation of aligned translation data. For example, the translation data of <Chinese-English> is converted into the translation data of <Chinese unit-English unit>. Among them, the source of the unit is not limited to model reasoning and / or clustering, and the form of the unit is not limited to language (units can be established by language or only one unit). Based on the aligned unit data, combined with the feature vector (embedding) of the user input information, the mBart model of the encoder-decoder architecture is used for supervised training, and the cross-attention mechanism is used for information fusion to obtain the U2U model after training, so as to complete the conversion from the first content unit to the second content unit. Among them, the mBart model is a multi-language version of the transformer model.

[0109] (5) Based on the large amount of aligned unit data constructed above, a transformer model with an encoder-decoder architecture is used, with unit as input and the corresponding text as output. The model is trained using supervised learning to obtain a U2T model, which can complete the conversion from unit to the corresponding text.

[0110] (6) Perform training in the order of (1) / (2)->(3)->(4)->(5).

[0111] The following introduces the translation process of the speech translation module by taking speech as an example of the first content input by the user.

[0112] (1) S2U: Let R = [w t |t=1,…,T] is the original audio sequence of length T input by the user, where T is the number of frames obtained by dividing the original audio into 40ms frames. The audio sequence is converted into a spectrogram feature sequence X=[mel t |t=1,…,T], and the spectral feature sequence is sent to the S2U model for inference. After being encoded by the encoder (ENC), the latent variable H=[h t |t=1,…,T], and send the latent variable to the decoder (decoder, DEC) for decoding to obtain the speech semantic unit (source units) sequence U of the first content src =[U t |t=1,…,T].

[0113] (2)U2U:Set User Tone={tone|[self],[Peaceful],…,[Angry]} is the tone set by the user. The number of tones supported depends on the actual business situation. Timbre The timbre set for the user. The specific number of supported timbre is determined by the actual business situation. Convert the two into feature vectors (embedding) through one-hot encoding (OneHot) to obtain User Emb As user input. Emb and U src The two are sent to the U2U model for inference to obtain the second content unit (target units) sequence U tgt =[U t |t=1,…,T].

[0114] (3) U2S / U2T: The original spectrum feature sequence X = [mel t |t=1,…,T] and the obtained second content unit sequence U tgt =[U t |t=1,…,T] inputs the generator part of the trained U2S model for inference, and obtains the second content audio sequence Audio=[w t |t=1,…,T] as the translated audio; the obtained second content unit sequence U tgt =[U t |t=1,…,T] inputs the trained U2T model for inference to obtain the second content text sequence and complete the translation text output.

[0115] Figure 7 The voice translation module shown implements voice-to-text recognition, machine translation, and speech synthesis end-to-end, that is, when the user completes the settings, the corresponding text and voice output can be obtained by only interacting with the input voice once, so that the effect of generating a highly controllable personalized voice translation result can be achieved by combining the user's setting information and the first voice content input, and the translated second content can change with the change of tone. That is to say, in the translation method provided in the embodiment of the present application, the tone will affect the result of the translated content, not just as an audio feature in speech synthesis, or only affecting the text display effect.

[0116] Below Figure 7 An implementation method of the S2U module in the speech translation module is introduced.

[0117] Figure 8 A schematic diagram of an implementation of an S2U module provided in one embodiment of the present application is shown in FIG. Figure 8As shown, let R = [wn|n = 1, ..., N] be the original audio sequence of length N input by the user (N is the number of frames obtained by dividing the original audio into 40ms frames), Figure 8 In the example, the audio sequence is a 1×n speech sequence [0 0 42…5 8 0 0]. The audio sequence is converted into a spectrogram feature sequence X=[meln|n=1,…,N] according to its length. Figure 8 In the example, the spectrogram feature sequence is an n×128 spectrogram feature sequence. Then, the spectrogram feature sequence is sent to the encoder part of the S2U module for encoding to obtain a latent variable vector H=[hn|n=1,…,N], where hn is the latent variable of the nth frame, in the form of [1×256]. The latent variable is sent to the decoder part to obtain the speech semantic feature sequence F output =[fn|n=1,…,N], where f n is the feature matrix of the nth frame, in this case it is in the form of [1*768]. Afterwards, clustering is performed using the trained K-means clustering algorithm. Figure 8 In this case, k=1000, thus obtaining the first content unit sequence U src =[Un|n=1,…,N], where Un is the unit of the nth frame, Figure 8 Middle,U src is a 1×n sequence, U src =[132…99].

[0118] The S2U module is used to convert speech into discrete speech semantic units, that is, to convert the original speech input by the user into a first content unit sequence that retains the corresponding speech semantic information.

[0119] Below Figure 7 An implementation method of the U2U module in the speech translation module is introduced.

[0120] Fig. 9 This is a schematic diagram of an implementation of a U2U module provided in an embodiment of the present application. src =[U n |n=1,…,N] is the first content unit sequence of length N obtained by the above S2U module, Fig. 9 Middle,U src =[1 3 2…99]. Send the first content unit sequence to the U2U module for reasoning. Tone ={[peaceful]|[self],[peaceful],…,[angry]} is the tone set by the user. Timbre={[self]|[self],[Tony],…,[Xiaoyi]} is the timbre set by the user. Tone and User Timbre The two are embedded through OneHot method to obtain User Emb As user input. Emb is a 1×n vector, where n is the length of the OneHot encoding, such as Fig. 9 As shown, User Emb =[0 01…1 1 0 0]. Emb and U src They are sent to the U2U module for reasoning. src Enter the encoder part to perform inference and obtain the latent variable vector H u2u =[h n |n=1,…,N], where h n is the hidden variable corresponding to the nth Unit, in this case it is a 1×n vector. Then let the hidden variable vector H u2u and User Emb In the Decoder part, information fusion is performed through Cross-Attention to obtain the second content unit sequence U tgt =[U n |n=1,…,N], in this example, U tgt is a 1×n vector, U tgt =[123456…999 0]. In this example, the latent variable vector H u2u and User Emb When performing information fusion in the decoder part, not only the Cross-Attention method but also other information fusion methods may be used, such as direct concatenation. This embodiment does not limit the method used for information fusion.

[0121] The U2U module converts the first content unit into a second content unit that conforms to the target tone according to the user's setting information, that is, the voice semantic information of the first content expressed by the user is converted into a corresponding second content unit sequence through the U2U module and combined with the user's setting information.

[0122] Below Figure 7 An implementation method of the U2S module in the speech translation module is introduced.

[0123] Fig.10 A schematic diagram of the implementation of the U2S module provided in one embodiment of the present application, where the original spectrum feature sequence X=[mel n|n=1,…,N] and the second content unit sequence U obtained by the above U2U module tgt =[U n |n=1,…,N], Fig.10 Middle,U tgt = [123 456…999 0]. The original spectrum feature sequence X contains mel n is the feature matrix of the nth frame. In this example, the Mel spectrogram (a commonly used audio feature) is used. The matrix in this example is in the form of [1×m], where m is the number of dimensions for extracting features. In this example, the original spectrogram feature sequence is derived from the reference audio selected by the user, that is, the original spectrogram feature sequence X input here is also different when the timbre selected by the user is different.

[0124] Specifically, firstly, the original spectrogram feature sequence X is input into the speech encoder to obtain a feature matrix of [n×m], where n is the length of the sequence and m is the dimension of each element in the sequence. Then, the second content unit sequence U tgt Input the unit encoder to obtain the feature matrix of [n×k], where n is the length of the sequence and k is the dimension of each element in the sequence. Then merge the two. In this example, direct concatenation is used to obtain the feature matrix of [n×(m+k)], which is the latent variable H. audio =[h n |n=1,…,N],h n is the feature matrix with the dimension [n×(m+k)] mentioned above. audio The waveform is sent to the waveform decoder for inference, that is, the generator part of the U2S module, to obtain the second content waveform sequence Audio = [w n |n=1,…,N] as the final translated audio, Fig.10 In the figure, the second content waveform sequence Audio = [00 771166…993].

[0125] In this embodiment, the U2S module synthesizes the second content audio according to the required audio features of the second content unit. At the same time, the audio features of the reference input audio are maintained, that is, the speech semantic information of the second content expressed by the user is synthesized into the corresponding speech translation audio through the U2S module and combined with the reference audio set by the user.

[0126] As described above, the embodiment of the present application provides a translation method, which can combine the first content input by the user and the user's setting information to obtain the translated second content, and the translated second content changes with the change of the tone set by the user.

[0127] Fig.11A flowchart of a translation method provided in one embodiment of the present application, such as Fig.11 As shown, the above translation method may include:

[0128] Step 1101 : the electronic device 100 receives first content input by a user, where the first content is in a first language.

[0129] Step 1102: The electronic device 100 receives user setting information, where the user setting information includes a tone set by the user.

[0130] Step 1103 , the electronic device 100 translates the first content into second content, the second content uses a second language, the second language is different from the first language, and the second content is related to the user's setting information.

[0131] Specifically, the first language and the second language may be languages ​​of different types and / or languages ​​with different accents, etc. For example, the first language may be Chinese, and the second language may be English.

[0132] Specifically, the electronic device 100 translating the first content into the second content may include:

[0133] In step a, the electronic device 100 obtains a sequence of first contents according to the first contents; and obtains a feature vector corresponding to the setting information according to the setting information of the user.

[0134] In this embodiment, the user's setting information may include the tone set by the user. In this way, the electronic device 100 may perform feature conversion on the tone set by the user to obtain a feature vector corresponding to the tone set by the user.

[0135] In some examples, the user's setting information may also include: the timbre set by the user; thus, according to the user's setting information, obtaining the feature vector corresponding to the setting information may be: the electronic device 100 performs feature conversion on the tone and timbre set by the user to obtain the feature vector corresponding to the setting information. Specifically, the electronic device 100 may perform feature conversion on the tone and timbre set by the user through a OneHot method to obtain the feature vector corresponding to the setting information.

[0136] In step b, the electronic device 100 encodes the sequence of the first content to obtain a first latent variable vector; wherein the latent variable vector may also be referred to as a latent variable vector, which, in contrast to the observed variable vector, refers to an unobservable random variable vector. The latent variable vector can be inferred from the observed data through a mathematical model. One of the benefits of using a latent variable vector is that the latent variable vector can be used to reduce the dimension of the data. A large number of observed variables can be integrated into a latent variable vector to represent deep concepts, making the observed data easier to understand.

[0137] In one implementation, see Figure 7 and Fig. 9 When the first content is the first voice content, the sequence of the first content may be a sequence of voice semantic units of the first content. The electronic device 100 may encode the sequence of voice semantic units of the first content through the U2U module to obtain a first latent variable vector. The specific implementation process may refer to Fig. 9 The description of the illustrated embodiment will not be repeated here.

[0138] In step c, the electronic device 100 performs information fusion on the first latent variable vector and the feature vector corresponding to the setting information to obtain a sequence of the second content.

[0139] Continue to see Figure 7 and Fig. 9 When the second content is the second voice content, the sequence of the second content may be a sequence of voice semantic units of the second content. The electronic device 100 may fuse the first latent variable vector and the feature vector corresponding to the setting information through the decoder in the U2U module to obtain the sequence of voice semantic units of the second content. When fusing the first latent variable vector and the feature vector corresponding to the setting information, the Cross-Attention method may be used, or direct concatenation may be used. This embodiment does not limit the method used for information fusion.

[0140] In the above translation method, after the electronic device 100 receives the first content input by the user and the tone set by the user, it translates the first content into the second content, wherein the first content uses a first language and the second content uses a second language, the second language is different from the first language, and the second content is related to the user's setting information, so that the corresponding translation result can be obtained by combining the first content input by the user and the tone set by the user, and the translated second content can change with the change of the tone set by the user.

[0141] This application Fig.11 In the illustrated embodiment, the content in the first language may be voice or text. Similarly, the content in the second language after translation may be voice or text. The implementation processes are described in detail below.

[0142] In one implementation, the first content may be the first voice content; thus, the electronic device 100 may obtain the first content sequence according to the first content by: the electronic device 100 converts the first voice content into a first spectrogram feature sequence according to its length, encodes the first spectrogram feature sequence, and obtains a second latent variable vector; decodes the second latent variable vector, and clusters the sequence obtained by decoding to obtain a speech semantic unit sequence of the first content. For the specific implementation process, please refer to Figure 8 The description of the illustrated embodiment will not be repeated here. This implementation can realize the conversion of speech into discrete speech semantic units, that is, the original speech input by the user is converted into a unit sequence of the first content that retains the corresponding speech semantic information.

[0143] In another implementation, the first content may be a first text content; thus, the electronic device 100 may obtain a sequence of the first content according to the first content by: the electronic device 100 encodes the first text content to obtain a third latent variable vector; and decodes the third latent variable vector to obtain a sequence of the first content; wherein the sequence of the first content includes a text sequence of the first content or a sequence of speech and semantic units of the first content. This implementation can realize the conversion of the first text content input by the user into a text sequence of the first content or a sequence of speech and semantic units of the first content, thereby preparing for translating the first text content into a second text content or a second speech content.

[0144] In another implementation, the content of the second language may be a second text content, and the sequence of the second content may be a text sequence of the second content; thus, the electronic device 100 may obtain the translated second content according to the sequence of the second content by: the electronic device 100 decodes the text sequence of the second content to obtain the translated second text content.

[0145] In another implementation, the content of the second language may be a second voice content, and the sequence of the second content may be a sequence of voice semantic units of the second content; thus, before obtaining the translated second content based on the sequence of the second content, the electronic device 100 may also obtain a second spectrogram feature sequence corresponding to the timbre set by the user; thus, obtaining the translated second content based on the sequence of the second content may be as follows: the electronic device 100 performs voice encoding on the second spectrogram feature sequence to obtain a voice feature matrix; and performs voice semantic encoding on the sequence of voice semantic units of the second content to obtain a feature matrix of the second content; then, the electronic device 100 fuses the voice feature matrix with the feature matrix of the second content, decodes the fused feature matrix, and obtains the second voice content. For the specific implementation process, please refer to Fig.10The description of the illustrated embodiment is not repeated here. This implementation method can synthesize the second content audio according to the required audio features of the speech semantic unit sequence of the second content. At the same time, the audio features of the reference input audio are maintained, that is, the speech semantic information of the second content expressed by the user is combined with the reference audio set by the user to synthesize the corresponding speech translation audio.

[0146] In another implementation, the second content may be a second text content, and the sequence of the second content may be a sequence of speech and semantic units of the second content; thus, obtaining the translated second content according to the sequence of the second content may be: performing speech and semantic encoding on the sequence of speech and semantic units of the second content to obtain a feature matrix of the second content; and decoding the feature matrix of the second content to obtain the second text content. This implementation can obtain the second text content according to the sequence of speech and semantic units of the second content, thereby translating the speech content into text content.

[0147] Fig.12 A flowchart of a translation method provided in another embodiment of the present application is as follows: Fig.12 As shown, this application Fig.11 In the illustrated embodiment, after step 1103, the following steps may also be included:

[0148] Step 1201: The electronic device 100 displays the first content and the second content.

[0149] Specifically, in a call translation scenario, the interface of the electronic device 100 displaying the first content and the second content may be as shown in FIG. 5(b) and FIG. 5(c), referring to dialog box 52 in FIG. 5(b) and dialog box 53 in FIG. 5(c), both of which include Chinese content input by the user and translated English content.

[0150] Furthermore, after step 1201, the electronic device 100 may also obtain the first content modified by the user and / or the setting information modified by the user, and then the electronic device 100 may re-execute step 1101 and its subsequent steps. That is, after the electronic device 100 displays the first content and the second content, if the user wants to modify the first content and / or the user wants to change the tone, the user may modify the first content and / or the setting information, and the electronic device 100 obtains the first content modified by the user and / or the setting information modified by the user, and then re-executes step 1101 and its subsequent steps to re-acquire the translated second content.

[0151] In addition, after step 1201, the electronic device 100 may also obtain and display the second content modified by the user. That is, after the electronic device 100 displays the first content and the second content, if the user is not satisfied with the translated second content, or the user wants to modify the second content, the user may modify the second content, and the electronic device 100 obtains the second content modified by the user and displays it.

[0152] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0153] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of this application.

[0154] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0155] Fig.13 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, in which each functional module is divided according to each function, Fig.13 A possible schematic diagram of the composition of the electronic device 1300 involved in the above embodiment is shown. Fig.13 As shown, the electronic device 1300 may include: a receiving unit 131 and a translation unit 132;

[0156] The receiving unit 131 is configured to receive first content input by a user, the first content using a first voice; and receive user setting information, the user setting information including a tone set by the user;

[0157] The translation unit 132 is configured to translate the first content into second content, where the second content is in a second language that is different from the first language, and the second content is related to the user's setting information.

[0158] The translation unit 132 may include: an acquisition module 1302, an encoding module 1303, a fusion module 1304 and a translation module 1305;

[0159] The obtaining module 1302 is used to obtain a sequence of the first content according to the first content; and obtain a feature vector corresponding to the setting information according to the setting information of the user;

[0160] An encoding module 1303 is used to encode the sequence of the first content to obtain a first latent variable vector;

[0161] A fusion module 1304 is used to fuse the first latent variable vector and the feature vector corresponding to the above setting information to obtain a sequence of second content;

[0162] The translation module 1305 is used to obtain the translated second content according to the sequence of the second content.

[0163] It should be noted that this application Fig.11 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0164] The electronic device 1300 provided in this embodiment is used to execute the present application Fig.11 The translation method provided by the illustrated embodiment can therefore achieve the same effect as the above method.

[0165] Fig.14 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, Fig.13 Compared with the electronic device shown, the difference is that, in this embodiment, when the setting information of the above-mentioned user also includes: the timbre set by the user, the acquisition module 1302 is specifically used to perform feature conversion on the tone and timbre set by the above-mentioned user to obtain the feature vector corresponding to the setting information.

[0166] In an implementation of this embodiment, the first content includes first voice content; the obtaining module 1302 may include: a conversion submodule 13021, a first encoding submodule 13022 and a first decoding submodule 13023;

[0167] The conversion submodule 13021 is used to convert the first speech content into a first spectrogram feature sequence according to its length;

[0168] The first encoding submodule 13022 is used to encode the first spectrum feature sequence to obtain a second latent variable vector;

[0169] The first decoding submodule 13023 is used to decode the second latent variable vector and cluster the sequence obtained by decoding to obtain a sequence of speech and semantic units of the first content.

[0170] In another implementation, the first content includes first text content; the obtaining module 1302 may include: a first encoding submodule 13022 and a first decoding submodule 13023;

[0171] A first encoding submodule 13022 is used to encode the first text content to obtain a third latent variable vector;

[0172] The first decoding submodule 13023 is used to decode the third latent variable vector to obtain a sequence of the first content; wherein the sequence of the first content includes a text sequence of the first content or a sequence of speech and semantic units of the first content.

[0173] In another implementation, the second content includes a second text content, and the sequence of the second content includes a text sequence of the second content; the translation module 1305 may include: a second decoding submodule 13052;

[0174] The second decoding submodule 13052 is used to decode the text sequence of the second content to obtain the translated second text content.

[0175] In another implementation, the second content includes second voice content, and the sequence of the second content includes a sequence of voice semantic units of the second content; the translation unit 132 may further include: an acquisition module 1301;

[0176] The acquisition module 1301 is further used to acquire a second spectrogram feature sequence corresponding to the timbre set by the user before the translation module 1305 obtains the translated second content;

[0177] The translation module 1305 may include: a second encoding submodule 13051, a second decoding submodule 13052 and a matrix fusion submodule 13053;

[0178] The second encoding submodule 13051 is used to perform speech encoding on the second spectrogram feature sequence to obtain a speech feature matrix; and perform speech semantic encoding on the speech semantic unit sequence of the second content to obtain a feature matrix of the second content;

[0179] A matrix fusion submodule 13053, used for fusing the speech feature matrix and the feature matrix of the second content;

[0180] The second decoding submodule 13052 is used to decode the feature matrix obtained by fusion of the matrix fusion submodule 13053 to obtain the second voice content.

[0181] In another implementation, the second content includes a second text content, and the sequence of the second content includes a sequence of speech semantic units of the second content; the translation module 1305 may include: a second encoding submodule 13051 and a second decoding submodule 13052;

[0182] The second encoding submodule 13051 is used to perform speech and semantic encoding on the speech and semantic unit sequence of the second content to obtain a feature matrix of the second content;

[0183] The second decoding submodule 13052 is used to decode the feature matrix of the second content to obtain the second text content.

[0184] Furthermore, the electronic device 1300 may further include: a display module 1306;

[0185] The display module 1306 is configured to display the first content and the second content after the translation unit 132 translates the first content into the second content.

[0186] In this embodiment, the acquisition module 1301 is also used to obtain the first content modified by the user and / or the setting information modified by the user after the display module 1306 displays the above-mentioned first content and the above-mentioned second content; then the receiving unit 131 and the translation unit 132 re-execute the step of receiving the first content input by the user and its subsequent steps.

[0187] Furthermore, the acquisition module 1301 is further configured to acquire the second content modified by the user after the display module 1306 displays the first content and the second content; the display module 1306 is further configured to display the second content modified by the user.

[0188] It should be noted that this application Figure 11-12 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0189] The electronic device 1300 provided in this embodiment is used to execute the present application Figure 11-12 The translation method provided by the illustrated embodiment can therefore achieve the same effect as the above method.

[0190] It should be understood that the electronic device 1300 may correspond to Figure 4 The electronic device 100 shown in FIG. The function of the receiving unit 131 can be Figure 4The processor 110, the audio module 170 and the microphone 170C in the electronic device 100 shown in FIG. 1 are implemented, and the function of the translation unit 132 can be implemented by Figure 4 The processor 110 in the electronic device 100 is shown implemented.

[0191] In the case of adopting an integrated unit, the electronic device 1300 may include a processing module, a storage module, and a communication module.

[0192] The processing module can be used to control and manage the actions of the electronic device 1300, for example, it can be used to support the electronic device 1300 to execute the steps executed by the above-mentioned units and / or modules. The storage module can be used to support the electronic device 1300 to store program codes and data, etc. The communication module can be used to support the communication between the electronic device 1300 and other devices.

[0193] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0194] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 1300 involved in this embodiment can be a Figure 4 Device of the structure shown.

[0195] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 11-12 The method provided by the illustrated embodiment.

[0196] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 11-12 The method provided by the illustrated embodiment.

[0197] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0198] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0199] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0200] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0201] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A translation method, characterized in that: include: receiving first content input by a user, where the first content is in a first language; receiving setting information of the user, wherein the setting information of the user includes the tone set by the user; The first content is translated into second content, the second content is in a second language, the second language is different from the first language, and the second content is related to the setting information of the user.

2. The method according to claim 1, characterized in that The user's setting information also includes the timbre set by the user.

3. The method according to claim 1 or 2, characterized in that: The translating the first content into second content comprises: According to the first content, obtaining a sequence of the first content; and according to the setting information of the user, obtaining a feature vector corresponding to the setting information; Encoding the sequence of the first content to obtain a first latent variable vector; Performing information fusion on the first latent variable vector and the feature vector corresponding to the setting information to obtain a sequence of second content; According to the sequence of the second content, the translated second content is obtained.

4. The method according to claim 3, characterized in that: When the setting information of the user also includes the timbre set by the user, obtaining the feature vector corresponding to the setting information according to the setting information of the user includes: The tone and timbre set by the user are subjected to feature conversion to obtain a feature vector corresponding to the setting information.

5. The method according to any one of claims 1 to 4, characterized in that: The first content includes first voice content; and the sequence of obtaining the first content according to the first content includes: Converting the first speech content into a first spectrogram feature sequence according to its length; Encoding the first spectrum feature sequence to obtain a second latent variable vector; The second latent variable vector is decoded, and the sequence obtained by decoding is clustered to obtain a sequence of speech and semantic units of the first content.

6. The method according to any one of claims 1 to 4, characterized in that: The first content includes first text content; and the sequence of obtaining the first content according to the first content includes: Encoding the first text content to obtain a third latent variable vector; The third latent variable vector is decoded to obtain a sequence of the first content; wherein the sequence of the first content includes a text sequence of the first content or a sequence of speech semantic units of the first content.

7. The method according to claim 1, characterized in that The second content includes second text content, and the sequence of the second content includes: a text sequence of the second content; The obtaining the translated second content according to the sequence of the second content comprises: The text sequence of the second content is decoded to obtain translated second text content.

8. The method according to claim 4, characterized in that The second content includes second voice content, and the sequence of the second content includes a sequence of voice semantic units of the second content; Before obtaining the translated second content according to the sequence of the second content, the method further includes: Acquire a second spectrogram feature sequence corresponding to the timbre set by the user; The obtaining the translated second content according to the sequence of the second content comprises: Performing speech coding on the second spectrogram feature sequence to obtain a speech feature matrix; and performing speech semantic coding on the speech semantic unit sequence of the second content to obtain a feature matrix of the second content; Merging the speech feature matrix and the feature matrix of the second content; The fused feature matrix is ​​decoded to obtain the second speech content.

9. The method according to claim 4, characterized in that The second content includes second text content, and the sequence of the second content includes a sequence of speech semantic units of the second content; The obtaining the translated second content according to the sequence of the second content comprises: Performing speech and semantic coding on the speech and semantic unit sequence of the second content to obtain a feature matrix of the second content; The feature matrix of the second content is decoded to obtain second text content.

10. The method according to any one of claims 1 to 4, characterized in that: After translating the first content into the second content, the method further includes: The first content and the second content are displayed.

11. The method according to claim 10, characterized in that After displaying the first content and the second content, the method further includes: Acquiring the first content modified by the user and / or the setting information modified by the user; The step of receiving the first content input by the user and subsequent steps are re-executed.

12. The method according to claim 10, characterized in that After displaying the first content and the second content, the method further includes: The second content modified by the user is acquired and displayed.

13. A translation device, characterized in that: include: A receiving unit, configured to receive first content input by a user, wherein the first content uses a first voice; and receiving setting information of the user, wherein the setting information of the user includes the tone set by the user; The translation unit is configured to translate the first content into second content, where the second content is in a second language that is different from the first language, and the second content is related to the setting information of the user.

14. An electronic device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 12.

Images