Real-time translation method and device and storage medium
By leveraging multimodal contextual understanding of audio and video information, combined with large models and edge computing, the problems of overlapping speech interference and emotion neglect in real-time translation are solved, achieving high accuracy and human-like translation, applicable to various scenarios.
Patent Information
- Application Number
- CN202511029250.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-25
- Publication Date
- 2025-12-23
AI Technical Summary
Existing real-time translation technologies are susceptible to overlapping speech interference, ignore emotional information, produce mechanical translation outputs, and have a high error rate in noisy environments.
By acquiring audio and video information, performing timbre labeling and facial feature recognition, establishing matching relationships, and achieving multimodal contextual understanding, the system combines large-scale ASR and LLM models for translation, supports the recognition of minority languages and proper nouns, optimizes offline translation through edge computing, and collects gestures and micro-expressions for humanized translation.
It improves speaker separation accuracy, provides more human-like translation results, adapts to noisy environments, supports less common languages and proper nouns, and is suitable for virtual meetings and anonymous social scenarios.
Smart Images
Figure CN121189342A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of multi-person speech processing technology, and in particular to a real-time translation method, device, and storage medium. Background Technology
[0002] In traditional translation solutions, real-time translation generally relies solely on voiceprints, which are susceptible to overlapping speech interference; its translation output is mechanical and ignores emotional information; moreover, the noisy environment of exhibitions can cause speech overlap, leading to a high translation error rate. Summary of the Invention
[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a real-time translation method, device and storage medium with high speaker separation accuracy and more human-like translation.
[0004] In a first aspect, the present invention provides a real-time translation method, comprising: Obtain the first audio and video information of the first communicator in the current conversation; The first audio information is timbre-marked to obtain at least one first timbre-marked segment; wherein the first timbre-marked segment includes first time segment information of the first timbre segment; Facial feature recognition is performed on the first video information to obtain at least one first facial feature set; wherein, the first facial feature set includes second time period information on changes in mouth features; The second timbre marker segment is matched with the second facial feature set to obtain the first matching relationship between the second timbre marker segment and the second facial feature set corresponding to the second communicator; The second matching relationship is used to translate the timbre marker segments corresponding to the third communicator in real time.
[0005] In a second aspect, the present invention also provides an apparatus comprising one or more processors and a memory, wherein the memory contains instructions executable by the one or more processors to cause the one or more processors to perform a real-time translation method provided according to various embodiments of the present invention.
[0006] Thirdly, the present invention also provides a storage medium storing a computer program that enables a computer to execute a real-time translation method provided according to various embodiments of the present invention. Attached Figure Description
[0007] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of a real-time translation method provided in an embodiment of the present invention.
[0008] Figure 2This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation
[0009] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.
[0010] It should be noted that the real-time translation method provided in this application can be applied to any device that requires real-time translation, such as portable devices (e.g., glasses / earphones), professional translation devices (industry-customized translation machines), conference equipment (international conferences, virtual conferences), and in-vehicle systems.
[0011] Furthermore, the real-time translation method provided in this application does not limit the entity that performs it. Optionally, the entity that performs the real-time translation method provided in this application can be a real-time translation system. Specifically, the entity that performs the real-time translation method provided in this application can be a translation unit within a real-time translation system.
[0012] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0013] Figure 1 This is a flowchart illustrating a real-time translation method according to an embodiment of the present invention. Figure 1 As shown, in this embodiment, the present invention provides a real-time translation method, including: S111: Obtain the first audio and video information of the first communicator in the current communication; S112: The first audio information is timbre-marked to obtain at least one first timbre-marked segment; wherein, the first timbre-marked segment includes the first time segment information of the first timbre segment; S113: Perform facial feature recognition on the first video information to obtain at least one first facial feature set; wherein, the first facial feature set includes second time period information on changes in mouth features; S13: Match the second timbre marker segment with the second facial feature set to obtain the first matching relationship between the second timbre marker segment and the second facial feature set corresponding to the second communicator; S15: Real-time translation of each timbre marker segment corresponding to the third communicator through the second matching relationship.
[0014] Specifically, let's take the example of communicators A, B, and C communicating in real time through AI glasses at an exhibition. In step S111, communicator A obtains the audio information of communicator B through AI glasses and the video information of communicator B through a camera. The camera can be configured according to actual needs. For example, it can be configured on AI glasses or on other wearable devices (such as helmets) to achieve the same technical effect.
[0015] The acquired audio information is processed to remove noise and other interference data. Whether the current step is executed can be configured according to actual needs. For example, in a noisy environment such as an exhibition, the current step needs to be executed to improve the accuracy of real-time translation. In step S112, the audio information is marked with timbre to obtain at least one timbre-marked segment; the time period information of the timbre-marked segment can be configured according to actual needs, for example, configured as a start time point and an end time point (t_start, t_end), or configured as a start time point and an incremental duration [t_start, +ns], or configured in other ways to achieve the same technical effect; In step S113, facial feature recognition is performed on the video information to obtain at least one first facial feature set; similarly, the time period information of mouth feature changes can be configured according to actual needs to achieve the same technical effect. In step S13, it is assumed that there are several timbre marker segments and facial feature sets at this time. The time period information of the timbre marker segments and the time period information of mouth feature changes are combined to match them to obtain several matching relationships, which are assumed to be: Communicator A {timbre marker segment 1, timbre marker segment 2}, Communicator B {timbre marker segment 3, timbre marker segment 4}, Communicator C {timbre marker segment 5, timbre marker segment 6}; In step S15, the {timbre marker segment 3, timbre marker segment 4} of communicator B and the {timbre marker segment 5, timbre marker segment 6} of communicator C are translated in chronological order. The display on the AI glasses can be configured according to actual needs. Taking communicator A as an example, it can be displayed as "the above {timbre marker segment 3~timbre marker segment 6} translated in chronological order", or it can be displayed as "the above {timbre marker segment 3~timbre marker segment 6} translated in chronological order, and the {timbre marker segment 1, timbre marker segment 2} of communicator A"", which can achieve the same technical effect. The above embodiments introduce facial features, which upgrades real-time translation for multiple people from "simple speech to text" to "multimodal contextual understanding", resulting in high speaker separation accuracy and more humanized translation.
[0016] In some special scenarios, such as exhibitions, AI glasses need to be able to translate proper nouns and terminology, as well as support less common languages (Vietnamese / Indonesian / Hindu, etc.).
[0017] In a preferred embodiment, the above problems can be solved by the following method: Before obtaining the first audio and video information of the first communicator in the current communication, the process also includes: Large-scale ASR model: Audio is converted into an encoder + adapter, and then into a token. The recognition rate is improved by combining user context and optimizing the recognition of polyphonic characters / similar sounds / personal and place names. Minor languages: Whisper Encoder processes audio and outputs feature sequences. H=[h1,h2,...,hT] (T is the time step); LLM generates text Y=[y1,y2,...,yN] step by step, using feature H as a condition.
[0018] By training with a high-precision corpus through the above steps, it can translate proper nouns and terminology, and supports less commonly spoken languages, making it suitable for exhibitions, especially foreign trade exhibitions.
[0019] The above embodiments can support offline real-time translation in exhibition scenarios, especially offline real-time translation for foreign trade exhibitions.
[0020] Furthermore, in an offline environment, the above method also includes: Edge computing temporarily stores and transcribes audio recordings; When a network connection is detected, the above-mentioned audio transcription will be automatically synchronized.
[0021] The above embodiments can support the generation of meeting minutes in an offline environment.
[0022] In certain scenarios, such as online meetings, participants may not want to disclose too much sensitive information.
[0023] In a preferred embodiment, the above problems can be solved by the following method: The first video information is generated using the following method: The first source video was acquired by the first camera; The first source video transforms the character image of the first communicator into the first deformable virtual image, and the first video information is generated based on the first deformable virtual image.
[0024] The above embodiments convert user voice into a deformable virtual avatar to avoid leaking sensitive information.
[0025] In a preferred embodiment, the camera is also used to capture the gestures of the communicator, and to perform real-time translation of the timbre marker segments corresponding to the third communicator through a second matching relationship, including: The third communicator's corresponding timbre marker segments are translated in real time using the second matching relationship and the third communicator's gestures.
[0026] The gesture library contains different gestures representing different moods, including the meanings of different gestures in different cultures.
[0027] Table 1 Gesture / Mood Comparison Chart Gesture type Common Moods waving vigorously Excitement, eagerness Spread your hands Confusion, helplessness crossed arms defensive, indifferent Thumbs up Agree / Satisfaction (Western) / Offend (Middle East) The aforementioned gesture library enables better cultural adaptation and more user-friendly real-time translation.
[0028] Furthermore, the camera is also used to capture the micro-expressions of the communicator, combining micro-expressions with gestures to obtain more human-like real-time translation.
[0029] In a preferred embodiment, the method further includes: If the system detects that the user is about to speak, it will increase the NPU frequency in advance. If speech recognition is detected as complete, the DVFS frequency will be reduced to wait for the translation task. If there is no new input within 3 seconds, the NPU is turned off, and only microphone listening is enabled.
[0030] The above embodiments dynamically adjust hardware resources by predicting the computing power requirements of AI glasses in the next few seconds, avoiding continuous high power consumption while maintaining low latency.
[0031] The above-mentioned method of collecting video information via camera has the following problems: Environmental issues: Recognition rate drops significantly in strong light, backlight, or side-facing scenarios; Facial occlusion issue: Masks, glasses, helmets, etc. can cause facial feature extraction to fail, requiring increased computing power to compensate for the lack of facial features; Battery life issue: Keeping the camera on continuously will reduce the battery life of the AI glasses; Camera cost: Cameras are expensive, and the higher the image resolution required, the higher the cost.
[0032] In a preferred embodiment, the above problems can be solved by the following method: The first video information is generated using the following method: Extracting first speech features from the first audio information to obtain a first speech embedding vector; wherein, the first speech features include pitch, intonation, and speech rate; A first deformable face model is generated based on a first speech embedding vector, and first video information is generated based on the first deformable face model.
[0033] In the above embodiments, there is no need to configure a camera, and there is no need to consider issues such as environment, facial occlusion, battery life, and camera cost. Video information of the communicator can be generated simply based on voice characteristics.
[0034] The above embodiments are particularly suitable for scenarios such as virtual meetings and anonymous social networking (metaverse).
[0035] The above embodiments can generate video information, but its accuracy is not as high as that of video information captured by a camera. However, in scenarios such as virtual meetings and anonymous social networking, the gender, age, and even race of communicators can be inferred from their voice, and some communicators do not wish to disclose such information.
[0036] In a preferred embodiment, the above problems can be solved by the following method: Extracting the first speech features from the first audio information to obtain the first speech embedding vector includes: The first audio information is desensitized locally to generate the first desensitized audio information; Extract the first speech feature from the first desensitized audio information to obtain the first speech embedding vector.
[0037] The above embodiments ensure that audio information is processed locally, and only anonymized audio information is uploaded to generate speech embedding vectors, preventing the leakage of communicator information.
[0038] Suppose we have the following scenario: Original audio: A 30-year-old man speaking in a quiet environment generates a face that appears to be around 30 years old.
[0039] By overlaying the same voice with classical music background, a face of someone around 50 years old is generated.
[0040] Training data bias leads to the association of "classical music" with the "middle-aged and elderly" group, resulting in the generation of incorrect communicator video information.
[0041] In a preferred embodiment, the problems encountered in the above scenario can be solved by the following method: Before obtaining the first audio and video information of the first communicator in the current communication, the process also includes: Acquire second audio information, extract second speech features from the second audio information to obtain a second speech embedding vector, and generate a second deformable face model based on the second speech embedding vector. A third speech embedding vector is generated by retaining non-physiologically relevant factors but changing physiologically relevant factors in the second audio information, and a third deformable face model is generated based on the third speech embedding vector. Determine whether there are significant changes in facial features between the second and third deformable face models: If yes, then the samples of the communicator to whom the second audio information belongs are mixed in the training set.
[0042] In the above embodiments, when training the model, if certain features in the training data (such as "middle-aged and elderly people like classical music") are over-captured by the model, samples in the training set (such as the same person's voice with different background sounds) are mixed to improve robustness.
[0043] Figure 2 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.
[0044] like Figure 2 As shown, in another aspect, this application also provides a device 200, including one or more central processing units (CPUs) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage portion 208 into a random access memory (RAM) 203. The RAM 203 also stores various programs and data required for the operation of the device 200. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.
[0045] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.
[0046] In particular, according to embodiments of this disclosure, the methods described in any of the above embodiments can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211.
[0047] In another aspect, this application also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the apparatus of the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores one or more programs that are used by one or more processors to perform the methods described in this application.
[0048] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.
[0049] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor; for example, each unit can be a software program located in a computer or mobile smart device, or a separately configured hardware device. The names of these units or modules do not, in some cases, constitute a limitation on the unit or module itself.
[0050] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.
Claims
1. A real-time translation method, characterized in that, include: Obtain the first audio and video information of the first communicator in the current conversation; The first audio information is timbre-tagged to obtain at least one first timbre-tagged segment; wherein the first timbre-tagged segment includes first time segment information of the first timbre segment; Facial feature recognition is performed on the first video information to obtain at least one first facial feature set; wherein, the first facial feature set includes second time period information on changes in mouth features; The second timbre marker segment is matched with the second facial feature set to obtain the first matching relationship between the second timbre marker segment and the second facial feature set corresponding to the second communicator; The second matching relationship is used to translate the timbre marker segments corresponding to the third communicator in real time.
2. The method according to claim 1, characterized in that, The first video information was acquired by the first camera.
3. The method according to claim 2, characterized in that, The first video information is generated using the following method: The first source video is obtained by capturing data through the first camera; The first communicator's character image is converted into a first deformable virtual image from the first source video, and first video information is generated based on the first deformable virtual image.
4. The method according to claim 2, characterized in that, The camera is also used to capture the communicator's gestures, and the real-time translation of each timbre marker segment corresponding to the third communicator through the second matching relationship includes: The third communicator's corresponding timbre marker segments are translated in real time using the second matching relationship and the third communicator's gestures.
5. The method according to claim 1, characterized in that, The first video information is generated using the following method: Extract the first speech features from the first audio information to obtain the first speech embedding vector; wherein, the first speech features include pitch, intonation, and speech rate; A first deformable face model is generated based on the first speech embedding vector, and first video information is generated based on the first deformable face model.
6. The method according to claim 5, characterized in that, The step of extracting the first speech feature from the first audio information to obtain the first speech embedding vector includes: The first audio information is desensitized locally to generate the first desensitized audio information; The first speech feature of the first desensitized audio information is extracted to obtain the first speech embedding vector.
7. The method according to claim 6, characterized in that, Before obtaining the first audio information and first video information of the first communicator in the current communication, the method further includes: The trained encoder decomposes speech features into physiologically relevant factors and non-physiologically relevant factors; wherein, the physiologically relevant factors include fundamental frequency and formants, and the non-physiologically relevant factors include accent and speech rate; The step of extracting the first speech feature from the first desensitized audio information to obtain the first speech embedding vector includes: The first speech feature of the physiologically related factors in the first desensitized audio information is extracted to obtain the first speech embedding vector.
8. The method according to claim 7, characterized in that, Before obtaining the first audio information and first video information of the first communicator in the current communication, the method further includes: Acquire second audio information, extract second speech features from the second audio information to obtain a second speech embedding vector, and generate a second deformable face model based on the second speech embedding vector. A third speech embedding vector is generated by retaining non-physiologically relevant factors but changing physiologically relevant factors from the second audio information, and a third deformable face model is generated based on the third speech embedding vector. Determine whether there are significant changes in the facial features of the second deformable face model compared to the third deformable face model: If yes, then the samples of the communicator to whom the second audio information belongs are mixed in the training set.
9. A real-time translation device, characterized in that, The device includes: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in any one of claims 1-8.
10. A storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.