Intelligent glasses multi-language simultaneous interpretation method and system based on cloud collaborative translation

Abstract

The application provides a kind of intelligent glasses multilingual simultaneous interpretation method and system based on cloud collaborative translation, it is related to intelligent interactive device technical field.Intelligent glasses multilingual simultaneous interpretation method based on cloud collaborative translation, it is applied to the intelligent glasses including microphone module, communication module and output module, comprising: real-time acquisition environment audio data by microphone module;Through communication module, the environment audio data is transmitted to cloud translation server in real time;Receive the translation result that cloud translation server returns, wherein, translation result is generated after cloud translation server carries out speaker separation, language recognition and translation processing to environment audio data, and translation result corresponds to the translation content of target voice segment in environment audio data not belonging to the speech of user oneself;Present the translation result by output module.The application can solve the technical problem that existing intelligent glasses is difficult to efficiently perform complex translation processing under the constraint of light weight.

Description

Technical Field

[0001] This invention relates to the field of intelligent interactive device technology, specifically to a multilingual simultaneous interpretation method and system for smart glasses based on cloud-based collaborative translation. Background Technology

[0002] With increasingly frequent global exchanges, the need for cross-language face-to-face communication is becoming more and more common in scenarios such as business negotiations, outbound tourism, and international conferences. Smart glasses, due to their first-person perspective and hands-free wearing characteristics, are considered an ideal medium for multilingual simultaneous interpretation. However, existing local translation solutions based on smart glasses are limited by the computing power and power consumption constraints of the glasses, and can only run lightweight translation models, resulting in limited translation quality and the ability to handle complex sentences.

[0003] The above description of the problem discovery process is only used to assist in understanding the technical solution of the present invention and does not mean that the above content is prior art. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for multilingual simultaneous interpretation of smart glasses based on cloud-based collaborative translation, so as to solve the technical problem of the contradiction between the need for lightweight physical form of smart glasses and the need for high computing power for complex translation tasks in the prior art.

[0005] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation, applied to smart glasses. The smart glasses include a microphone module, a communication module, and an output module, comprising: real-time acquisition of environmental audio data through the microphone module; real-time transmission of the environmental audio data to a cloud translation server through the communication module; receiving translation results returned by the cloud translation server, wherein the translation results are generated by the cloud translation server after performing speaker separation, language recognition, and translation processing on the environmental audio data, and the translation results correspond to the translation content of target speech segments in the environmental audio data that do not belong to the user's own speech; and presenting the translation results through the output module.

[0006] Optionally, the microphone module includes at least one bone conduction microphone and at least one air conduction microphone; the real-time acquisition of environmental audio data through the microphone module includes: acquiring bone vibration signals when the wearer speaks through the bone conduction microphone, acquiring air conduction speech signals in the environment through the air conduction microphone, and fusing the two to suppress the voice components belonging to the wearer in the environmental audio data.

[0007] Optionally, the step of transmitting the environmental audio data to the cloud translation server in real time via the communication module includes: using a streaming segmented transmission method, after detecting a valid voice signal, continuously pushing incremental audio data to the cloud translation server in segments of preset duration, without waiting for the complete sentence to end.

[0008] Optionally, the step of transmitting the environmental audio data to the cloud translation server in real time via the communication module further includes: compressing and encoding the environmental audio data, and adaptively adjusting the bitrate parameter of the compression encoding according to the current network bandwidth and round-trip latency, so as to maintain transmission continuity when network quality fluctuates.

[0009] Optionally, receiving the translation results returned by the cloud translation server includes: receiving the translation results word by word or phrase by phrase in a streaming manner, and during the receiving process, the output module begins to present the received partial translation content without waiting for the complete translation result of the whole sentence to be returned.

[0010] Optionally, when performing speaker separation, the cloud translation server uses a pre-registered user voiceprint model to identify and filter audio segments belonging to the user's own voice, and only performs the translation processing on the remaining non-user voice segments.

[0011] Optionally, the output module includes a bone conduction speaker; presenting the translation result through the output module includes: playing the speech-synthesized audio of the translation result through the bone conduction speaker in a bone conduction transmission mode that is only perceptible to the wearer.

[0012] Optionally, the output module includes a display module; presenting the translation result through the output module includes: displaying the text content of the translation result as text subtitles within the wearer's field of vision through the display module.

[0013] Optionally, the cloud translation server is also used to identify the speaker's emotional state or tone characteristics in the target speech segment; the translation result is supplemented with marking information corresponding to the emotional state or tone characteristics; when the output module presents the translation result, it synchronously presents the marking information with visual identifiers or specific tones.

[0014] Optionally, it also includes: after the cloud translation server completes the translation, pushing the text content of the translation result to the mobile terminal associated with the smart glasses in the form of a notification or message card, so that the user can view the translation record afterward.

[0015] Optionally, it also includes: caching the translation results returned by the cloud translation server to the local storage of the smart glasses; when a communication link with the cloud translation server is detected to be interrupted and the interruption duration exceeds a preset threshold, automatically switching the smart glasses to a preset local emergency prompt mode, and sending a prompt message to the wearer through the output module to indicate that the current translation service is unavailable.

[0016] Secondly, the present invention provides a multilingual simultaneous interpretation system for smart glasses based on cloud-based collaborative translation, comprising: an audio acquisition module for real-time acquisition of environmental audio data, the environmental audio data including the wearer's own voice and the voice of at least one other party in the dialogue; a communication module for real-time transmission of the environmental audio data to a cloud translation server and receiving translation results returned by the cloud translation server; wherein the translation results are generated by the cloud translation server after performing speaker separation, language recognition, and translation processing on the environmental audio data, and the translation results correspond to the translation content of target speech segments in the environmental audio data that do not belong to the wearer's own voice; and an output module for presenting the translation results.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention provides a multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation. Applied to smart glasses including a microphone module, a communication module, and an output module, the method includes: real-time acquisition of environmental audio data via the microphone module; real-time transmission of the environmental audio data to a cloud translation server via the communication module; receiving translation results from the cloud translation server, wherein the translation results are generated by the cloud translation server after speaker separation, language recognition, and translation processing of the environmental audio data, and the translation results correspond to the translation of target speech segments in the environmental audio data that do not belong to the user's own speech; and presenting the translation results through the output module. This invention offloads all translation computation tasks to the cloud server, with the glasses only responsible for audio acquisition, streaming transmission, and result output. This allows the glasses hardware to fully utilize the high-quality translation capabilities of large-scale cloud translation models without requiring a high-performance processor, generating high power consumption, or causing high heat generation, achieving an optimal balance between lightweight physical form and translation quality. Attached Figure Description

[0018] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 is a schematic diagram of the structure of an electronic device for implementing a cloud-based collaborative translation method for multilingual simultaneous interpretation in smart glasses, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation, provided as an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings of the embodiments, and the technical solutions of the embodiments will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The need for cross-language face-to-face communication is becoming increasingly common in scenarios such as business negotiations, international conferences, and outbound tourism. However, a core contradiction has long existed in existing technologies: on the one hand, to achieve comfortable wear all day, smart glasses need to be as lightweight as possible, resulting in extremely limited local computing power, storage, and heat dissipation capabilities; on the other hand, high-quality multilingual translation, especially tasks involving complex long sentences, professional terminology, and contextual understanding, highly relies on powerful computing resources and large-scale language models. Purely local translation solutions are limited by the computing power of the glasses themselves, and can only deploy lightweight translation models, resulting in insufficient translation quality and adaptability to complex scenarios. While pure cloud-based translation solutions can utilize high-performance cloud models, traditional implementations have several significant drawbacks: First, audio transmission typically employs an "offline" mode, where entire sentences are captured and then compressed before being uploaded, resulting in significant end-to-end latency. The translated audio is delivered only after the conversation ends, severely disrupting the natural rhythm of face-to-face communication. Second, the microphone on the glasses simultaneously picks up the voices of both the wearer and the other party. If the cloud lacks effective speaker differentiation mechanisms, it will translate and transmit the wearer's own speech along with the audio, leading to information redundancy and auditory confusion, as if "one's own words are translated and read back to oneself." Third, the translation output method is singular, usually relying solely on air-conduction speakers. This results in unclear audio in noisy environments and disturbs others in quiet settings, lacking the ability to adapt the output modality based on the usage scenario.

[0021] Therefore, there is an urgent need for a smart glasses multilingual simultaneous interpretation solution that can organically combine the lightweight constraints of glasses with the high computing power of the cloud, while also possessing low latency transmission, accurate speaker separation, and adaptive multimodal output.

[0022] To facilitate understanding of how the embodiments of the present invention implement a multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation, please refer to Figures 1(a) and 1(b). Figure 1 is a schematic diagram of the structure of a smart glasses provided in an embodiment of the present invention. The glasses include a speaker 1, a touch button 2, a first physical button 3, an infrared sensor 4, a first microphone 5, a camera 6, a biosensor 7, a second physical button 8, an eye-tracking sensor 9, and a second microphone 10. When worn, the speaker 1 is located above and behind the user's ears, the touch button 2 is located on the outside of the temples, the first physical button 3 is located in the front middle section of the right temple, the infrared sensor 4 is located on the outside of the frame to acquire environmental information, the first microphone 5 is located on the temples, the camera 6 is located on the outside of the frame to acquire image information, the biosensor 7 is located in the middle and rear section of the temples to acquire the user's biometric information, the second physical button 8 is located in the front middle section of the left temple, the eye-tracking sensor 9 is located on the inside of the frame, and the second microphone 10 is located near the nose pad. An audio acquisition module is a combination of components on the glasses responsible for "hearing sound." In this embodiment, two microphones are included: an air conduction microphone (first microphone 5) located on the temple of the glasses, used to collect the speech of the other party in the environment; and a bone conduction microphone (second microphone 10) located near the nose pad, used to collect the vibration signal of the wearer's skull when speaking. It is important to note that the function of the bone conduction microphone and the air conduction microphone is to "provide a basis for separation," not to "directly complete the separation." The air conduction microphone collects a mixed audio—containing both the voice of the other party and the wearer's own voice. The bone conduction microphone only collects the wearer's bone vibration signal, which contains "only the wearer's own voice, with almost no other party's voice." After receiving these two signals, the cloud uses the bone conduction signal as a reference to identify and filter out components "consistent with the bone conduction signal" in the mixed audio. What remains is the clean speech belonging only to the other party. Both audio streams are uploaded to the cloud, where the cloud uses the bone conduction signal to identify "this is the user speaking," thus filtering it out and only translating what the other party said. The communication module, integrated inside the temples, is responsible for transmitting data to the cloud. It typically uses a Wi-Fi or cellular network chip. It continuously transmits the sound captured by the microphone to the cloud translation server in real-time and receives the translated results from the cloud. The transmission method is "speak-as-you-go," streaming the audio in short segments, with the cloud also streaming back to reduce latency. The output module, on the other hand, is responsible for displaying the translation results to the user. This solution mainly includes two components: a bone conduction speaker (speaker 1), placed above and behind the ear, transmitting the translated speech directly to the inner ear via skull vibrations, audible only to the user and not disturbing others; and a display module on the lenses, displaying the translated content as text subtitles. The system can choose between voice broadcasting and text display based on ambient noise levels or privacy requirements.In the cloud-based collaborative translation-based multilingual simultaneous interpretation method for smart glasses described in this invention, the above-mentioned components work together to achieve efficient coordination between cloud translation services and the acquisition and output of the glasses.

[0023] To achieve the above objectives, this invention provides a multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation. Please refer to [link to relevant documentation]. Figure 2 This invention provides a multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation, applied to smart glasses. The smart glasses include a microphone module, a communication module, and an output module. The method includes the following steps: S1, Real-time acquisition of ambient audio data via the microphone module; The microphone module collects audio data from the wearer's environment in real time. This environmental audio data typically includes both the wearer's own voice and the voice of at least one other person in the conversation.

[0024] S2, the environmental audio data is transmitted to the cloud translation server in real time through the communication module; The communication module transmits the collected environmental audio data to the cloud translation server in real time. Unlike traditional methods that collect entire sentences before uploading, this invention uses a streaming, segmented transmission method. Incremental audio data is continuously pushed to the cloud as soon as a valid voice signal is detected, without waiting for the complete sentence to finish, thus significantly reducing transmission latency. Simultaneously, the system adaptively adjusts the audio compression encoding bitrate parameters based on the current network bandwidth and round-trip latency to maintain transmission continuity even when network quality fluctuates.

[0025] S3, receive the translation result returned by the cloud translation server, wherein the translation result is generated by the cloud translation server after performing speaker separation, language recognition and translation processing on the environmental audio data, and the translation result corresponds to the translation content of the target speech segment in the environmental audio data that does not belong to the user's own voice; After receiving environmental audio data, the cloud-based translation server performs speaker separation, speech recognition, and translation processing. Speaker separation utilizes a pre-registered user voiceprint model to identify and filter audio segments belonging to the wearer's own voice, translating only the remaining non-user speech segments. This ensures the translation corresponds only to the speaker's speech content, preventing mistranslation and transmission of the user's own voice. The translation results are streamed back to the glasses, word by word or phrase by phrase, allowing the glasses to begin displaying partial translations during reception, without waiting for the complete translation, further reducing perceived latency for the user.

[0026] S4, The translation result is presented through the output module.

[0027] The output module may include a bone conduction speaker and / or a display module. The bone conduction speaker is located on the inside of the temple and plays the synthesized audio of the translation result using bone conduction, a method only perceptible to the wearer, ensuring privacy and not disturbing others. The display module displays the text content of the translation result as subtitles within the wearer's field of vision. The system can select the most suitable output mode based on contextual information about the wearer's environment, such as noise level and privacy requirements.

[0028] As an optional implementation, the microphone module includes at least one bone conduction microphone and at least one air conduction microphone; the real-time acquisition of environmental audio data through the microphone module includes: acquiring bone vibration signals when the wearer speaks through the bone conduction microphone, acquiring air conduction speech signals in the environment through the air conduction microphone, and fusing the two to suppress the voice components belonging to the wearer in the environmental audio data.

[0029] As an optional implementation, the step of transmitting the environmental audio data to the cloud translation server in real time via the communication module includes: using a streaming segmented transmission method, after detecting a valid voice signal, continuously pushing incremental audio data to the cloud translation server in segments of a preset duration (e.g., 20 milliseconds to 200 milliseconds), without waiting for the complete sentence to end.

[0030] As an optional implementation, the real-time transmission of the environmental audio data to the cloud translation server via the communication module further includes: compressing and encoding the environmental audio data, and adaptively adjusting the bitrate parameter of the compression encoding according to the current network bandwidth and round-trip latency (reducing the bitrate to maintain transmission continuity when the network quality index is lower than a preset threshold, and increasing the bitrate to improve audio quality when the network quality recovers), so as to maintain transmission continuity when network quality fluctuates. The communication module monitors the network round-trip latency and available bandwidth with the cloud translation server in real time, specifically by measuring the data packet round-trip time and acknowledging the reception rate.

[0031] As an optional implementation, receiving the translation results returned by the cloud translation server includes: receiving the translation results word by word or phrase by phrase in a streaming manner, and during the receiving process, the output module begins to present the received partial translation content without waiting for the complete translation result of the whole sentence to be returned.

[0032] As an optional implementation, when performing speaker separation, the cloud-based translation server uses a pre-registered user voiceprint model to identify and filter audio segments belonging to the user's own voice, performing the translation process only on the remaining non-user voice segments. The user voiceprint model is generated when the user first wears the glasses, by collecting bone vibration signals from the user's bones as they read a preset text via the bone conduction microphone, extracting voiceprint feature vectors, and uploading them to the cloud for registration. The filtering includes calculating the similarity between the voiceprint features of each time-frequency band in the air-conducted speech signal and the user voiceprint model, suppressing or eliminating audio segments with similarity higher than a preset threshold.

[0033] As an optional implementation, the output module includes a bone conduction speaker; presenting the translation result through the output module includes: playing the speech-synthesized audio of the translation result through the bone conduction speaker in a bone conduction transmission mode that is only perceptible to the wearer.

[0034] As an optional implementation, the output module includes a display module; presenting the translation result through the output module includes: displaying the text content of the translation result as text subtitles within the wearer's field of vision through the display module.

[0035] As an optional implementation, the cloud translation server is also used to identify the speaker's emotional state or tone characteristics in the target speech segment; the translation result is supplemented with marking information corresponding to the emotional state or tone characteristics; when the output module presents the translation result, it synchronously presents the marking information with visual identifiers or specific tones.

[0036] As an optional implementation, after the cloud translation server completes the translation, the text content of the translation result is pushed to the mobile terminal associated with the smart glasses in the form of a notification or message card, so that the user can view the translation record afterward.

[0037] As an optional implementation, the translation results returned by the cloud translation server are cached in the local storage of the smart glasses; when the communication link with the cloud translation server is detected to be interrupted and the interruption duration exceeds a preset threshold, the smart glasses are automatically switched to a preset local emergency prompt mode, and a prompt message is sent to the wearer through the output module to indicate that the current translation service is unavailable.

[0038] In a specific embodiment, Li Ming (a Chinese national who speaks Chinese) and Mr. Zhang (a French national who speaks French) have a face-to-face conversation at an international exhibition. Both wear smart glasses equipped with a cloud collaborative translation function, and have preset their native language and target translation language in the mobile App. First, Mr. Zhang speaks in French: "J'ai beaucoup entendu parlerde votre entreprise." The workflow on Li Ming's glasses side: The air conduction microphone collects mixed audio (Mr. Zhang's French speech + exhibition background noise, etc.). The bone conduction microphone mainly collects the bone vibration signals of Li Ming himself, and at this time captures Li Ming uttering "hmm". The communication module streams the two audio tracks to the cloud. The cloud uses the bone conduction signal to filter out Li Ming's "hmm", and the remaining clean French speech is recognized and translated into Chinese: "I have long heard of your company's reputation." The output module privately broadcasts this Chinese sentence in Li Ming's ear through the bone conduction speaker, while the lens displays the subtitle. Li Ming understands it. After that, Li Ming responds in Chinese and says: "Thank you, I have also been looking forward to meeting you." The workflow on Mr. Zhang's glasses side (completely symmetrical to the above): The air conduction microphone collects mixed audio (Li Ming's Chinese speech + exhibition background noise, etc.). The bone conduction microphone mainly collects the bone vibration signals of Mr. Zhang himself. The communication module streams the two audio tracks to the cloud. The cloud uses the bone conduction signal to filter out Mr. Zhang's own speech, and the remaining Chinese speech is recognized and translated into French: "Merci, j'attendais aussi cette rencontre avec impatience." The output module privately broadcasts this French sentence in Mr. Zhang's ear through the bone conduction speaker, while the lens displays the subtitle. Throughout the whole process, each of the two persons hears the translation of the other's speech, and their own speech will not be translated and transmitted back, just as if a simultaneous interpreter is translating beside each of their ears.

[0039] Based on the same inventive concept as the above method, in another embodiment of the present disclosure, a smart glasses multilingual simultaneous interpretation system based on cloud collaborative translation is further disclosed, comprising: an audio acquisition module, configured to collect environmental audio data in real time, where the environmental audio data includes the speech of the wearer himself / herself and the speech of at least one conversation party; a communication module, configured to transmit the environmental audio data to a cloud translation server in real time, and receive a translation result returned by the cloud translation server; wherein the translation result is generated after the cloud translation server performs speaker separation, speech recognition and translation processing on the environmental audio data, and the translation result corresponds to the translated content of the target speech segment that does not belong to the wearer's own speech in the environmental audio data; and an output module, configured to present the translation result.

[0040] This invention offloads all translation computation tasks to a cloud server, with the glasses only responsible for audio acquisition, streaming, and result output. This allows the glasses hardware to fully utilize the high-quality translation capabilities of large-scale cloud-based translation models without requiring a high-performance processor, generating high power consumption, or causing excessive heat. This achieves an optimal balance between lightweight physical form and translation quality. Cloud-based collaborative translation-based multilingual simultaneous interpretation in smart glasses refers to a technical means of representing the feature information of audio data through image data, thereby achieving mutual expression between audio and image modalities.

[0041] Furthermore, this embodiment provides a cloud-based collaborative translation-based smart glasses multilingual simultaneous interpretation system, including a first smart glasses, a second smart glasses, and a cloud translation server. The first smart glasses are worn by a first wearer, and the second smart glasses are worn by a second wearer. The first and second wearers communicate face-to-face using different languages. The first and second smart glasses have the same hardware configuration, both including: at least one bone conduction microphone, at least one air conduction microphone, a communication module, a bone conduction speaker, and a display module. The bone conduction microphone is used to pick up the bone vibration signals when the wearer speaks, and the air conduction microphone is used to pick up the air conduction speech signals in the wearer's environment. The first and second smart glasses establish and maintain a persistent network connection with the same cloud translation server through their respective communication modules.

[0042] Taking the example of the first wearer speaking Chinese and the second wearer speaking French, a complete dialogue interaction process is as follows: (1) The second wearer speaks French. The second wearer's air conduction microphone collects ambient audio, which includes the French speech of the second wearer, as well as possible Chinese responses from the first wearer, ambient background noise, etc. At the same time, the second wearer's bone conduction microphone collects the bone vibration signal when the second wearer speaks. The communication module of the second smart glasses uploads these two audio signals—the mixed audio from air conduction and the pure reference signal from bone conduction—to the cloud translation server in real time in a streaming segmented manner.

[0043] (2) Speaker separation and translation are performed in the cloud. After receiving the dual-channel audio uploaded by the second smart glasses, the cloud translation server performs the following processing: using the bone conduction signal of the second smart glasses as a reference, it separates and filters out the second wearer's own voice component from the air conduction mixed audio to obtain the pure first wearer's voice segment. Since the first wearer is not speaking at this time, the actual separation result of the first wearer's voice segment is empty or contains only a few interjections. The remaining audio is the second wearer's own French voice (which has been identified as "the second wearer himself", but the second wearer does not need to listen to his own translation, so the cloud marks it as "the other party's voice" to be translated). In the symmetrical architecture, the dual-channel audio uploaded by each pair of glasses is used to separate the "other party's" voice. In the mixed audio uploaded by the second glasses, the second wearer's own voice is dominant, and the first wearer's voice is "the mixed-in other party's voice". The cloud uses the bone conduction signal of the second glasses to filter out the second wearer's own voice, and what is left is the first wearer's voice. This can be understood as follows: After the cloud translation server receives the dual-channel audio uploaded by the second smart glasses: using the bone conduction signal as a reference, it subtracts the second wearer's own speech from the mixed audio, and the remaining audio component is the first wearer's Chinese speech. The cloud recognizes this Chinese speech, determines that the language is Chinese, which is consistent with the first wearer's preset native language, and therefore no translation is needed (or the translation is empty). At this time, the first wearer is not speaking, and the processing result is empty. After the cloud translation server receives the dual-channel audio uploaded by the first smart glasses: using the bone conduction signal of the first glasses as a reference, it subtracts the first wearer's own speech from the mixed audio, and the remaining audio component is the second wearer's French speech. The cloud recognizes this French speech, translates it into Chinese, and obtains the translation result: "I have long admired your company's name." (3) The cloud will push the translation results to the first smart glasses in a targeted manner. The cloud translation server will stream the Chinese translation results word by word back to the first smart glasses, without pushing them to the second smart glasses (because the second wearer does not need to hear the Chinese translation in their own language).

[0044] (4) The first wearer receives the translation result. The communication module of the first smart glasses receives the translation result transmitted from the cloud and broadcasts it in the ear of the first wearer in a private voice through a bone conduction speaker: "I have long admired your company." At the same time, the display module displays the Chinese content in the form of text subtitles.

[0045] (5) The first wearer responds in Chinese. The first wearer says in Chinese, “Thank you, I look forward to this collaboration.” The air conduction microphone of the first smart glasses collects ambient audio (including the first wearer’s Chinese voice and possibly the second wearer’s French response), and the bone conduction microphone collects the first wearer’s bone vibration signal. The dual audio streams are uploaded to the cloud. The cloud translation server uses the bone conduction signal from the first glasses to filter out the first wearer’s own voice, leaving the second wearer’s French voice (at this time, the second wearer is not speaking, so the result is empty). At the same time, the cloud uses the dual audio streams uploaded by the second glasses to filter out the second wearer’s own voice and obtains the first wearer’s Chinese voice, which is translated into French: “Merci, j'attends cette collaboration avec impatience.” The cloud pushes the French translation result to the second smart glasses in a streaming manner. The second smart glasses privately broadcast the French translation through the bone conduction speaker and simultaneously display the French subtitles.

[0046] The above steps are repeated cyclically within a single conversation. The first wearer always hears the Chinese translation of the second wearer's French speech; the second wearer always hears the French translation of the first wearer's Chinese speech. Both individuals speak naturally in their native languages, with the glasses providing private translation behind their ears, acting like an invisible simultaneous interpreter offering real-time two-way translation services.

[0047] In this embodiment, the bone conduction microphone of each pair of glasses provides the wearer with a pure voiceprint reference. The cloud uses this reference to accurately separate "what the other party said." Compared with the single-glasses solution, there is no need to manually specify the translation direction or guess which speaker is the target translation object. Two pairs of glasses and one cloud naturally form a two-way translation closed loop, with simple logic and a smooth user experience.

[0048] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0049] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0050] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the electronic devices, apparatuses, etc. described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0051] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multilingual simultaneous interpretation method for smart glasses based on cloud-based collaborative translation, applied to smart glasses, characterized in that: The smart glasses include a microphone module, a communication module, and an output module, including: The microphone module collects ambient audio data in real time. The environmental audio data is transmitted to the cloud translation server in real time via the communication module. The system receives translation results returned by the cloud translation server, wherein the translation results are generated by the cloud translation server after performing speaker separation, language recognition and translation processing on the environmental audio data, and the translation results correspond to the translation content of the target speech segment in the environmental audio data that does not belong to the user's own speech; The translation result is presented through the output module.

2. The method according to claim 1, characterized in that, The microphone module includes at least one bone conduction microphone and at least one air conduction microphone; The real-time acquisition of environmental audio data via the microphone module includes: acquiring bone vibration signals when the wearer speaks through the bone conduction microphone, acquiring air conduction speech signals in the environment through the air conduction microphone, and fusing the two to suppress the wearer's own speech components in the environmental audio data.

3. The method according to claim 1, characterized in that, The step of transmitting the environmental audio data to the cloud translation server in real time via the communication module includes: using a streaming segmented transmission method, after detecting a valid voice signal, continuously pushing incremental audio data to the cloud translation server in segments of preset duration, without waiting for the complete sentence to end.

4. The method according to claim 1, characterized in that, The step of transmitting the environmental audio data to the cloud translation server in real time via the communication module further includes: compressing and encoding the environmental audio data, and adaptively adjusting the bitrate parameter of the compression encoding according to the current network bandwidth and round-trip latency, so as to maintain transmission continuity when network quality fluctuates.

5. The method according to claim 1, characterized in that, The process of receiving translation results from the cloud translation server includes: receiving the translation results word by word or phrase by phrase in a streaming manner, and during the receiving process, the output module begins to present the received partial translation content without waiting for the complete translation result of the whole sentence to be returned.

6. The method according to claim 1, characterized in that, The output module includes a bone conduction speaker; presenting the translation result through the output module includes: playing the speech-synthesized audio of the translation result through the bone conduction speaker in a bone conduction transmission mode that is only perceptible to the wearer.

7. The method according to claim 1, characterized in that, The output module includes a display module; presenting the translation result through the output module includes: displaying the text content of the translation result as text subtitles within the wearer's field of vision through the display module.

8. The method according to claim 1, characterized in that, Also includes: After the cloud translation server completes the translation, the translated text is pushed to the mobile terminal associated with the smart glasses in the form of a notification or message card, so that the user can view the translation record afterward.

9. The method according to claim 1, characterized in that, Also includes: The translation results returned by the cloud translation server are cached in the local storage of the smart glasses; When a communication link with the cloud translation server is detected to be interrupted and the interruption duration exceeds a preset threshold, the smart glasses are automatically switched to a preset local emergency alert mode, and a prompt message is sent to the wearer through the output module to indicate that the current translation service is unavailable.

10. A multilingual simultaneous interpretation system for smart glasses based on cloud-based collaborative translation, characterized in that: include: An audio acquisition module is used to acquire environmental audio data in real time, the environmental audio data including the wearer's own voice and the voice of at least one other party in the conversation; The communication module is used to transmit the environmental audio data to the cloud translation server in real time and receive the translation results returned by the cloud translation server; wherein, the translation results are generated by the cloud translation server after performing speaker separation, language recognition and translation processing on the environmental audio data, and the translation results correspond to the translation content of the target speech segment in the environmental audio data that does not belong to the wearer's own voice; The output module is used to present the translation results.

Images