Artificial-intelligence-based translation system, smart glasses for translation, and translation method

Abstract

An artificial-intelligence-based translation system, smart glasses for translation, and a translation method. The translation system comprises: first smart glasses, a first smart mobile terminal and a cloud server, wherein the cloud server is configured with a generative artificial intelligence large language model; the first smart glasses pick up first speech of a first language from a preset direction and send the first speech to the first smart mobile terminal; the first smart mobile terminal converts the first speech into first text and sends the first text to the cloud server; by means of the model, the cloud server translates the first text on the basis of a translation prompt to obtain second text of a second language and sends the second text to the first smart mobile terminal; and the first smart mobile terminal converts the second text into second speech and sends the second speech to the first smart glasses for playing. The present application realizes high-precision smart translation based on smart glasses, thereby increasing product stickiness.

Description

Artificial intelligence-based translation system, smart glasses for translation, and translation method

[0001] This application claims priority to Chinese patent application number CN 2023112227424, entitled “ARTIFICIAL INTELLIGENCE-BASED TRANSLATION SYSTEM, SMART GLASSES FOR TRANSLATION AND TRANSLATION METHOD,” filed with the Patent Office of the State Intellectual Property Office of China on September 20, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of the present application relate to the field of smart glasses technology, and in particular to an artificial intelligence-based translation system, smart glasses for translation, and a translation method. Background Art

[0003] With the development of computer technology, smart glasses are becoming more and more popular. However, existing smart glasses are expensive, and in addition to their own functions as smart glasses, they usually only have the functions of listening to music and making or receiving calls. The functions are relatively simple and the product stickiness is low. Technical issues

[0004] The embodiments of the present application provide an artificial intelligence-based translation system, smart glasses for translation, and a translation method, which are used to implement a high-precision translation function based on smart glasses, thereby improving the practicality, interactivity, and intelligence of smart glasses, and increasing product stickiness. Technical Solutions

[0005] In one aspect, an embodiment of the present application provides an artificial intelligence-based translation system, comprising: first smart glasses, a first smart mobile terminal, and a cloud server, wherein the cloud server is configured with a generative artificial intelligence large language model;

[0006] The first smart glasses are configured to pick up a first voice in a first language from a preset direction and send the first voice to the first smart mobile terminal via Bluetooth;

[0007] The first smart mobile terminal is configured to convert the first speech into a first text using a speech-to-text engine, and send the first text to the cloud server, wherein the speech-to-text engine is disposed in the first smart mobile terminal or the cloud server;

[0008] The cloud server is configured to translate the first text using the generative artificial intelligence large language model according to the translation prompt to obtain a second text in a second language, and send the second text to the first smart mobile terminal;

[0009] The first smart mobile terminal is further configured to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses via Bluetooth, wherein the text-to-speech engine is disposed in the first smart mobile terminal or in the cloud;

[0010] The first smart glasses are further configured to play the second voice.

[0011] In one aspect, an embodiment of the present application further provides an artificial intelligence-based translation smart glasses, comprising: a sound pickup device, an output device, a processor, and a memory, wherein the processor is electrically connected to the sound pickup device, the output device, and the memory;

[0012] The memory stores one or more programs executable by the processor, wherein the one or more programs include multiple instructions, and the multiple instructions are used to:

[0013] Picking up a first speech in a first language from a preset direction by the sound pickup device;

[0014] Translate the first speech into a second speech in a second language using a local or cloud-based generative artificial intelligence language model; and

[0015] The second speech is outputted through the output device.

[0016] In one aspect, an embodiment of the present application further provides an artificial intelligence-based translation method, which is applied to a smart mobile terminal. The method includes:

[0017] Receiving a first speech in a first language sent by the smart wearable device, converting the first speech into a first text using a local or cloud-based speech-to-text engine, and obtaining a translation prompt;

[0018] Translating the first text into a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence language model;

[0019] The second text is converted into a second voice using a local or cloud-based text-to-speech engine, and the second voice is sent to the smart wearable device for playback. Beneficial effects

[0020] The various embodiments of this application, by combining smart glasses, a smart mobile terminal, and a server, achieve intelligent translation based on smart glasses, thereby improving the practicality, interactivity, and intelligence of the smart glasses, and increasing product stickiness. Furthermore, due to the scalability and self-creativity of the generative AI large language model, translation accuracy can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] FIG1 is a schematic diagram of the structure of an artificial intelligence-based translation system provided in one embodiment of the present application;

[0023] FIG2 is an application scenario diagram of the translation system shown in FIG1 ;

[0024] FIG3 is a schematic diagram of the structure of an artificial intelligence-based translation system provided in another embodiment of the present application;

[0025] FIG4 is a schematic diagram of a first system framework based on the translation system shown in FIG1 and FIG3 ;

[0026] FIG5 is a schematic diagram of a second system framework based on the translation system shown in FIG1 and FIG3;

[0027] FIG6 is a schematic diagram of a third system framework based on the translation systems shown in FIG1 and FIG3 ;

[0028] FIG7 is a schematic diagram of a fourth system framework based on the translation systems shown in FIG1 and FIG3 ;

[0029] FIG8 is a schematic diagram of the internal structure of the AI-based translation smart glasses provided in one embodiment of the present application;

[0030] FIG9 is a schematic diagram of the external structure of the smart glasses in the embodiment shown in FIG8 ;

[0031] FIG10 is a flowchart of an artificial intelligence-based translation method provided in one embodiment of the present application;

[0032] FIG11 is a schematic diagram of implementing the first translation mode in the translation system shown in FIG1 and the translation method shown in FIG10 ;

[0033] FIG12 is a schematic diagram of implementing the second translation mode in the translation system shown in FIG1 and the translation method shown in FIG10 ;

[0034] FIG13 is a schematic diagram of implementing the third translation mode in the translation system shown in FIG1 and the translation method shown in FIG10 ;

[0035] FIG14 is a schematic diagram of implementing the fourth translation mode in the translation system shown in FIG1 and the translation method shown in FIG10 . Modes for Carrying Out the Invention

[0036] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0038] Furthermore, the terms “first,” “second,” “third,” etc., are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0039] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0040] 1 , which is a schematic diagram of the structure of an artificial intelligence-based translation system according to an embodiment of the present application. As shown in FIG1 , the translation system 100 includes: first smart glasses 110 , a first smart mobile terminal 120 , and a cloud server 130 .

[0041] In each embodiment of the present application, a corresponding application (APP) is installed in each smart glasses or each smart mobile terminal. The smart glasses and the smart mobile terminal establish a data connection via Bluetooth and use the APP to interact with data.

[0042] The smart glasses may be open smart glasses, and their specific structure may refer to the relevant description in the embodiments of the smart glasses shown in FIG8 and FIG9 below, which will not be repeated here.

[0043] The smart mobile terminal may include, but is not limited to, a cellular phone, a smart phone, other wireless communication devices, a personal digital assistant, an audio player, other media players, a music recorder, a video recorder, a camera, other media recorders, a smart radio, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a digital camera, and a smart wearable device (such as a smart watch, a smart bracelet, etc.). The smart mobile terminal may also be installed with an Android, iOS, or other operating system.

[0044] The cloud server 130 may be a single server or a distributed server cluster consisting of multiple servers, on which a Generative Artificial Intelligence Large Language Model (GAILLM) is configured.

[0045] The GAILLM can be, for example, but not limited to: ChatGPT of Open AI, Bard of Google, and other models with similar functions. The GAILLM integrates multiple functions, such as: obtaining task execution commands corresponding to the voice commands issued by the user in natural language (as shown in Figure 2, sending messages, sending emails, making calls, searching the Internet, calling network services, calling other third-party software development tools (Software Development Kit, SDK) to execute tasks provided by the SDK), translation, etc. Among them, the part of the GAILLM that obtains task execution commands is trained with a large number of semantics and corresponding task execution commands as samples. The translation part of the GAILLM is trained with a large number of words and / or sentences in the source language and the corresponding words and / or sentences in the target language as samples. Among them, one source language can correspond to multiple different target languages.

[0046] Specifically, the first smart glasses 110 are configured to pick up a first speech in a first language (i.e., the source language) from a preset direction and transmit the first speech to the first smart mobile terminal 120 via Bluetooth. Optionally, the preset direction may be in front of the user of the first smart glasses 110, i.e., in listening mode. In this mode, the language used by the user's chat partner is the source language. Alternatively, the preset direction may be toward the user's mouth, i.e., in speaking mode. In this mode, the language used by the user is the source language. The first smart glasses 110 may switch the current translation mode to a corresponding mode based on the user's voice command, or the user pressing a control button on the first smart glasses 110 for switching translation modes, or the user operating the user interface (UI) of the first smart mobile terminal 120. The source and target languages ​​are determined based on the current translation mode.

[0047] The first smart mobile terminal 120 is configured to convert the first speech into a first text using a speech-to-text engine, and send the first text to the cloud server 130 , wherein the speech-to-text engine is disposed on the first smart mobile terminal 120 or in the cloud (e.g., a speech-to-text server, a model server, a translation server, or other cloud server).

[0048] The cloud server 130 is configured to translate the first text through the GAILLM according to the translation prompt to obtain a second text in a second language (ie, the target language), and send the second text to the first smart mobile terminal 120 .

[0049] The first smart mobile terminal 120 is further configured to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses 110 via Bluetooth, wherein the text-to-speech engine is configured in the first smart mobile terminal 120 or in the cloud (e.g., a text-to-speech server, a model server, a translation server, or other cloud server).

[0050] The first smart glasses 110 are further configured to play the second voice.

[0051] The above-mentioned translation prompt may include instruction information for instructing GAILLM to perform the translation task. Furthermore, the translation prompt may also include: identification information of the first language and / or the second language. The translation prompt can specifically be generated by a prompt generator. The prompt generator can be a software module or a microcontroller configured with the software module, which is used to generate corresponding prompt information based on the semantics in the parsed user voice, and the semantics are obtained by parsing each user voice. Alternatively, the prompt generator can also generate translation prompts according to other preset rules, for example, receiving a translation instruction, receiving text sent by a specific object, receiving language identification information in the received data, etc.

[0052] The prompt generator can be configured in the smart glasses 110, the first smart mobile terminal 120 or any server in the cloud.

[0053] Optionally, the prompt generator may obtain identification information of the first language and / or the second language from the smart glasses 110 or the first smart mobile terminal 120 .

[0054] Optionally, in other embodiments of the present application, the first smart mobile terminal 120 is further configured to generate the translation prompt and send the translation prompt to the cloud server.

[0055] Optionally, in other embodiments of the present application, a mobile application (mobile APP) is installed on the first smart mobile terminal 120 .

[0056] The first smart mobile terminal 120 is further configured to send a translation instruction to the first smart glasses 110 via the mobile application in response to a preset user operation. The preset operation may be, for example, but not limited to, any of the following operations: the user pressing a preset virtual button for starting translation on the UI of the mobile application, the user performing a preset sliding gesture on the screen of the first smart glasses 110 or the first smart mobile terminal 120, the user pressing a translation control button provided on the temple of the first smart glasses 110, or the user issuing a preset translation voice command, etc.

[0057] The first smart glasses 110 are further configured to enter a translation mode in response to the translation instruction, and pick up the first voice from the preset direction in the translation mode.

[0058] Optionally, in other embodiments of the present application, as shown in Figure 3, a mobile application is installed on the first smart mobile terminal 120, and the cloud server 130 includes a translation server 131 and a model server 132 (or, also referred to as a GAILLM server), and the GAILLM is configured in the model server 132.

[0059] The first smart mobile terminal 120 is further configured to send a voice pickup instruction to the first smart glasses 110 through the mobile application when the user presses a button on the screen of the first smart mobile terminal 120 , wherein the button is generated by the mobile application.

[0060] The first smart glasses 110 are further configured to pick up the first voice coming from the preset direction in response to the voice pickup instruction.

[0061] The first smart mobile terminal 120 is further configured to send the first text to the translation server 131 via the mobile application when the user releases the button.

[0062] The translation server 131 is configured to generate the translation prompt and send the first text and the translation prompt to the model server 132 .

[0063] The model server 132 is configured to translate the first text according to the translation prompt through the GAILLM to obtain the second text, and send the second text to the translation server 131 .

[0064] The translation server 131 is further configured to send the second text to the first smart mobile terminal 120 .

[0065] Optionally, in other embodiments of the present application, as shown in Figure 3, the cloud server 130 includes a translation server 131 and a model server 132, a mobile application is installed on the first smart mobile terminal 120, and the GAILLM is configured in the model server 132.

[0066] The first smart glasses 110 are further configured to pick up the first voice from the preset direction when the user presses a virtual button on the temple of the first smart glasses 110 .

[0067] The first smart glasses 110 are further configured to send notification information to the mobile application via Bluetooth when the user releases the virtual button.

[0068] The first smart mobile terminal 120 is further configured to send the first text to the translation server 131 via the mobile application according to the notification information.

[0069] The translation server 131 is configured to generate the translation prompt and send the first text and the translation prompt to the model server 132 .

[0070] The model server 132 is configured to translate the first text according to the translation prompt through the GAILLM to obtain the second text, and send the second text to the translation server 131 .

[0071] The translation server 131 is further configured to send the second text to the first smart mobile terminal 120 .

[0072] Optionally, in other embodiments of the present application, a mobile application is installed on the first smart mobile terminal 120, and the first smart mobile terminal 120 is also used to: generate a translation record based on the second text, or the first text and the second text, through the mobile application, and display the translation record on the screen of the first smart mobile terminal 120; and save the translation record in the first smart mobile terminal 120 or the storage server through the mobile application.

[0073] Optionally, in other embodiments of the present application, the first smart mobile terminal 120 is also used to share the translation record through the mobile application in accordance with the sharing method indicated by the operation when it detects that the user operates the share button in the user interface of the mobile application.

[0074] The above-mentioned sharing may include, for example but not limited to: sharing the translation record with friends of the first smart glasses 110 registered in instant messaging software such as WeChat, or sharing the translation record with other terminal devices via email, or sharing the translation record on online social platforms such as Facebook, Instagram, etc. The user can view the translation record in full translation or in a comparison between the original text and the translation on the sharing terminal. The sharing may be part or all of the translation record generated within one or more translation cycles specified by the sharing user. The process from the smart glasses entering the translation mode to exiting the translation mode can be regarded as a translation cycle.

[0075] Optionally, in other embodiments of the present application, a mobile application is installed on the first smart mobile terminal 120, and the first smart mobile terminal 120 is also used to: generate the second voice with a preset playback speed through the mobile application, and send the second voice with the preset playback speed to the first smart glasses 110 via the Bluetooth for playback.

[0076] Optionally, in other embodiments of the present application, a mobile application is installed on the first smart mobile terminal 120 .

[0077] The first smart mobile terminal 120 is further configured to set the second language according to a setting operation performed by the user through the mobile application.

[0078] The first smart glasses 110 are further configured to perform a voice pickup operation and send the picked-up voice to the first smart mobile terminal 120 in real time in the form of a data stream via Bluetooth.

[0079] The first smart mobile terminal 120 is further configured to: use the speech-to-text engine to detect whether the language of the received speech is the first language; when the language is the first language, send a start translation instruction to the first smart glasses 110 via the Bluetooth, and convert the received speech as the first speech into a first text via the speech-to-text engine, and send the first text and the translation prompt to the model server 132, wherein the translation prompt includes information in the first language and the second language; and when silence of a preset duration is detected, send a stop translation instruction to the first smart glasses 110 via the Bluetooth.

[0080] The first smart glasses 110 are further configured to enter a translation mode in response to the start translation instruction, and exit the translation mode and end the voice pickup operation in response to the stop translation instruction.

[0081] Specifically, upon powering on or in response to a user-triggered monitoring command, the smart glasses perform a voice pickup operation to monitor the user's voice in real time and transmit the monitored voice in real time to the smart mobile terminal in the form of a data stream. The voice is buffered in the smart glasses' memory. During the monitoring process, the smart glasses utilize multiple threads to continuously overwrite older voices with newly picked up voices of a preset duration in memory 203 (for example, overwriting previously picked up 3 seconds of voice with the currently picked up 3 seconds of voice). Simultaneously, the buffered newly picked up voices are transmitted to the smart mobile terminal. The smart mobile terminal stores the received voice in local memory and determines whether the language of the voice is the first language. If the language is not the first language, the smart mobile terminal deletes the previously received voice from the local memory and continues to perform language determination based on newly received voices until a preset duration has elapsed without receiving any new voices from the smart glasses, or until a notification message is received from the smart glasses to stop monitoring. If the language is the first language, the smart mobile terminal transmits a start translation command to the smart glasses and performs translation on the received voice. The smart glasses enter translation mode according to the translation start instruction and continue to perform the voice pickup operation. Further, when a preset period of silence is detected in the voice stored in the local memory, the smart mobile terminal sends a translation stop instruction to the smart glasses, so that the smart glasses exit the translation mode and end the voice pickup operation according to the translation stop instruction.

[0082] Optionally, in other embodiments of the present application, as shown in FIG3 , the cloud server 130 includes a translation server 131 and a model server 132 , and the GAILLM is configured in the model server 132 .

[0083] The first smart glasses 110 are further configured to send the first speech to the translation server 131 via a wireless network.

[0084] The translation server 131 is configured to convert the first speech into a first text using a speech-to-text engine, generate a translation prompt, and send the first text and the generated translation prompt to the model server 132 .

[0085] The model server 132 is configured to translate the first text according to the translation prompt through the GAILLM to obtain the second text, and send the second text to the translation server 131 .

[0086] The translation server 131 is further configured to convert the second text into a second speech using a text-to-speech engine, and send the second speech to the first smart glasses 110 via the wireless network.

[0087] Optionally, in other embodiments of the present application, the preset direction points to the mouth of the user of the first smart glasses 110 . As shown in FIG3 , the translation system 100 further includes a second pair of smart glasses 140 and a second smart mobile terminal 150 .

[0088] The first smart mobile terminal 120 is further configured to send the second voice to the second smart mobile terminal 150 .

[0089] The second smart mobile terminal 150 is configured to send the second voice to the second smart glasses 140 .

[0090] The second smart glasses 140 are configured to play the second voice.

[0091] Optionally, in other embodiments of the present application, the first smart mobile terminal 120 is also used to generate a translation record in real time based on the first text and the second text through a mobile application on the first smart mobile terminal 120, save the translation record in the first smart mobile terminal 120, display the translation record, and synchronize the translation record to the second smart mobile terminal 150.

[0092] The second smart mobile terminal 150 is further configured to save the translation record on the second smart mobile terminal 150 and display the translation record via a mobile application on the second smart mobile terminal 150 .

[0093] Specifically, the mobile application on the second smart mobile terminal 150 can display the translation record in a list on the screen of the second smart mobile terminal 150. The user of the second smart mobile terminal 150 can view different parts of the translation record by performing preset flipping gestures on the screen, using physical or virtual buttons on the second smart mobile terminal 150, or through voice control commands.

[0094] Optionally, in other embodiments of the present application, as shown in FIG3 , the cloud server 130 includes a translation server 131 , the translation system 100 further includes a plurality of second smart glasses, and the GAILLM is configured in the translation server 131 .

[0095] The first smart glasses 110 are further configured to pick up a first voice of a user of the first smart mobile terminal 120 and send the first voice to the first smart mobile terminal 120 via Bluetooth when the user of the first smart glasses 110 presses a button on the temple of the first smart glasses 110.

[0096] The first smart mobile terminal 120 is further configured to convert the first speech into the first text using the speech-to-text engine through a mobile application in the first smart mobile terminal 120 , and send the first text to the translation server 131 .

[0097] The translation server 131 is configured to generate a first translation prompt, and through the GAILLM, translate the first text according to the first translation prompt to obtain multiple third texts, and distribute each of the third texts to the corresponding second smart glasses, wherein the language of each third text corresponds to the language used by the user of each second smart glasses 140.

[0098] The second smart glasses 140 are configured to convert the received third text into a third voice using a text-to-speech engine, and play the third voice.

[0099] Optionally, in other embodiments of the present application, the second smart glasses 140 are further used to pick up a fourth voice of the user of the second smart glasses 140 when the user of the second smart glasses 140 presses a virtual button on the temple of the second smart glasses 140, convert the fourth voice into a fourth text through a speech-to-text engine, and send the fourth text to the translation server 131.

[0100] The translation server 131 is configured to generate a second translation prompt, and through the GAILLM, translate the fourth text according to the second translation prompt to obtain a fifth text, and send the fifth text to the first smart glasses 110, wherein the language of the fifth text corresponds to the language used by the user of the first smart glasses 110.

[0101] The first smart glasses 110 are further configured to convert the fifth text into a fifth voice using a text-to-speech engine, and play the fifth voice.

[0102] Optionally, in other embodiments of the present application, as shown in FIG3 , the cloud server 130 includes a model server 132 , and the GAILLM is configured in the model server 132 .

[0103] Referring to Figure 4 , a schematic diagram of a first system framework based on the translation system shown in Figures 1 and 3 is shown. As shown in Figure 4 , the translation system includes smart glasses 401, a smart mobile terminal 402, a text-to-speech server 403, a speech-to-text server 404, a translation server 405, and a model server 406. The text-to-speech engine is configured in text-to-speech server 403, the speech-to-text engine is configured in speech-to-text server 404, the prompt generator is configured in translation server 405, and GAILLM is configured in model server 406.

[0104] In steps 1 to 6 in FIG4 , when the user presses the virtual button on the smart glasses 401, the smart glasses 401 pick up the user's local voice (i.e., the first voice) and send the local voice in audio format to the mobile application (mobile APP) on the smart mobile terminal 402; the mobile The APP converts the local speech into local text (i.e., the first text) through data interaction with the speech-to-text server 404, and sends the local text returned by the speech-to-text server 404 to the translation server 405; the translation server 405 generates a translation prompt, and sends the translation prompt and the local text to the model server 406, so that the model server 406 uses GAILLM to perform a natural language translation operation on the local text according to the translation prompt and obtains a translated text (i.e., the second text); the translation server 405 receives the translated text returned by the model server 406, and forwards the translated text to the smart mobile terminal 402; the smart mobile terminal 402 converts the translated text into corresponding speech through data interaction with the text-to-speech server 403, and sends the corresponding speech returned by the text-to-speech server 403 to the smart glasses 401 for playback, and displays the translated text on the screen of the smart mobile terminal 402.

[0105] See Figure 5, which is a schematic diagram of a second system framework based on the translation system shown in Figures 1 and 3. As shown in Figure 5, the translation system includes smart glasses 401, a smart mobile terminal 402, and a model server 406. The text-to-speech engine, speech-to-text engine, and prompt generator are configured in the smart mobile terminal 402, and GAILLM is configured in the model server 406.

[0106] In steps 1 to 5 in FIG5 , when the user presses a virtual button on the smart glasses 401 being worn, the smart glasses 401 pick up the user's local voice (i.e., the first voice) and send the local voice in audio format to the mobile APP on the smart mobile terminal 402; the mobile APP converts the local voice into local text (i.e., the first text) through the voice-to-text engine, generates a translation prompt through the prompt generator, and sends the local text and the translation prompt to the model server 406, so that the model server 406 performs a natural language translation operation on the local text using GAILLM according to the translation prompt and obtains a translated text (i.e., the second text); the mobile APP receives the translated text returned by the model server 406, converts the translated text into corresponding voice through the text-to-speech engine, and sends the translated text to the smart glasses 401 for playback, while displaying the translated text on the screen of the smart mobile terminal 402.

[0107] It is understandable that the above-mentioned text-to-speech engine and speech-to-text engine can also be set up on one or more cloud servers, and the mobile APP performs text-to-speech and speech-to-text operations by interacting with the one or more servers.

[0108] Referring to Figure 6, Figure 6 is a schematic diagram of a third system framework based on the translation systems shown in Figures 1 and 3. As shown in Figure 6, the translation system includes: smart glasses 401 and a model server 406. GAILLM is configured in the model server 406.

[0109] In steps 1 to 6 in FIG6 , when the user presses a virtual button on the smart glasses 401 being worn, the smart glasses 401 pick up the user's local voice (i.e., the first voice) through the built-in client APP, convert the local voice into local text (i.e., the first text) using a local or cloud-based speech-to-text engine, generate a translation prompt using a local or cloud-based prompt generator, and send the local text and the translation prompt to the model server 406 so that the model server 406 performs a natural language translation operation on the local text using GAILLM according to the translation prompt and obtains a translated text (i.e., the second text); the smart glasses 401 receive the translated text returned by the model server 406 through the client APP, convert the translated text into corresponding speech using a local or cloud-based text-to-speech engine and play it, and display the translated text on the screen of the smart glasses 401.

[0110] Furthermore, the client APP can also generate a translation record based on the local text and the translated text, and allow the user to view the translation record on the screen of the smart glasses 401 through a preset operation. Furthermore, the client APP can also share the translation record with other terminals or online social platforms specified by the user based on the user's sharing voice command.

[0111] It can be understood that the above-mentioned text-to-speech engine, speech-to-text engine and prompt generator can be configured on one or more cloud servers. The client APP performs operations such as text-to-speech, speech-to-text and generates translation prompts by interacting with data of the one or more servers.

[0112] See Figure 7, which is a schematic diagram of a fourth system framework based on the translation system shown in Figures 1 and 3. As shown in Figure 7, the translation system includes smart glasses 401 and a smart mobile terminal 402. A text-to-speech engine, a speech-to-text engine, a prompt generator, and GAILLM are configured in smart mobile terminal 402.

[0113] In steps 1 to 6 in FIG7 , when the user presses a virtual button on the smart glasses 401 being worn, the smart glasses 401 pick up the user's local voice (i.e., the first voice) and send the local voice in audio format to the mobile APP on the smart mobile terminal 402; the mobile APP converts the local voice into local text (i.e., the first text) through the voice-to-text engine, generates a translation prompt through the prompt generator, and inputs the translation prompt and the local text into the GAILLM, so that the GAILLM performs a natural language translation operation on the local text according to the translation prompt and outputs a translated text (i.e., the second text); the mobile APP converts the translated text into corresponding voice through the text-to-speech engine and sends it to the smart glasses 401 for playback, while displaying the translated text on the screen of the smart mobile terminal 402.

[0114] Furthermore, if the mobile APP confirms that the smart glasses 401 have a screen based on the device information record, the translated text is sent to the smart glasses 401 for display on the screen of the smart glasses 401.

[0115] Furthermore, the mobile app can also generate a translation record based on the local text and the translated text, and allow the user to view the translation record on the screen of smart glasses 401 or the screen of smart mobile terminal 402 through a preset operation. Furthermore, the client app can also share the translation record with other terminals or online social platforms specified by the user based on the user's sharing voice command or the user's preset sharing operation on the mobile app's UI interface.

[0116] It is understandable that the above-mentioned text-to-speech engine, speech-to-text engine and prompt generator can also be configured on one or more cloud servers. The mobile APP performs operations such as text-to-speech, speech-to-text and generates translation prompts by interacting with the data of the one or more servers.

[0117] In this embodiment, by combining smart glasses, a smart mobile terminal, and a server, intelligent translation is achieved based on the smart glasses. This improves the practicality, interactivity, and intelligence of the smart glasses, while also increasing product stickiness. Furthermore, due to the scalability and self-creativity of the generative AI large language model, translation accuracy can be further improved.

[0118] Referring to Figures 8 and 9, Figure 8 is a schematic diagram of the internal structure of AI-based translation smart glasses provided in one embodiment of the present application, and Figure 9 is a schematic diagram of the external structure of the smart glasses in the embodiment shown in Figure 8. For ease of illustration, only the parts relevant to the embodiment of the present application are shown in the figure. In actual applications, the smart glasses may have more or fewer structures. As shown in Figure 8, the smart glasses 200 include: a sound pickup device 201, an output device 202, at least one processor 203, and at least one memory 204.

[0119] The processor 203 is electrically connected to the sound pickup device 201 , the output device 202 , and the memory 204 .

[0120] The sound pickup device 201 includes at least one microphone. Optionally, the sound pickup device 201 includes at least one directional microphone and / or at least one omnidirectional microphone. Specifically, the sound pickup device 201 can be a single microphone or a microphone array consisting of multiple microphones, for example, a microphone array consisting of at least one directional microphone and / or at least one omnidirectional microphone. The output device 202 includes at least one speaker.

[0121] The processor 203 includes a central processing unit (CPU) and a digital signal processor (DSP). The DSP is used to process the voice data acquired by the microphone 203. The CPU is preferably a microcontroller unit (MCU).

[0122] Memory 204 is a non-transitory memory, specifically including RAM (Random Access Memory) and flash memory components. It stores one or more programs executable by processor 203. These programs include multiple instructions for: capturing a first speech in a first language from a preset direction via sound pickup device 201; translating the first speech into a second speech in a second language using a local or cloud-based generative artificial intelligence language model; and outputting the second speech via output device 202.

[0123] Optionally, in other embodiments of the present application, the preset direction points forward of the user.

[0124] Optionally, in other embodiments of the present application, a client program is installed in the smart glasses 200, and the multiple instructions are also used to set the first language and / or the second language according to a first preset operation performed by the user in the client program.

[0125] Optionally, in other embodiments of the present application, the multiple instructions are also used to set the playback speed of the second voice according to a second preset operation performed by the user in the client program.

[0126] Optionally, in other embodiments of the present application, the smart glasses 200 also include a Bluetooth component 211 electrically connected to the processor 203, and the multiple instructions are also used to: receive a translation instruction sent by a mobile application on the smart mobile terminal through the Bluetooth component 211, enter a translation mode in response to the translation instruction, and in the translation mode, pick up the first voice from the preset direction through the sound pickup device 201.

[0127] The Bluetooth component 211 includes a Bluetooth signal transceiver and surrounding circuits, and can be specifically disposed in the inner cavity of the frame 209 and / or at least one temple 205. The Bluetooth component 211 can be connected to a smart mobile terminal such as a smartphone or smartwatch for phone calls, music, and data communications.

[0128] Optionally, in other embodiments of the present application, as shown in Figure 9, the smart glasses 200 also include: at least one temple 205 and a touch sensor-based virtual button 206 arranged on the temple 205 and electrically connected to the processor 203. The multiple instructions are also used to: when the user presses the virtual button, control the sound pickup device 201 to pick up the first voice; when the user releases the virtual button, send the first voice to the smart mobile terminal through the Bluetooth component 211, so that the smart mobile terminal uses the generative artificial intelligence large language model set on the smart mobile terminal or the cloud server to translate the first voice into the second voice; and receive the second voice sent by the smart mobile terminal through the Bluetooth component 211.

[0129] The at least one temple 205 may include, for example, a left temple and a right temple.

[0130] Optionally, in other embodiments of the present application, the multiple instructions are also used to: perform language detection on the first voice to determine whether the first language is a preset language through a local or cloud-based speech-to-text engine; when the first language is the preset language, enter translation mode, and convert the first voice into a first text through the speech-to-text engine; generate a translation prompt, and translate the first text through the generative artificial intelligence large language model to obtain a second text in the second language, wherein the translation prompt includes information of the first language and the second language; convert the second text into the second voice through a local or cloud-based speech-to-text engine; and exit the translation mode when silence of a preset length is detected by the sound pickup device 201.

[0131] Optionally, in other embodiments of the present application, the smart glasses 200 also include a wireless communication component 207 electrically connected to the processor 203, and the multiple instructions are also used to: send the first voice to the translation server through the wireless communication component 207, so that the translation server uses the generative artificial intelligence large language model set on the translation server or the model server to translate the first voice into the second voice; and receive the second voice sent by the translation server through the wireless communication component 207.

[0132] The wireless communication component 207 includes a wireless signal transceiver and its surrounding circuitry, and may be disposed within the frame 209 and / or the inner cavity of at least one temple 205 of the smart glasses 200. The wireless signal transceiver may utilize, but is not limited to, at least one of the following protocols for data transmission: WiFi (Wireless Fidelity), NFC (Near Field Communication), ZigBee, UWB (Ultra Wideband), RFID (Radio Frequency Identification), and cellular mobile communication protocols (e.g., 3G / 4G / 5G).

[0133] Optionally, in other embodiments of the present application, the output device 202 includes a front speaker, the preset direction of which points to the user's mouth, and the multiple instructions are also used to play the second voice through the front speaker.

[0134] Optionally, in other embodiments of the present application, the smart glasses 200 further include a Bluetooth component 211 electrically connected to the processor 203, the preset direction points to the user's mouth, and the multiple instructions are also used to send the second voice to an external player through the Bluetooth component 211 for playback.

[0135] Optionally, in other embodiments of the present application, the smart glasses 200 further include: an inertial measurement unit (IMU) sensor 208 electrically connected to the processor 203, and the multiple instructions are further used to: detect whether the first voice has ended by using a voice activity detection algorithm (Voice Activity Detection) and / or sensing data of the IMU sensor; and when it is detected that the first voice has ended, translate the first voice through the generative artificial intelligence large language model to obtain a second voice.

[0136] The voice activity detection algorithm works by detecting changes in speech energy to determine whether a user is speaking. When a user speaks, the energy of their speech increases, and the algorithm detects this energy change to distinguish whether speech is present or absent.

[0137] The sensing data of the IMU sensor is used to detect whether the first voice has ended, mainly by using the IMU sensor to detect the vibration (for example, accelerator) generated when the user speaks. If the vibration stops for more than a preset time, it means that the user has finished speaking.

[0138] Optionally, in addition to the inertial sensor 208, the smart glasses 200 may further include other data sensing components electrically connected to the processor 203, such as at least one of a positioning component, a touch sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer. The positioning component may be, but is not limited to, a positioning component based on the Global Positioning System (GPS) or Beidou satellites.

[0139] Furthermore, the smart glasses 200 may also include a frame 209 and a battery 210. The frame 209 may be, for example, a front frame with lenses (e.g., sunglass lenses, clear lenses, or corrective lenses). Preferably, the temples 205 are detachably connected to the frame 209.

[0140] Furthermore, at least one physical control button may be provided on the temple 205 and the frame 209, such as a power button, a volume control button, a translation mode switch button, and the like.

[0141] Optionally, in other embodiments of the present application, the smart glasses 200 further include a display screen electrically connected to the processor 203, and the display screen is used to display the above-mentioned second text or translation record.

[0142] The battery 210 is used to provide power support for the various electronic components of the above-mentioned smart glasses 200, such as the above-mentioned sound pickup device 201, the output device 202, at least one processor 203, at least one memory 204, the virtual button 206, the wireless communication component 207, the inertial sensor 208 and the Bluetooth component 211.

[0143] The electronic components of the above-mentioned smart glasses can be connected through a bus.

[0144] It should be noted that the relationships between the components of the aforementioned smart glasses can be either replacement or superposition. That is, a pair of smart glasses can be equipped with all of the components described in this embodiment, or, depending on the needs, selectively install a portion of the components. In the case of replacement, the smart glasses also include a peripheral connection interface, which can be, for example, at least one of a PS / 2 interface, a serial interface, a parallel interface, an IEEE1394 interface, and a USB (Universal Serial Bus) interface. The functionality of the replaced component can be implemented by a peripheral connected to the interface, such as an external speaker or sensor.

[0145] For details about the smart glasses not covered in this embodiment, please refer to the relevant descriptions in the translation system embodiments shown in Figures 1 to 7 and the translation method embodiments shown in Figures 10 to 14, and will not be repeated here.

[0146] In this embodiment, smart glasses utilize a generative AI language model to achieve intelligent translation, thereby improving the practicality, interactivity, and intelligence of the smart glasses and increasing product stickiness. Furthermore, due to the scalability and self-creativity of the generative AI language model, translation accuracy can be further improved.

[0147] Referring to FIG10 , FIG10 is a flow chart of an artificial intelligence-based translation method provided in an embodiment of the present application. The method can be applied to the first smart mobile terminal 120 or the second smart mobile terminal 150 in the translation system shown in FIG1 . As shown in FIG10 , the method includes:

[0148] S501, receiving a first speech in a first language sent by a smart wearable device, converting the first speech into a first text using a local or cloud-based speech-to-text engine, and obtaining a translation prompt;

[0149] S502, translating the first text according to the translation prompt using a local or cloud-based generative artificial intelligence language model to obtain a second text in a second language;

[0150] S503: Convert the second text into a second voice using a local or cloud-based text-to-speech engine, and send the second voice to the smart wearable device for playback.

[0151] The smart wearable device may include, but is not limited to: a smart helmet, smart headphones, smart earrings, a smart watch, and smart glasses as shown in Figures 8 and 9.

[0152] Optionally, the first smart mobile terminal 120 may generate the translation prompt through a built-in prompt generator, or the first smart mobile terminal 120 may also generate the translation prompt through a prompt generator configured in a server in the cloud.

[0153] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and the method further includes:

[0154] The mobile application responds to a preset operation of the user and sends a translation instruction to the smart wearable device, so that the smart wearable device responds to the translation instruction, enters a translation mode, and picks up the first voice in the translation mode.

[0155] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and before receiving the first voice in the first language sent by the smart wearable device, the method further includes:

[0156] When a user presses a button on the screen of the smart mobile terminal, a voice pickup instruction is sent to the smart wearable device through the mobile application to instruct the smart wearable device to pick up the first voice, wherein the button is generated by the mobile application;

[0157] The generative artificial intelligence language model is configured on the model server, and the first text is translated according to the translation prompt by the local or cloud-based generative artificial intelligence language model to obtain a second text in a second language, including:

[0158] When the user releases the button or receives a notification message sent by the smart wearable device, the first text is sent to the translation server through the mobile application, so that the translation server generates the translation prompt and sends the first text and the translation prompt to the model server, wherein the notification message is sent by the smart wearable device when the user of the smart wearable device releases the virtual button on the temple of the smart wearable device;

[0159] The second text forwarded by the translation server is received, wherein the second text is sent to the translation server by the model server, and the model server translates the first text according to the translation prompt by utilizing the generative artificial intelligence large language model.

[0160] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and the method further includes:

[0161] generating a translation record based on the second text, or the first text and the second text, through the mobile application, and displaying the translation record on a screen of the smart mobile terminal; and

[0162] The translation record is saved in the smart mobile terminal or storage server through the mobile application.

[0163] Optionally, in other embodiments of the present application, the method further includes: when detecting a user operation on a share button in the user interface of the mobile application, sharing the translation record through the mobile application in accordance with the sharing method indicated by the operation.

[0164] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and before sending the second voice to the smart wearable device for playback, the method further includes:

[0165] Through the mobile application, the playback speed of the second voice is adjusted to a preset playback speed.

[0166] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, the generative artificial intelligence large language model is configured on the model server, the smart wearable device performs a voice pickup operation, and sends the picked-up voice to the smart mobile terminal in real time via Bluetooth in the form of a data stream, and the method further includes:

[0167] Setting the second language according to a setting operation performed by the user through the mobile application;

[0168] Detecting, using the speech-to-text engine, whether the language of the received speech is the first language;

[0169] When the language is the first language, sending a translation start instruction to the smart wearable device via the Bluetooth to instruct the smart wearable device to enter a translation mode, and using the received voice as the first voice to perform the operations of converting the first voice into a first text using a local or cloud-based voice-to-text engine and generating a translation prompt; translating the first text into a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence large language model, wherein the translation prompt includes information of the first language and the second language; and

[0170] When silence of a preset length of time is detected, a stop translation instruction is sent to the smart wearable device via the Bluetooth to instruct the smart wearable device to exit the translation mode and end the voice pickup operation.

[0171] Optionally, in other embodiments of the present application, the first voice is a voice from a user of the smart wearable device, and the method further includes:

[0172] The second voice is sent to the target terminal, so that the target terminal sends the second voice to the target smart wearable device for playback.

[0173] Optionally, in other embodiments of the present application, a mobile application is installed in the smart mobile terminal, and the method further includes:

[0174] When the smart wearable device is paired with multiple slave smart wearable devices as a master smart wearable device, a translation prompt is generated through the mobile application;

[0175] Translating the first text using the generative artificial intelligence large language model according to the translation prompt generated by the smart wearable device to obtain the plurality of third texts, wherein the language of each of the third texts corresponds to the language used by the user of each of the slave smart wearable devices;

[0176] Each of the third texts is distributed to the corresponding slave smart wearable device, so that the corresponding slave smart wearable device uses a text-to-speech engine to convert the received third text into a third voice and play the third voice.

[0177] In the translation method provided in this application, the smart wearable device can have multiple translation modes, including: listening mode, speaking mode, pairing mode, and one-to-many mode. The one-to-many mode can be applied to application scenarios such as tour guides and conferences.

[0178] As an application example, take smart glasses and mobile phones as an example. In scenario A in monitoring mode:

[0179] Aa, users wearing smart glasses can use this mode to listen to the other party's foreign language voice through smart glasses, and let the smart glasses translate the voice into the local language and play it;

[0180] Ab, users can set the translated foreign and local languages ​​through the built-in application of smart glasses;

[0181] Ac, users can set the playback speed of local language voice;

[0182] Ad, users can start the translation function through the mobile application in their mobile phone to trigger the smart glasses to enter the translation mode;

[0183] Ae, the user presses the virtual button on the smart glasses, or the user presses the screen button on the UI interface of the mobile application on the screen of the mobile phone, and the smart glasses microphone starts to listen to the other party's foreign language voice;

[0184] Af, the smart glasses send the foreign language speech in audio format to the mobile application in real time via Bluetooth;

[0185] Ag, the mobile app runs a speech-to-text engine (cloud or local device) that converts speech into text in a foreign language;

[0186] Ah, at the same time, the mobile application displays transcripts in foreign languages, wherein transcripts in foreign languages ​​can be shared or saved;

[0187] Ai, when a virtual button on the smart glasses is released, or an on-screen button on the mobile application is released on the screen of a mobile phone, the mobile application sends the foreign language text to the translation server;

[0188] Aj, the translation server sends the foreign language text along with the translation hint (generated on the translation server) to GAILLM (cloud or local);

[0189] Ak, GAILLM translates the text format into the local language and sends it back to the translation server;

[0190] Al, the translation server sends the local language back to the mobile application in text format;

[0191] Am, a mobile app that converts local language text into speech via a text-to-speech engine (cloud or local device);

[0192] An, a mobile application that displays the translated local language transcript on the screen, where the translated local language transcript can be shared or saved on the phone;

[0193] Ao, a mobile application that sends the translated local language speech back to the smart glasses in audio format at the desired playback speed;

[0194] Ap, the smart glasses play the translated local language speech through the speakers.

[0195] As an application example, take smart glasses and mobile phones as an example. In scenario B in monitoring mode:

[0196] Ba, similar to scenario A, but users do not need to set a foreign language, they only need to send the local language;

[0197] Bb, the speech-to-text engine is able to automatically detect which foreign language is being used.

[0198] As an application example, take smart glasses and mobile phones as an example. In scenario C in monitoring mode:

[0199] Ca, extending to scenario B, the user does not need to use the virtual button on the smart glasses or the on-screen button on the mobile app to start or stop listening.

[0200] Cb, when the speech-to-text engine detects that the voice is in a foreign language, it will automatically start translating the picked up voice and stop translating after detecting a period of silence.

[0201] As an application example, take smart glasses and mobile phones as an example. In scenario D in monitoring mode:

[0202] Da, similar to scenario A, B, or C above, but without the mobile app.

[0203] Db, the smart glasses directly send the foreign language speech to the translation server and play the local language speech received by the translation server directly on the smart glasses.

[0204] As an application example, take smart glasses and mobile phones as an example. In scenario E of speaking mode:

[0205] Ea, users wearing smart glasses can use this mode to listen to their own local language voice through smart glasses, and let the smart glasses play the translated foreign language voice through the attached front speakers;

[0206] Eb, users can set local and foreign languages ​​through the built-in application of smart glasses;

[0207] Ec, users can set the playback speed of foreign language voice;

[0208] Ed, the user starts the translation function on the mobile application in the phone;

[0209] Ee, the user presses the virtual button on the smart glasses, or presses the screen button in the UI interface of the mobile application on the phone screen, and the smart glasses microphone starts listening to the user's local language voice;

[0210] Ef, the smart glasses send local language speech in audio format to the mobile phone mobile application in real time via Bluetooth;

[0211] For example, the mobile app runs a speech-to-text engine (cloud or local device) to convert the speech into text in the local language;

[0212] Eh, at the same time, the mobile application displays the transcript in the local language, wherein the transcript in the local language can be shared or saved;

[0213] Ei, when a virtual button on the smart glasses or a screen button on the mobile application is released, the mobile application sends the local language in text form to the translation server;

[0214] Ej, the translation server sends the local language text along with the translation hint (generated on the translation server) to the generative GAILLM (cloud or local);

[0215] Ek, GAILLM translates the local language text in text format into a foreign language text format and sends it back to the translation server;

[0216] El, the translation server sends the foreign language in text format back to the mobile application;

[0217] Em, the mobile application converts the foreign language text into the corresponding foreign language speech through a text-to-speech engine (cloud or local device);

[0218] En, the mobile application displays a foreign language transcript on a screen, wherein the foreign language transcript can be shared or saved;

[0219] Eo, the mobile app sends the foreign language speech back to the smart glasses in audio format at the desired playback speed;

[0220] Ep, the smart glasses play the foreign language voice through the front speakers so that the person talking can listen.

[0221] As an application example, take smart glasses and mobile phones as an example. In scenario F of speaking mode:

[0222] Fa, similar to scenario D, the foreign language speech is played through the mobile application in the phone instead of through the front speakers on the smart glasses or the external speakers paired with the smart glasses.

[0223] As an application example, take smart glasses and mobile phones as an example. In the speaking mode scenario G:

[0224] Ga, similar to scenario E or F, but the user does not need to release the virtual button on the glasses or the on-screen button on the mobile application to indicate the end of the speech;

[0225] Gb, users press and immediately release a virtual button on the smart glasses or an on-screen button on the mobile app to start listening;

[0226] Gc, smart glasses can automatically detect whether the user's speech has ended through voice activity detection algorithms and / or IMU signals.

[0227] As an application example, taking smart glasses and mobile phones as an example, in scene H of speaking mode, the following operations can also be performed:

[0228] Ha, similar to scenario G, users do not need to use the on-screen button in the mobile app or the virtual button on the smart glasses to start listening;

[0229] Hb, when the speech-to-text engine detects that the picked-up speech is in the local language, it will automatically start translating the picked-up speech.

[0230] As an application example, take smart glasses and mobile phones as an example. In scenario I of speaking mode:

[0231] Ia, similar to scenario E, G, or H, but without the mobile app;

[0232] Ib, the smart glasses directly send the local language speech to the translation server, and play the translated foreign language speech directly through the front speakers on the smart glasses or the external speakers paired with the smart glasses.

[0233] As an application example, take smart glasses and a mobile phone as an example. In scenario J of pairing mode:

[0234] Ja, users A and B wear smart glasses A and B, respectively, and use languages ​​A and B, respectively. Smart glasses A and B correspond to mobile phones A and B, respectively, and these smart glasses are paired for translation mode;

[0235] Jb, user A presses the virtual button on smart glasses A, or presses the on-screen button on mobile application A in phone A, and the microphone of smart glasses A starts listening to user A's voice in language A;

[0236] Jc, smart glasses A sends the speech of language A in audio format to mobile phone A in real time via Bluetooth;

[0237] Jd, mobile application A utilizes a speech-to-text engine (cloud or local device) to convert speech in language A into a first text in language A;

[0238] Je, when user A releases the virtual button on smart glasses A, or releases the screen button on mobile application A, mobile application A sends the first text in language A in text form to the translation server;

[0239] Jf, the translation server sends the first text in language A together with the translation prompt in text form to GAILLM (cloud or local);

[0240] Jg, GAILLMS translates the first text in the text format of language A into the second text in the text format of language B, and then sends it back to the translation server;

[0241] Jh, the translation server sends the second text in language B to the mobile application B in mobile phone B in text format;

[0242] Ji, mobile application B converts the second text in language B into speech in language B through a text-to-speech engine (in the cloud or on a local device);

[0243] Jj, mobile application B sends the speech in language B to smart glasses B in audio format;

[0244] Jk, smart glasses B play the speech in language B through the speaker so that user B can listen;

[0245] Jl, the transcripts for languages ​​A and B are both displayed on mobile apps A and B and can be shared and saved;

[0246] Jm, on the contrary, when user B wants to communicate with A, he only needs to repeat steps Jb to JI, and user B's spoken voice will be translated into language A's voice and played on user A's smart glasses A.

[0247] As an application example, take smart glasses and mobile phones as an example, in a one-to-many mode scenario K:

[0248] Ka, users A, B, C, and D wear smart glasses A, B, C, and D respectively, and use languages ​​A, B, C, and D respectively. Smart glasses A, B, C, and D are paired for translation mode. Smart glasses A, B, C, and D interact with mobile phones A, B, C, and D via Bluetooth. Mobile applications A, B, C, and D are installed in mobile phones A, B, C, and D respectively.

[0249] Kb, user A presses the right temple of smart glasses A and speaks. The microphone of smart glasses A picks up user A's speech A in language A and sends it to the translation server in text form through mobile application A. The translation server translates the text in language A into text in languages ​​B, C, and D and sends the translated text to the corresponding smart glasses B, C, and D respectively. Smart glasses B, C, and D each translate the received text into the corresponding speech through their speech-to-text engines and then play the converted speech to their respective users through their respective speakers.

[0250] Kc, when user B wants to communicate with user A, user B presses the right temple of smart glasses B to speak. The microphone of smart glasses B picks up user B's speech and converts the picked-up speech in language B into text through the speech engine. The smart glasses send the text in language B to the translation server, which translates the text in language B into text in language A and then sends the text in language A to smart glasses A. Smart glasses A convert the text in language A into speech in language A through the text-to-speech engine, and then play the speech in language A through the speakers of smart glasses A.

[0251] Kd, users C and D communicate with user A in different languages ​​through step Kc.

[0252] To further illustrate the above four translation modes, the following will be described with reference to FIG. 11 to FIG. 14 .

[0253] As shown in Figure 11, in monitoring mode, user A is not wearing smart glasses, while user B is wearing them. The smart glasses worn by user B establish a data connection via Bluetooth with a mobile app on user B's phone. User B can use the mobile app to set Language A for user A and Language B for user B, and switch the current translation mode to monitoring mode. In this monitoring mode, Language A used by user A is the first language (i.e., the source language), and Language B used by user B is the second language (i.e., the target language).

[0254] When user B presses the virtual button on the right temple of their smart glasses, user A begins speaking. The smart glasses pick up user A's voice through the microphone and send the picked-up voice in language A to the mobile app. When user B releases the virtual button, the mobile app uses GAILLM to translate the received voice into language B and sends it to the smart glasses, which then play the voice in language B for user B through the built-in speakers.

[0255] At the same time, the mobile APP can display the original text in language A and the translated text in language B corresponding to user A's voice on the mobile phone screen of user B.

[0256] As shown in Figure 12, in speaking mode, user A is not wearing smart glasses, while user B is wearing them. The smart glasses worn by user B establish a data connection via Bluetooth with a mobile app on user B's phone. User B can use the mobile app to set the language used by user A and user B, and switch the current translation mode to speaking mode. In speaking mode, user B's language B is the first language (i.e., the source language), and user A's language A is the second language (i.e., the target language).

[0257] When user B presses the virtual button on the right temple of their smart glasses and begins speaking, the smart glasses pick up user B's voice through the microphone and send it to the mobile app in language B. When user B releases the virtual button, the mobile app uses GAILLM to translate the received voice into language A and sends it to the smart glasses, which then play the language A voice for user A through the front speakers.

[0258] At the same time, the mobile APP can display the original text in language B corresponding to user B's voice and the translated text in language A on user B's mobile phone screen.

[0259] Furthermore, the listening mode shown in Figure 11 can be combined with the speaking mode shown in Figure 12. Specifically, user A does not wear smart glasses, while user B wears smart glasses. The smart glasses worn by user B establish a data connection with the mobile app on user B's phone via Bluetooth.

[0260] On the one hand, when user B presses the virtual button on the smart glasses he is wearing, the smart glasses enter the monitoring mode, and the smart glasses pick up the voice of user A through the microphone and send the picked-up voice of user A in language A to the mobile APP.

[0261] When user B releases the virtual button, the smart glasses use GAILLM via the mobile app to translate the speech in language A into language B, and play the speech in language B for user B through the built-in speaker. Simultaneously, the mobile app displays the original text in language A and the translated text in language B on user B's phone screen.

[0262] On the other hand, when the smart glasses detect that user B starts speaking using the voice activity detection algorithm and / or the IMU sensor on the smart glasses, the smart glasses switch the current translation mode from listening mode to speaking mode, pick up the voice of user B through the microphone, and send the picked-up voice of user B in language B to the mobile APP.

[0263] When the smart glasses detect that user B has stopped speaking using a voice activity detection algorithm and / or the IMU sensor on the smart glasses, the smart glasses use GAILLM via the mobile app to translate the speech in language B into speech in language A. The speech in language A is then played back to user A via the smart glasses' front speakers, external speakers, or the phone. Simultaneously, the mobile app displays the original text in language B and the translated text in language A on user B's phone screen.

[0264] As shown in Figure 13, in pairing mode, user A wears smart glasses A, and user B wears smart glasses B. User A's smart glasses A establish a data connection with the mobile app on user A's phone A via Bluetooth. User B's smart glasses B also establish a data connection with the mobile app on user B's phone B via Bluetooth. Smart glasses A and B are paired via phones A and B.

[0265] User A can use the mobile app on phone A to set the language used by user A and the language used by user B, and switch the current translation mode to pairing mode. In pairing mode, on phone A, language A used by user A is the first language (i.e., the source language), and language B used by user B is the second language (i.e., the target language).

[0266] User B can use the mobile app on phone B to set the language used by user B and the language used by user A, and switch the current translation mode to pairing mode. In pairing mode, on phone B, the language B used by user B is the first language (i.e., the source language), and the language A used by user A is the second language (i.e., the target language).

[0267] When user B presses and holds the virtual button on the right temple of their smart glasses B and begins speaking, the microphone in smart glasses B picks up user B's voice and sends the captured speech in language B to the mobile app on phone B. When user B releases the virtual button, the mobile app on phone B uses GAILLM to translate the speech in language B into text in language A and sends it to the mobile app on phone A. The mobile app on phone A then converts the text in language A into the corresponding speech in language A and sends the speech in language A to smart glasses A for playback.

[0268] When user A presses the virtual button on the right temple of their smart glasses A, smart glasses A and phone A translate the captured speech in language A into text in language B, similar to how user B does. The translated text is then sent to the mobile app on phone B. The mobile app on phone B then converts the text into the corresponding speech in language B and sends the speech to smart glasses B for playback.

[0269] Furthermore, the mobile app in mobile phone A can display the original text and translation processed by itself on the screen of mobile phone A. The mobile app in mobile phone B can display the original text and translation processed by itself on the screen of mobile phone B.

[0270] Furthermore, mobile phones A and B can also display the original text and translation processed by the other party on their respective screens in a sharing manner.

[0271] As shown in Figure 14, in a one-to-many tour guide mode, users A, B, C, and D wear smart glasses A, B, C, and D, respectively. User A is the tour guide, and the smart glasses A they wear act as the server (i.e., the master glasses). Users B, C, and D are tourists, and the smart glasses B, C, and D they wear act as the clients (i.e., the slave glasses). Smart glasses A, B, C, and D establish data connections with mobile phones A, B, C, and D, respectively.

[0272] When user A presses the virtual button on smart glasses A, smart glasses A pick up the voice of user A in language A, and then use the translation method in the above embodiment to translate the voice in language A into voice in language B, language C, and language D through data interaction with mobile phone A and send them to smart glasses B, C, and D respectively.

[0273] When user B presses the virtual button on smart glasses B, smart glasses B pick up the voice of user B in language B, and then use the translation method in the above embodiment to translate the voice in language B into voice in language A through data interaction with mobile phone B and send it to smart glasses A.

[0274] Furthermore, the terminal responsible for translation can also generate a translation record based on the original text and the translation, and distribute the translation record to each smart glass and / or each mobile phone for sharing.

[0275] For details not yet provided about the translation method in this embodiment, please refer to the relevant descriptions in the embodiments shown in Figures 1 to 9, which will not be repeated here.

[0276] In this embodiment, by utilizing a smart mobile terminal in conjunction with a generative AI large language model, intelligent translation is achieved for smart glasses. This improves the practicality, interactivity, and intelligence of the smart glasses, while also increasing product stickiness. Furthermore, due to the scalability and self-creativity of the generative AI large language model, translation accuracy can be further enhanced.

[0277] The present application also provides a non-transitory computer-readable storage medium, which may be provided in the smart glasses or smart wearable devices described in the above embodiments. The non-transitory computer-readable storage medium may be the memory 204 described in the embodiment shown in FIG8 . The computer-readable storage medium stores a computer program that, when executed by a processor, implements the artificial intelligence-based translation method described in the above embodiments. Furthermore, the computer-readable storage medium may be a USB flash drive, a mobile hard drive, a read-only memory (ROM), RAM, a magnetic disk, or an optical disk, among other media capable of storing program code.

[0278] In the several embodiments provided herein, it should be understood that the disclosed smart glasses, translation systems, and translation methods can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be omitted or not implemented. Furthermore, the connections or direct or communicative connections shown or discussed can be indirect or communicative connections through interfaces, devices, or modules, and can be electrical, mechanical, or other forms.

[0279] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0280] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0281] The above is a description of the smart glasses, translation system, and translation method provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A translation system based on artificial intelligence, characterized in that: include: First smart glasses, a first smart mobile terminal and a cloud server, wherein the cloud server is configured with a generative artificial intelligence large language model; The first smart glasses are used to pick up a first voice in a first language from a preset direction, and send the first voice to the first smart mobile terminal via Bluetooth; The first smart mobile terminal is used to convert the first speech into a first text using a speech-to-text engine, and send the first text to the cloud server, wherein the speech-to-text engine is arranged in the first smart mobile terminal or the cloud; The cloud server is used to translate the first text through the generative artificial intelligence large language model according to the translation prompt to obtain a second text in a second language, and send the second text to the first smart mobile terminal; The first smart mobile terminal is further used to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses via Bluetooth, wherein the text-to-speech engine is arranged in the first smart mobile terminal or in the cloud; The first smart glasses are further configured to play the second voice.

2. The translation system according to claim 1, wherein: The first intelligent mobile terminal has a mobile application installed thereon; The first smart mobile terminal is further configured to send a translation instruction to the first smart glasses in response to a preset operation of the user through the mobile application; The first smart glasses are further configured to enter a translation mode in response to the translation instruction, and pick up the first voice from the preset direction in the translation mode.

3. The translation system according to claim 1, wherein: The cloud server includes a translation server and a model server, a mobile application is installed on the first smart mobile terminal, and the generative artificial intelligence large language model is configured in the model server; The first smart mobile terminal is further configured to send a voice pickup instruction to the first smart glasses through the mobile application when a user presses a button on a screen of the first smart mobile terminal, wherein the button is generated by the mobile application; The first smart glasses are further configured to pick up the first voice from the preset direction in response to the voice pickup instruction; The first intelligent mobile terminal is further configured to send the first text to the translation server through the mobile application when the user releases the button; The translation server is used to generate the translation prompt and send the first text and the translation prompt to the model server; The model server is used to translate the first text to obtain the second text according to the translation prompt by using the generative artificial intelligence large language model, and send the second text to the translation server; The translation server is further configured to send the second text to the first smart mobile terminal.

4. The translation system according to claim 1, wherein: The cloud server includes a translation server and a model server, a mobile application is installed on the first smart mobile terminal, and the generative artificial intelligence large language model is configured in the model server; The first smart glasses are further configured to pick up the first voice from the preset direction when the user presses a virtual button on the temple of the first smart glasses; The first smart glasses are further configured to send notification information to the mobile application via Bluetooth when the user releases the virtual button; The first intelligent mobile terminal is further configured to send the first text to the translation server through the mobile application according to the notification information; The translation server is used to generate the translation prompt and send the first text and the translation prompt to the model server; The model server is used to translate the first text to obtain the second text according to the translation prompt by using the generative artificial intelligence large language model, and send the second text to the translation server; The translation server is further configured to send the second text to the first smart mobile terminal.

5. The translation system according to claim 1, wherein: The first smart mobile terminal has a mobile application installed thereon, and the first smart mobile terminal is further used for: generating a translation record according to the second text, or the first text and the second text, through the mobile application, and displaying the translation record on the screen of the first smart mobile terminal; The translation record is stored in the first smart mobile terminal or storage server through the mobile application.

6. The translation system according to claim 5, wherein: The first intelligent mobile terminal is further configured to share the translation record through the mobile application in a sharing manner indicated by the operation when detecting that the user operates a share button in the user interface of the mobile application.

7. The translation system according to claim 1, wherein: The first smart mobile terminal has a mobile application installed thereon, and the first smart mobile terminal is further used for: The second voice with a preset playing speed is generated through the mobile application, and the second voice with the preset playing speed is sent to the first smart glasses through the Bluetooth for playing.

8. The translation system according to claim 1, wherein: The first intelligent mobile terminal has a mobile application installed thereon. The first intelligent mobile terminal is further used to set the second language according to a setting operation performed by a user through the mobile application; The first smart glasses are further used to perform a voice pickup operation and send the picked-up voice to the first smart mobile terminal in real time in the form of a data stream via Bluetooth; The first intelligent mobile terminal is further used for: Using the speech-to-text engine to detect whether the language of the received speech is the first language; When the language is the first language, sending a translation start instruction to the first smart glasses via the Bluetooth, converting the received speech as the first speech into a first text via the speech-to-text engine, and sending the first text and the translation prompt to the model server, wherein the translation prompt includes information of the first language and the second language; as well as When silence of a preset duration is detected, a stop translation instruction is sent to the first smart glasses via the Bluetooth; The first smart glasses are further configured to enter a translation mode in response to the start translation instruction, and to exit the translation mode and end the voice pickup operation in response to the stop translation instruction.

9. The translation system according to claim 1, wherein: The cloud server includes a translation server and a model server, and the generative artificial intelligence large language model is configured in the model server; The first smart glasses are further used to send the first voice to the translation server via a wireless network; The translation server is used to convert the first speech into a first text using a speech-to-text engine, generate a translation prompt, and send the first text and the generated translation prompt to the model server; The model server is used to translate the first text to obtain the second text according to the translation prompt by using the generative artificial intelligence large language model, and send the second text to the translation server; The translation server is further configured to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses via the wireless network.

10. The translation system according to claim 1, wherein: The preset direction points to the front of the user of the first smart glasses or the mouth of the user of the first smart glasses.

11. The translation system according to claim 1, wherein: The preset direction points to the mouth of the user of the first smart glasses, and the translation system further includes a second pair of smart glasses and a second smart mobile terminal; The first smart mobile terminal is further configured to send the second voice to the second smart mobile terminal; The second smart mobile terminal is used to send the second voice to the second smart glasses; The second smart glasses are used to play the second voice.

12. The translation system according to claim 11, wherein: The first smart mobile terminal is further used to generate a translation record in real time according to the first text and the second text through a mobile application on the first smart mobile terminal, save the translation record in the first smart mobile terminal, display the translation record, and synchronize the translation record to the second smart mobile terminal; The second smart mobile terminal is further used to save the translation record in the second smart mobile terminal and display the translation record through a mobile application on the second smart mobile terminal.

13. The translation system according to claim 1, wherein: The cloud server includes a translation server, the translation system also includes a plurality of second smart glasses, and the generative artificial intelligence large language model is configured in the translation server; The first smart glasses are further configured to pick up a first voice of a user of the first smart mobile terminal and send the first voice to the first smart mobile terminal via Bluetooth when the user of the first smart glasses presses a button on the temple of the first smart glasses; The first smart mobile terminal is further used to convert the first speech into the first text by using the speech-to-text engine through a mobile application in the first smart mobile terminal, and send the first text to the translation server; The translation server is used to generate a first translation prompt, translate the first text according to the first translation prompt through the generative artificial intelligence large language model to obtain multiple third texts, and distribute each of the third texts to the corresponding second smart glasses, wherein the language of each of the third texts corresponds to the language used by the user of each of the second smart glasses; The second smart glasses are used to convert the received third text into a third voice using a text-to-speech engine, and play the third voice.

14. The translation system according to claim 13, wherein: The second smart glasses are further configured to pick up a fourth voice of the user of the second smart glasses when the user of the second smart glasses presses a virtual button on the temple of the second smart glasses, convert the fourth voice into a fourth text through a speech-to-text engine, and send the fourth text to the translation server; The translation server is configured to generate a second translation prompt, translate the fourth text according to the second translation prompt through the generative artificial intelligence large language model to obtain a fifth text, and send the fifth text to the first smart glasses, wherein the language of the fifth text corresponds to the language used by the user of the first smart glasses; The first smart glasses are further configured to utilize a text-to-speech engine to convert the fifth text into a fifth voice and play the fifth voice.

15. The translation system according to claim 1, wherein: The cloud server includes a model server, and the generative artificial intelligence large language model is configured in the model server.

16. The translation system according to claim 1, wherein: The first intelligent mobile terminal is further used to generate the translation prompt and send the translation prompt to the cloud server.

17. A smart glasses for translation based on artificial intelligence, characterized in that: include: A sound pickup device, an output device, a processor and a memory, wherein the processor is electrically connected to the sound pickup device, the output device and the memory; The memory stores one or more programs executable by the processor, wherein the one or more programs include multiple instructions, and the multiple instructions are used to: Picking up a first speech of a first language from a preset direction by the sound pickup device; The first speech is translated into a second speech in a second language by using a generative artificial intelligence large language model locally or in the cloud; as well as The second speech is outputted through the output device.

18. The smart glasses according to claim 17, wherein: The preset direction points to the front of the user.

19. The smart glasses according to claim 17, wherein: The smart glasses are installed with a client program, and the plurality of instructions are further used for: The first language and / or the second language is set according to a first preset operation performed by the user in the client program.

20. The smart glasses according to claim 19, wherein: The multiple instructions are also used to set the playback speed of the second voice according to a second preset operation performed by the user in the client program.

21. The smart glasses according to claim 17, wherein: The smart glasses further include a Bluetooth component electrically connected to the processor, and the plurality of instructions are further used to: The translation instruction sent by the mobile application on the smart mobile terminal is received through the Bluetooth component, a translation mode is entered in response to the translation instruction, and in the translation mode, the first voice from the preset direction is picked up by the sound pickup device.

22. The smart glasses according to claim 21, wherein: The smart glasses further include: temples and virtual buttons based on touch sensors disposed on the temples and electrically connected to the processor, and the plurality of instructions are further used to: When the user presses the virtual button, controlling the sound pickup device to pick up the first voice; When the user releases the virtual button, the first voice is sent to the smart mobile terminal through the Bluetooth component, so that the smart mobile terminal translates the first voice into the second voice by using a generative artificial intelligence language model set on the smart mobile terminal or the cloud server; and The second voice sent by the smart mobile terminal is received through the Bluetooth component.

23. The smart glasses according to claim 17, wherein: The plurality of instructions are also used to: Performing language detection on the first speech to determine whether the first language is a preset language by a local or cloud-based speech-to-text engine; When the first language is the preset language, entering a translation mode, and converting the first speech into a first text through the speech-to-text engine; Generate a translation prompt, translate the first text through the generative artificial intelligence large language model to obtain a second text in the second language, wherein the translation prompt includes information of the first language and the second language; Converting the second text into the second voice through a local or cloud-based voice-to-text engine; as well as When silence of a preset duration is detected by the sound pickup device, the translation mode is exited.

24. The smart glasses according to claim 17, wherein: The smart glasses further include a wireless communication component electrically connected to the processor, and the plurality of instructions are further used to: Sending the first speech to a translation server through the wireless communication component, so that the translation server translates the first speech into the second speech by using a generative artificial intelligence large language model set on the translation server or the model server; as well as The second voice sent by the translation server is received through the wireless communication component.

25. The smart glasses according to claim 17, wherein: The output device includes a front speaker, the preset direction points to the user's mouth, and the multiple instructions are also used to play the second voice through the front speaker.

26. The smart glasses according to claim 17, wherein: The smart glasses also include a Bluetooth component electrically connected to the processor, the preset direction points to the user's mouth, and the multiple instructions are also used to send the second voice to an external player through the Bluetooth component for playback.

27. The smart glasses according to claim 17, wherein: The smart glasses further include: an inertial sensor electrically connected to the processor, and the plurality of instructions are further used to: detecting whether the first speech has ended using a voice activity detection algorithm and / or sensing data from the inertial sensor; and When it is detected that the first speech ends, the first speech is translated by the generative artificial intelligence large language model to obtain the second speech.

28. A translation method based on artificial intelligence, characterized in that: Applied to a smart mobile terminal, the method comprises: Receiving a first voice in a first language sent by the smart wearable device, converting the first voice into a first text using a local or cloud-based voice-to-text engine, and obtaining a translation prompt; Translating the first text to obtain a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence large language model; The second text is converted into a second voice using a local or cloud-based text-to-speech engine, and the second voice is sent to the smart wearable device for playback.

29. The translation method according to claim 28, characterized in that The smart mobile terminal has a mobile application installed on it, and the method further includes: The mobile application sends a translation instruction to the smart wearable device in response to a preset operation of the user, so that the smart wearable device enters a translation mode in response to the translation instruction and picks up the first voice in the translation mode.

30. The translation method according to claim 28, wherein: The smart mobile terminal has a mobile application installed thereon. Before receiving the first voice in the first language sent by the smart wearable device, the method further includes: When a user presses a button on the screen of the smart mobile terminal, a voice pickup instruction is sent to the smart wearable device through the mobile application to instruct the smart wearable device to pick up the first voice, wherein the button is generated by the mobile application; The generative artificial intelligence big language model is configured on the model server, and the first text is translated according to the translation prompt by the local or cloud-based generative artificial intelligence big language model to obtain a second text in a second language, including: When the user releases the button, or receives a notification message sent by the smart wearable device, the first text is sent to the translation server through the mobile application, so that the translation server generates the translation prompt and sends the first text and the translation prompt to the model server, wherein the notification message is sent by the smart wearable device when the user of the smart wearable device releases the virtual button on the temple of the smart wearable device; The second text forwarded by the translation server is received, wherein the second text is sent by the model server to the translation server, and the model server translates the first text by utilizing the generative artificial intelligence large language model according to the translation prompt.

31. The translation method according to claim 28, wherein: The smart mobile terminal has a mobile application installed on it, and the method further includes: generating a translation record according to the second text, or the first text and the second text, through the mobile application, and displaying the translation record on the screen of the smart mobile terminal; The translation record is stored in the smart mobile terminal or storage server through the mobile application.

32. The translation method according to claim 31, characterized in that The method further comprises: When it is detected that the user operates a share button in the user interface of the mobile application, the translation record is shared through the mobile application in a sharing manner indicated by the operation.

33. The translation method according to claim 28, wherein: The smart mobile terminal has a mobile application installed thereon, and before sending the second voice to the smart wearable device for playing, the method further includes: Through the mobile application, the playback speed of the second voice is adjusted to a preset playback speed.

34. The translation method according to claim 28, wherein: The smart mobile terminal is installed with a mobile application, the generative artificial intelligence large language model is configured on the model server, the smart wearable device performs a voice pickup operation, and sends the picked-up voice to the smart mobile terminal in real time via Bluetooth in the form of a data stream, and the method further includes: Setting the second language according to a setting operation performed by the user through the mobile application; Using the speech-to-text engine to detect whether the language of the received speech is the first language; When the language is the first language, sending a translation start instruction to the smart wearable device through the Bluetooth to instruct the smart wearable device to enter a translation mode, and using the received voice as the first voice to perform the operations of converting the first voice into a first text using a local or cloud-based voice-to-text engine, and generating a translation prompt; translating the first text into a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence large language model, wherein the translation prompt includes information of the first language and the second language; and When silence of a preset duration is detected, a stop translation instruction is sent to the smart wearable device via the Bluetooth to instruct the smart wearable device to exit the translation mode and end the voice pickup operation.

35. The translation method according to claim 28, wherein: The first voice is a voice from a user of the smart wearable device, and the method further includes: The second voice is sent to the target terminal, so that the target terminal sends the second voice to the target smart wearable device for playback.

36. The translation method according to claim 28, wherein: The smart mobile terminal has a mobile application installed in it, and the method further includes: When the smart wearable device as a master smart wearable device completes pairing with a plurality of slave smart wearable devices, a translation prompt is generated through the mobile application; By means of the generative artificial intelligence large language model, according to the translation prompt generated by the smart wearable device, the first text is translated to obtain the plurality of third texts, wherein the language of each of the third texts corresponds to the language used by the user of each subordinate smart wearable device; Distribute each of the third texts to the corresponding slave smart wearable device so that the corresponding slave smart wearable device uses a text-to-speech engine to convert the received third text into a third voice and play the third voice.

Images