Translation system based on artificial intelligence, intelligent glasses for translation and translation method

Abstract

The invention discloses a translation system based on artificial intelligence, intelligent glasses for translation and a translation method. The translation system comprises first intelligent glasses, a first intelligent mobile terminal and a cloud server, and a generative artificial intelligence large language model is configured in the cloud server. The first intelligent glasses pick up a first voice of a first language from a preset direction and send the first voice to the first intelligent mobile terminal. The first intelligent mobile terminal converts the first voice into a first text and sends the first text to the cloud server; and the cloud server translates the first text through the model according to the translation prompt to obtain a second text of a second language and sends the second text to the first intelligent mobile terminal. And the first intelligent mobile terminal converts the second text into a second voice and sends the second voice to the first intelligent glasses for playing. According to the invention, high-precision intelligent translation based on the intelligent glasses is realized, and the product viscosity can be increased.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of smart glasses technology, and in particular to an artificial intelligence-based translation system, smart glasses for translation, and a translation method. Background Art

[0002] With the development of computer technology, smart glasses are becoming more and more popular. However, existing smart glasses are expensive, and in addition to their own functions as smart glasses, they usually only have the functions of listening to music and making or receiving calls. The functions are relatively simple and the product viscosity is low. Summary of the invention

[0003] The embodiments of the present application provide an artificial intelligence-based translation system, smart glasses for translation, and a translation method, which are used to implement a high-precision translation function based on smart glasses, thereby improving the practicality, interactivity, and intelligence of the smart glasses, and increasing product stickiness.

[0004] On the one hand, an embodiment of the present application provides an artificial intelligence-based translation system, comprising: a first pair of smart glasses, a first smart mobile terminal, and a cloud server, wherein the cloud server is configured with a generative artificial intelligence large language model;

[0005] The first smart glasses are used to pick up a first voice in a first language from a preset direction, and send the first voice to the first smart mobile terminal via Bluetooth;

[0006] The first smart mobile terminal is used to convert the first speech into a first text using a speech-to-text engine, and send the first text to the cloud server, wherein the speech-to-text engine is arranged in the first smart mobile terminal or the cloud;

[0007] The cloud server is used to translate the first text through the generative artificial intelligence large language model according to the translation prompt to obtain a second text in a second language, and send the second text to the first smart mobile terminal;

[0008] The first smart mobile terminal is further used to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses via Bluetooth, wherein the text-to-speech engine is arranged in the first smart mobile terminal or in the cloud;

[0009] The first smart glasses are further configured to play the second voice.

[0010] On one hand, an embodiment of the present application further provides a smart glasses for translation based on artificial intelligence, comprising: a sound pickup device, an output device, a processor and a memory, wherein the processor is electrically connected to the sound pickup device, the output device and the memory;

[0011] The memory stores one or more programs executable by the processor, wherein the one or more programs include multiple instructions, and the multiple instructions are used to:

[0012] Picking up a first speech of a first language from a preset direction by the sound pickup device;

[0013] Translating the first speech into a second speech in a second language by using a generative artificial intelligence large language model locally or in the cloud; and

[0014] The second speech is outputted through the output device.

[0015] On the one hand, an embodiment of the present application further provides a translation method based on artificial intelligence, which is applied to a smart mobile terminal, and the method includes:

[0016] Receiving a first voice in a first language sent by the smart wearable device, converting the first voice into a first text using a local or cloud-based voice-to-text engine, and obtaining a translation prompt;

[0017] Translating the first text to obtain a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence large language model;

[0018] The second text is converted into a second voice using a local or cloud-based text-to-speech engine, and the second voice is sent to the smart wearable device for playback.

[0019] The embodiments of the present application realize intelligent translation based on smart glasses by combining smart glasses, smart mobile terminals and servers, thereby improving the practicality, interactivity and intelligence of smart glasses and increasing product stickiness. In addition, due to the scalability and self-creativity of the generative artificial intelligence large language model, the accuracy of translation can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1A schematic diagram of the structure of an artificial intelligence-based translation system provided in one embodiment of the present application;

[0022] Figure 2 for Figure 1 The application scenario diagram of the translation system shown;

[0023] Figure 3 A schematic diagram of the structure of an artificial intelligence-based translation system provided in another embodiment of the present application;

[0024] Figure 4 Based on Figure 1 and Figure 3 A schematic diagram of a first system framework of the illustrated translation system;

[0025] Figure 5 Based on Figure 1 and Figure 3 A schematic diagram of a second system framework of the illustrated translation system;

[0026] Figure 6 Based on Figure 1 and Figure 3 A schematic diagram of a third system framework of the illustrated translation system;

[0027] Figure 7 Based on Figure 1 and Figure 3 A schematic diagram of a fourth system framework of the illustrated translation system;

[0028] Figure 8 A schematic diagram of the internal structure of the artificial intelligence-based translation smart glasses provided in one embodiment of the present application;

[0029] Fig. 9 for Figure 8 A schematic diagram of the external structure of the smart glasses in the illustrated embodiment;

[0030] Fig.10 A flowchart of an artificial intelligence-based translation method provided in one embodiment of the present application;

[0031] Fig.11 for Figure 1 The translation system shown and Fig.10 A schematic diagram of implementing a first translation mode in the translation method shown;

[0032] Fig.12 for Figure 1 The translation system shown and Fig.10 A schematic diagram of implementing the second translation mode in the translation method shown;

[0033] Fig.13 for Figure 1 The translation system shown and Fig.10A schematic diagram of implementing the third translation mode in the translation method shown;

[0034] Fig.14 for Figure 1 The translation system shown and Fig.10 A schematic diagram of implementing the fourth translation mode in the translation method shown. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0037] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0038] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0039] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of an artificial intelligence-based translation system provided in one embodiment of the present application. Figure 1 As shown, the translation system 100 includes: a first pair of smart glasses 110 , a first smart mobile terminal 120 and a cloud server 130 .

[0040] In each embodiment of the present application, a corresponding application (APP) is installed in each smart glass or each smart mobile terminal. The smart glasses and the smart mobile terminal establish a data connection via Bluetooth and use the APP to interact with data.

[0041] The smart glasses may be open smart glasses, and their specific structure may refer to the following: Figure 8 and Fig. 9 The relevant description in the embodiment of the smart glasses shown is not repeated here.

[0042] The intelligent mobile terminal may include, but is not limited to: a cellular phone, a smart phone, other wireless communication devices, a personal digital assistant, an audio player, other media players, a music recorder, a video recorder, a camera, other media recorders, a smart radio, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a moving picture expert group (MPEG-1 or MPEG-2) audio layer 3 (MP3) player, a digital camera, and an intelligent wearable device (such as a smart watch, a smart bracelet, etc.). The intelligent mobile terminal is also installed with an Android, iOS or other operating system.

[0043] The cloud server 130 may be a single server or a distributed server cluster consisting of multiple servers, on which a Generative Artificial Intelligence Large Language Model (GAILLM) is configured.

[0044] The GAILLM can be, for example but not limited to: ChatGPT of Open AI, Bard of Google, and other models with similar functions. The GAILLM integrates multiple functions, such as obtaining the task execution command corresponding to the voice command issued by the user in natural language (such as Figure 2 As shown, GAILLM can send messages, send emails, make phone calls, search the Internet, call network services, call other third-party software development tools (Software Development Kit, SDK) to perform tasks provided by the SDK), translate, etc. Among them, the part of GAILLM that obtains task execution commands is trained with a large number of semantics and corresponding task execution commands as samples. The translation part of GAILLM is trained with a large number of words and / or sentences in the source language and the corresponding words and / or sentences in the target language as samples. Among them, one source language can correspond to multiple different target languages.

[0045] Specifically, the first smart glasses 110 are used to pick up the first voice of the first language (i.e., the source language) from the preset direction, and send the first voice to the first smart mobile terminal 120 via Bluetooth. Optionally, the preset direction can point to the front of the user of the first smart glasses 110, i.e., the listening mode, in which the language used by the chat partner of the user of the first smart glasses 110 is the source language. Alternatively, the preset direction can also point to the mouth of the user of the first smart glasses 110, i.e., the speaking mode, in which the language used by the user of the first smart glasses 110 is the source language. The first smart glasses 110 can switch the current translation mode to the corresponding mode according to the voice command of the user, or the operation of the user pressing the control button for switching the translation mode on the first smart glasses 110, or the operation of the user in the user interface (UI) of the first smart mobile terminal 120. And according to the current translation mode, the source language and the target language are determined.

[0046] The first smart mobile terminal 120 is used to convert the first speech into a first text using a speech-to-text engine, and send the first text to the cloud server 130, wherein the speech-to-text engine is set in the first smart mobile terminal 120 or the cloud (e.g., a speech-to-text server, a model server, a translation server or other cloud servers).

[0047] The cloud server 130 is configured to translate the first text through the GAILLM according to the translation prompt to obtain a second text in a second language (ie, the target language), and send the second text to the first smart mobile terminal 120 .

[0048] The first smart mobile terminal 120 is also used to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses 110 via Bluetooth, wherein the text-to-speech engine is set in the first smart mobile terminal 120 or in the cloud (e.g., a text-to-speech server, a model server, a translation server or other cloud servers).

[0049] The first smart glasses 110 are further configured to play the second voice.

[0050] The above-mentioned translation prompt may include instruction information for instructing the GAILLM to perform the translation task. Further, the translation prompt may also include: identification information of the first language and / or the second language. The translation prompt may be specifically generated by a prompt generator. The prompt generator may be a software module or a microcontroller configured with the software module, which is used to generate corresponding prompt information according to the semantics in the parsed user voice, and the semantics is obtained by parsing each user voice. Alternatively, the prompt generator may also generate a translation prompt according to other preset rules, for example, receiving a translation instruction, receiving a text sent by a specific object, and receiving data containing language identification information, etc.

[0051] The prompt generator can be configured in the smart glasses 110, the first smart mobile terminal 120 or any server in the cloud.

[0052] Optionally, the prompt generator may obtain identification information of the first language and / or the second language from the smart glasses 110 or the first smart mobile terminal 120 .

[0053] Optionally, in other embodiments of the present application, the first smart mobile terminal 120 is further used to generate the translation prompt and send the translation prompt to the cloud server.

[0054] Optionally, in other embodiments of the present application, a mobile application (mobile APP) is installed on the first smart mobile terminal 120 .

[0055] The first smart mobile terminal 120 is further used to send a translation instruction to the first smart glasses 110 in response to a preset operation of the user through the mobile application. The preset operation may be, for example but not limited to, any of the following operations: an operation in which the user presses a preset virtual button for starting translation on the UI interface of the mobile application, an operation in which the user performs a preset sliding gesture on the screen of the first smart glasses 110 or the first smart mobile terminal 120, an operation in which the user presses a translation control button set on the temple of the first smart glasses 110, and an operation in which the user issues a preset translation voice command, etc.

[0056] The first smart glasses 110 are further configured to enter a translation mode in response to the translation instruction, and pick up the first voice from the preset direction in the translation mode.

[0057] Optionally, in other embodiments of the present application, such as Figure 3 As shown, a mobile application is installed on the first smart mobile terminal 120 , and the cloud server 130 includes a translation server 131 and a model server 132 (or, also referred to as a GAILLM server), and the GAILLM is configured in the model server 132 .

[0058] The first smart mobile terminal 120 is also used to send a voice pickup instruction to the first smart glasses 110 through the mobile application when the user presses a button on the screen of the first smart mobile terminal 120, wherein the button is generated by the mobile application.

[0059] The first smart glasses 110 are further configured to pick up the first voice from the preset direction in response to the voice pickup instruction.

[0060] The first smart mobile terminal 120 is further configured to send the first text to the translation server 131 via the mobile application when the user releases the button.

[0061] The translation server 131 is used to generate the translation prompt and send the first text and the translation prompt to the model server 132.

[0062] The model server 132 is used to translate the first text through the GAILLM according to the translation prompt to obtain the second text, and send the second text to the translation server 131 .

[0063] The translation server 131 is further configured to send the second text to the first smart mobile terminal 120 .

[0064] Optionally, in other embodiments of the present application, such as Figure 3 As shown, the cloud server 130 includes a translation server 131 and a model server 132 , a mobile application is installed on the first smart mobile terminal 120 , and the GAILLM is configured in the model server 132 .

[0065] The first smart glasses 110 are further configured to pick up the first voice from the preset direction when the user presses a virtual button on the temple of the first smart glasses 110 .

[0066] The first smart glasses 110 are further configured to send notification information to the mobile application via Bluetooth when the user releases the virtual button.

[0067] The first smart mobile terminal 120 is further configured to send the first text to the translation server 131 through the mobile application according to the notification information.

[0068] The translation server 131 is used to generate the translation prompt and send the first text and the translation prompt to the model server 132.

[0069] The model server 132 is used to translate the first text through the GAILLM according to the translation prompt to obtain the second text, and send the second text to the translation server 131 .

[0070] The translation server 131 is further configured to send the second text to the first smart mobile terminal 120 .

[0071] Optionally, in other embodiments of the present application, a mobile application is installed on the first smart mobile terminal 120, and the first smart mobile terminal 120 is also used to: generate a translation record based on the second text, or the first text and the second text, through the mobile application, and display the translation record on the screen of the first smart mobile terminal 120; and save the translation record in the first smart mobile terminal 120 or the storage server through the mobile application.

[0072] Optionally, in other embodiments of the present application, the first smart mobile terminal 120 is also used to share the translation record through the mobile application in a sharing manner indicated by the operation when an operation of a user on a sharing button in the user interface of the mobile application is detected.

[0073] The above-mentioned sharing may include, for example but not limited to: sharing the translation record with friends of the first smart glasses 110 registered in instant messaging software such as WeChat, or sharing the translation record with other terminal devices via email, or sharing the translation record on social networking platforms such as Facebook, Instagram, etc. The user can view the translation record in the form of a full translation or a comparison between the original text and the translation on the sharing terminal. The sharing may be part or all of the translation records generated within one or more translation cycles specified by the sharing user. A process from the smart glasses entering the translation mode to exiting the translation mode can be regarded as a translation cycle.

[0074] Optionally, in other embodiments of the present application, a mobile application is installed on the first smart mobile terminal 120, and the first smart mobile terminal 120 is also used to: generate the second voice with a preset playback speed through the mobile application, and send the second voice with the preset playback speed to the first smart glasses 110 via the Bluetooth for playback.

[0075] Optionally, in other embodiments of the present application, a mobile application is installed on the first smart mobile terminal 120 .

[0076] The first smart mobile terminal 120 is further configured to set the second language according to a setting operation performed by the user through the mobile application.

[0077] The first smart glasses 110 are also used to perform a voice pickup operation and send the picked-up voice to the first smart mobile terminal 120 in real time in the form of a data stream via Bluetooth.

[0078] The first smart mobile terminal 120 is also used to: use the speech-to-text engine to detect whether the language of the received speech is the first language; when the language is the first language, send a start translation instruction to the first smart glasses 110 via the Bluetooth, and convert the received speech as the first speech into a first text via the speech-to-text engine, and send the first text and the translation prompt to the model server 132, wherein the translation prompt includes information of the first language and the second language; and when silence of a preset length is detected, send a stop translation instruction to the first smart glasses 110 via the Bluetooth.

[0079] The first smart glasses 110 are further configured to enter a translation mode in response to the start translation instruction, and to exit the translation mode and terminate the voice pickup operation in response to the stop translation instruction.

[0080] Specifically, after the smart glasses are turned on or according to the monitoring instructions triggered by the user, they perform a voice pickup operation to monitor the user's voice in real time, and send the real-time monitored voice to the smart mobile terminal in the form of a data stream. The voice is cached in the memory of the smart glasses. During the monitoring process, the smart glasses use multiple threads to continuously cover the old voice with the newly picked up voice of the preset duration in the memory 203 (for example, the currently picked up 3 seconds of voice covers the previously picked up 3 seconds of voice), and at the same time send the cached newly picked up voice to the smart mobile terminal. The smart mobile terminal stores the received voice in the local memory and determines whether the language of the voice is the first language. On the one hand, when the language is not the first language, the smart mobile terminal deletes the received voice from the local memory, and continues to perform the language judgment operation based on the newly received voice, until the preset duration is exceeded and no new voice sent by the smart glasses is received, or a notification message of stopping monitoring sent by the smart glasses is received. On the other hand, when the language is the first language, the smart mobile terminal sends a start translation instruction to the smart glasses and performs a translation operation on the received voice. The smart glasses enter the translation mode according to the start translation instruction and continue to perform the voice pickup operation. Further, when a preset length of silence is detected in the voice stored in the local memory, the smart mobile terminal sends a stop translation instruction to the smart glasses, so that the smart glasses exit the translation mode and end the voice pickup operation according to the stop translation instruction.

[0081] Optionally, in other embodiments of the present application, such as Figure 3 As shown, the cloud server 130 includes a translation server 131 and a model server 132 , and the GAILLM is configured in the model server 132 .

[0082] The first smart glasses 110 are further configured to send the first speech to the translation server 131 via a wireless network.

[0083] The translation server 131 is used to convert the first speech into a first text using a speech-to-text engine, generate a translation prompt, and send the first text and the generated translation prompt to the model server 132.

[0084] The model server 132 is used to translate the first text through the GAILLM according to the translation prompt to obtain the second text, and send the second text to the translation server 131 .

[0085] The translation server 131 is further configured to convert the second text into a second speech by using a text-to-speech engine, and send the second speech to the first smart glasses 110 via the wireless network.

[0086] Optionally, in other embodiments of the present application, the preset direction points to the mouth of the user of the first smart glasses 110, such as Figure 3 As shown, the translation system 100 further includes a second pair of smart glasses 140 and a second smart mobile terminal 150 .

[0087] The first smart mobile terminal 120 is further configured to send the second voice to the second smart mobile terminal 150 .

[0088] The second smart mobile terminal 150 is used to send the second voice to the second smart glasses 140.

[0089] The second smart glasses 140 are used to play the second voice.

[0090] Optionally, in other embodiments of the present application, the first smart mobile terminal 120 is also used to generate a translation record in real time based on the first text and the second text through a mobile application on the first smart mobile terminal 120, save the translation record in the first smart mobile terminal 120, display the translation record, and synchronize the translation record to the second smart mobile terminal 150.

[0091] The second smart mobile terminal 150 is further configured to store the translation record in the second smart mobile terminal 150 and display the translation record through a mobile application on the second smart mobile terminal 150 .

[0092] Specifically, the mobile application on the second smart mobile terminal 150 can display the translation record in a list on the screen of the second smart mobile terminal 150. The user of the second smart mobile terminal 150 can view different parts of the translation record by performing a preset flipping gesture on the screen, or using a physical or virtual button in the second smart mobile terminal 150, or through voice control instructions.

[0093] Optionally, in other embodiments of the present application, such as Figure 3 As shown, the cloud server 130 includes a translation server 131 , and the translation system 100 also includes a plurality of second smart glasses, and the GAILLM is configured in the translation server 131 .

[0094] The first smart glasses 110 are also used to pick up the first voice of the user of the first smart mobile terminal 120 when the user of the first smart glasses 110 presses the button on the temple of the first smart glasses 110, and send the first voice to the first smart mobile terminal 120 via Bluetooth.

[0095] The first smart mobile terminal 120 is further used to convert the first speech into the first text by using the speech-to-text engine through the mobile application in the first smart mobile terminal 120 , and send the first text to the translation server 131 .

[0096] The translation server 131 is used to generate a first translation prompt, and through the GAILLM, according to the first translation prompt, translate the first text to obtain multiple third texts, and distribute each of the third texts to the corresponding second smart glasses, wherein the language of each third text corresponds to the language used by the user of each second smart glasses 140.

[0097] The second smart glasses 140 are used to convert the received third text into a third voice by using a text-to-speech engine, and play the third voice.

[0098] Optionally, in other embodiments of the present application, the second smart glasses 140 are also used to pick up a fourth voice of the user of the second smart glasses 140 when the user of the second smart glasses 140 presses and holds a virtual button on the temple of the second smart glasses 140, convert the fourth voice into a fourth text through a speech-to-text engine, and send the fourth text to the translation server 131.

[0099] The translation server 131 is used to generate a second translation prompt, and through the GAILLM, according to the second translation prompt, translate the fourth text to obtain a fifth text, and send the fifth text to the first smart glasses 110, wherein the language of the fifth text corresponds to the language used by the user of the first smart glasses 110.

[0100] The first smart glasses 110 are further configured to utilize a text-to-speech engine to convert the fifth text into a fifth voice and play the fifth voice.

[0101] Optionally, in other embodiments of the present application, such as Figure 3 As shown, the cloud server 130 includes a model server 132 , and the GAILLM is configured in the model server 132 .

[0102] See also Figure 4 , Figure 4 Based on Figure 1 and Figure 3 Schematic diagram of the first system framework of the translation system shown in FIG. Figure 4 As shown, the translation system includes: smart glasses 401, smart mobile terminal 402, text-to-speech server 403, speech-to-text server 404, translation server 405 and model server 406. The text-to-speech engine is configured in the text-to-speech server 403, the speech-to-text engine is configured in the speech-to-text server 404, the prompt generator is configured in the translation server 405, and GAILLM is configured in the model server 406.

[0103] exist Figure 4 In steps 1 to 6 of the present invention, when the user presses the virtual button on the smart glasses 401 worn by the user, the smart glasses 401 pick up the local voice of the user (i.e., the first voice) and send the local voice in an audio format to the mobile application (mobile APP) on the smart mobile terminal 402; the mobile The APP converts the local speech into local text (i.e., the first text) by interacting with the speech-to-text server 404, and sends the local text returned by the speech-to-text server 404 to the translation server 405; the translation server 405 generates a translation prompt, and sends the translation prompt and the local text to the model server 406, so that the model server 406 performs a natural language translation operation on the local text using GAILLM according to the translation prompt and obtains a translated text (i.e., the second text); the translation server 405 receives the translated text returned by the model server 406, and forwards the translated text to the smart mobile terminal 402; the smart mobile terminal 402 converts the translated text into corresponding speech by interacting with the text-to-speech server 403, and sends the corresponding speech returned by the text-to-speech server 403 to the smart glasses 401 for playback, and displays the translated text on the screen of the smart mobile terminal 402.

[0104] See also Figure 5 , Figure 5 Based on Figure 1 and Figure 3 Schematic diagram of the second system framework of the translation system shown in FIG. Figure 5 As shown, the translation system includes: smart glasses 401, a smart mobile terminal 402 and a model server 406. The text-to-speech engine, the speech-to-text engine and the prompt generator are configured in the smart mobile terminal 402, and GAILLM is configured in the model server 406.

[0105] exist Figure 5In steps 1 to 5, when the user presses a virtual button on the smart glasses 401 worn by the user, the smart glasses 401 pick up the local voice of the user (i.e., the first voice) and send the local voice in audio format to the mobile APP on the smart mobile terminal 402; the mobile APP converts the local voice into local text (i.e., the first text) through the voice-to-text engine, generates a translation prompt through the prompt generator, and sends the local text and the translation prompt to the model server 406, so that the model server 406 performs a natural language translation operation on the local text using GAILLM according to the translation prompt and obtains a translated text (i.e., the second text); the mobile APP receives the translated text returned by the model server 406, converts the translated text into corresponding voice through the text-to-speech engine and sends it to the smart glasses 401 for playback, and displays the translated text on the screen of the smart mobile terminal 402.

[0106] It can be understood that the above-mentioned text-to-speech engine and speech-to-text engine can also be set on one or more cloud servers, and the mobile APP performs text-to-speech and speech-to-text operations by interacting with the one or more servers.

[0107] See also Figure 6 , Figure 6 Based on Figure 1 and Figure 3 The schematic diagram of the third system framework of the translation system shown in FIG. Figure 6 As shown, the translation system includes: smart glasses 401 and a model server 406. GAILLM is configured in the model server 406.

[0108] exist Figure 6 In steps 1 to 6, when the user presses a virtual button on the smart glasses 401 being worn, the smart glasses 401 pick up the local voice (i.e., the first voice) of the user through the built-in client APP, convert the local voice into local text (i.e., the first text) using a local or cloud-based voice-to-text engine, generate a translation prompt using a local or cloud-based prompt generator, and send the local text and the translation prompt to the model server 406, so that the model server 406 performs a natural language translation operation on the local text using GAILLM according to the translation prompt and obtains a translated text (i.e., the second text); the smart glasses 401 receive the translated text returned by the model server 406 through the client APP, convert the translated text into corresponding voice using a local or cloud-based text-to-speech engine and play it, and display the translated text on the screen of the smart glasses 401.

[0109] Furthermore, the client APP can also generate a translation record based on the local text and the translated text, and allow the user to view the translation record on the screen of the smart glasses 401 through a preset operation. Furthermore, the client APP can also share the translation record with other terminals or network social platforms specified by the user according to the user's sharing voice command.

[0110] It can be understood that the above-mentioned text-to-speech engine, speech-to-text engine and prompt generator can be configured on one or more cloud servers, and the client APP performs operations such as text-to-speech, speech-to-text and generating translation prompts by interacting with the one or more servers.

[0111] See also Figure 7 , Figure 7 Based on Figure 1 and Figure 3 Schematic diagram of the fourth system framework of the translation system shown in FIG. Figure 7 As shown, the translation system includes: smart glasses 401 and a smart mobile terminal 402. A text-to-speech engine, a speech-to-text engine, a prompt generator and GAILLM are configured in the smart mobile terminal 402.

[0112] exist Figure 7 In steps 1 to 6, when the user presses a virtual button on the smart glasses 401 worn by the user, the smart glasses 401 pick up the local voice of the user (i.e., the first voice) and send the local voice in an audio format to the mobile APP on the smart mobile terminal 402; the mobile APP converts the local voice into local text (i.e., the first text) through the voice-to-text engine, generates a translation prompt through the prompt generator, and inputs the translation prompt and the local text into the GAILLM, so that the GAILLM performs a natural language translation operation on the local text according to the translation prompt and outputs a translated text (i.e., the second text); the mobile APP converts the translated text into a corresponding voice through the text-to-speech engine and sends it to the smart glasses 401 for playback, and displays the translated text on the screen of the smart mobile terminal 402.

[0113] Furthermore, if the mobile APP confirms that the smart glasses 401 have a screen according to the device information record, the translated text is sent to the smart glasses 401 for display on the screen of the smart glasses 401 .

[0114] Furthermore, the mobile APP can also generate a translation record based on the local text and the translated text, and allow the user to view the translation record on the screen of the smart glasses 401 or the screen of the smart mobile terminal 402 through a preset operation. Furthermore, the client APP can also share the translation record with other terminals or network social platforms specified by the user according to the user's sharing voice command or the user's preset sharing operation on the UI interface of the mobile APP.

[0115] It can be understood that the above-mentioned text-to-speech engine, speech-to-text engine and prompt generator can also be configured on one or more cloud servers. The mobile APP performs operations such as text-to-speech, speech-to-text and generating translation prompts by interacting with the one or more servers.

[0116] In this embodiment, by combining smart glasses, smart mobile terminals and servers, smart translation based on smart glasses is realized, thereby improving the practicality, interactivity and intelligence of smart glasses and increasing product stickiness. In addition, due to the scalability and self-creativity of the generative artificial intelligence large language model, the accuracy of translation can be further improved.

[0117] See also Figure 8 and Fig. 9 , Figure 8 This is a schematic diagram of the internal structure of the artificial intelligence-based translation smart glasses provided in one embodiment of the present application. Fig. 9 for Figure 8 The external structure diagram of the smart glasses in the embodiment shown is shown. For the convenience of explanation, only the parts related to the embodiment of the present application are shown in the figure. In actual application, the smart glasses may have more or less structures. Figure 8 As shown, the smart glasses 200 include: a sound pickup device 201 , an output device 202 , at least one processor 203 and at least one memory 204 .

[0118] The processor 203 is electrically connected to the sound pickup device 201 , the output device 202 and the memory 204 .

[0119] The sound pickup device 201 includes at least one microphone. Optionally, the sound pickup device 201 includes at least one directional microphone and / or at least one omnidirectional microphone. Specifically, the sound pickup device 201 can be a single microphone or a microphone array composed of multiple microphones, for example, a microphone array composed of at least one directional microphone and / or at least one omnidirectional microphone. The output device 202 includes at least one speaker.

[0120] The processor 203 includes: a central processing unit (CPU) and a DSP (Digital Signal Processing). The DSP is used to process the voice data acquired by the microphone 203. The CPU is preferably a MCU (Microcontroller Unit).

[0121] The memory 204 is a non-temporary memory, which may specifically include: RAM (Random Access Memory) and flash memory components, in which one or more programs executable by the processor 203 are stored, and the one or more programs include multiple instructions. The multiple instructions are used to: pick up a first speech in a first language from a preset direction through the sound pickup device 201; translate the first speech to obtain a second speech in a second language through a local or cloud-based generative artificial intelligence language model; and output the second speech through the output device 202.

[0122] Optionally, in other embodiments of the present application, the preset direction points forward of the user.

[0123] Optionally, in other embodiments of the present application, a client program is installed in the smart glasses 200, and the multiple instructions are also used to set the first language and / or the second language according to a first preset operation performed by the user in the client program.

[0124] Optionally, in other embodiments of the present application, the multiple instructions are also used to: set the playback speed of the second voice according to a second preset operation performed by the user in the client program.

[0125] Optionally, in other embodiments of the present application, the smart glasses 200 also include a Bluetooth component 211 electrically connected to the processor 203, and the multiple instructions are also used to: receive a translation instruction sent by a mobile application on the smart mobile terminal through the Bluetooth component 211, enter a translation mode in response to the translation instruction, and in the translation mode, pick up the first voice from the preset direction through the sound pickup device 201.

[0126] The Bluetooth component 211 includes a Bluetooth signal transceiver and surrounding circuits, and can be specifically disposed in the inner cavity of the frame 209 and / or at least one temple 205. The Bluetooth component 211 can be linked to a smart mobile terminal such as a smart phone or a smart watch for phone calls, music, and data communications.

[0127] Optionally, in other embodiments of the present application, such as Fig. 9As shown, the smart glasses 200 also include: at least one temple 205 and a virtual button 206 based on a touch sensor arranged on the temple 205 and electrically connected to the processor 203, and the multiple instructions are also used to: when the user presses the virtual button, control the sound pickup device 201 to pick up the first voice; when the user releases the virtual button, send the first voice to the smart mobile terminal through the Bluetooth component 211, so that the smart mobile terminal uses the generative artificial intelligence large language model set on the smart mobile terminal or the cloud server to translate the first voice into the second voice; and receive the second voice sent by the smart mobile terminal through the Bluetooth component 211.

[0128] The at least one temple 205 may include, for example, a left temple and a right temple.

[0129] Optionally, in other embodiments of the present application, the multiple instructions are also used to: perform language detection on the first voice to determine whether the first language is a preset language through a local or cloud-based speech-to-text engine; when the first language is the preset language, enter a translation mode to convert the first voice into a first text through the speech-to-text engine; generate a translation prompt, translate the first text through the generative artificial intelligence large language model to obtain a second text in the second language, wherein the translation prompt includes information of the first language and the second language; convert the second text into the second voice through a local or cloud-based speech-to-text engine; and exit the translation mode when silence of a preset length is detected by the sound pickup device 201.

[0130] Optionally, in other embodiments of the present application, the smart glasses 200 also include a wireless communication component 207 electrically connected to the processor 203, and the multiple instructions are also used to: send the first voice to the translation server through the wireless communication component 207, so that the translation server translates the first voice into the second voice using a generative artificial intelligence large language model set on the translation server or the model server; and receive the second voice sent by the translation server through the wireless communication component 207.

[0131] The wireless communication component 207 includes a wireless signal transceiver and its surrounding circuits, which can be specifically arranged in the inner cavity of the frame 209 and / or at least one temple 205 of the smart glasses 200. The wireless signal transceiver can use, but is not limited to, at least one of the WiFi (Wireless Fidelity) protocol, NFC (Near Field Communication) protocol, ZigBee (ZigBee), UWB (UltraWideband, carrier-free communication), RFID (Radio Frequency Identification) protocol, and cellular mobile communication (Cellular Mobile Communication) protocol (such as 3G / 4G / 5G, etc.) for data transmission.

[0132] Optionally, in other embodiments of the present application, the output device 202 includes a front speaker, the preset direction points to the user's mouth, and the multiple instructions are also used to play the second voice through the front speaker.

[0133] Optionally, in other embodiments of the present application, the smart glasses 200 also include a Bluetooth component 211 electrically connected to the processor 203, the preset direction points to the user's mouth, and the multiple instructions are also used to send the second voice to an external player through the Bluetooth component 211 for playback.

[0134] Optionally, in other embodiments of the present application, the smart glasses 200 also include: an inertial measurement unit (IMU) sensor 208 electrically connected to the processor 203, and the multiple instructions are also used to: detect whether the first voice has ended by using a voice activity detection algorithm (Voice Activity Detection) and / or sensing data of the IMU sensor; and when it is detected that the first voice has ended, translate the first voice through the generative artificial intelligence large language model to obtain a second voice.

[0135] The working principle of the voice activity detection algorithm is to detect whether the user is speaking by detecting the change in voice energy. When the user speaks, the energy of the voice will increase, and the algorithm can detect the change in voice energy to distinguish whether there is voice or not.

[0136] The sensing data of the IMU sensor is used to detect whether the first voice has ended, mainly by using the IMU sensor to detect the vibration (for example, accelerator) generated when the user speaks. If the vibration stops for more than a preset time, it means that the user has finished speaking.

[0137] Optionally, in addition to the inertial sensor 208, the smart glasses 200 may also include other data sensing components electrically connected to the processor 203, such as: a positioning component, a touch sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and at least one of a pedometer. The positioning component may be, but is not limited to, a positioning component based on GPS (Global Positioning System) or Beidou satellite.

[0138] Furthermore, the smart glasses 200 may also include a frame 209 and a battery 210. The frame 209 may be, for example, a front frame with lenses (eg, sunglass lenses, transparent lenses, or corrective lenses). Preferably, the temples 205 are detachably connected to the frame 209.

[0139] Furthermore, at least one physical control button may be provided on the temple 205 and the frame 209, such as a power button, a volume control button, a translation mode switch button, etc.

[0140] Optionally, in other embodiments of the present application, the smart glasses 200 also include a display screen electrically connected to the processor 203, and the display screen is used to display the above-mentioned second text or translation record.

[0141] The battery 210 is used to provide power support for the electronic components of the above-mentioned smart glasses 200, such as the above-mentioned sound pickup device 201, the output device 202, at least one processor 203, at least one memory 204, the virtual button 206, the wireless communication component 207, the inertial sensor 208 and the Bluetooth component 211.

[0142] The electronic components of the above-mentioned smart glasses can be connected through a bus.

[0143] It should be noted that the relationship between the components of the above-mentioned smart glasses can be a replacement relationship or a superposition relationship. That is, all the components in the above-mentioned embodiment can be installed on one smart glass, or a part of the above-mentioned components can be selectively installed according to needs. When it is a replacement relationship, the smart glasses are also provided with a connection interface for peripherals, and the connection interface can be at least one of a PS / 2 interface, a serial interface, a parallel interface, an IEEE1394 interface, a USB (Universal Serial Bus) interface, etc. The function of the replaced component can be realized by a peripheral connected to the connection interface, such as: an external speaker, an external sensor, etc.

[0144] For details about the smart glasses not covered in this embodiment, please refer to Figures 1 to 7 The illustrated translation system embodiment and Figures 10 to 14The relevant description in the illustrated translation method embodiment will not be repeated here.

[0145] In this embodiment, the smart glasses use the generative artificial intelligence big language model to achieve intelligent translation based on the smart glasses, thereby improving the practicality, interactivity and intelligence of the smart glasses and increasing product stickiness. In addition, due to the scalability and self-creativity of the generative artificial intelligence big language model, the accuracy of the translation can be further improved.

[0146] See also Fig.10 , Fig.10 A flowchart of an artificial intelligence-based translation method provided in an embodiment of the present application. The method can be applied to Figure 1 The first smart mobile terminal 120 or the second smart mobile terminal 150 in the translation system shown. Fig.10 As shown, the method includes:

[0147] S501, receiving a first speech in a first language sent by a smart wearable device, converting the first speech into a first text using a local or cloud-based speech-to-text engine, and obtaining a translation prompt;

[0148] S502, translating the first text according to the translation prompt by a local or cloud-based generative artificial intelligence large language model to obtain a second text in a second language;

[0149] S503: Convert the second text into a second voice using a local or cloud-based text-to-speech engine, and send the second voice to the smart wearable device for playback.

[0150] The smart wearable device may include but is not limited to: a smart helmet, a smart headset, a smart earring, a smart watch, and the like. Figure 8 and Fig. 9 Smart glasses shown, etc.

[0151] Optionally, the first smart mobile terminal 120 may generate the translation prompt through a built-in prompt generator, or the first smart mobile terminal 120 may also generate the translation prompt through a prompt generator configured in a server in the cloud.

[0152] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and the method further includes:

[0153] The mobile application responds to a preset operation of the user and sends a translation instruction to the smart wearable device, so that the smart wearable device responds to the translation instruction, enters a translation mode, and picks up the first voice in the translation mode.

[0154] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and before receiving the first voice in the first language sent by the smart wearable device, the method further includes:

[0155] When a user presses a button on the screen of the smart mobile terminal, a voice pickup instruction is sent to the smart wearable device through the mobile application to instruct the smart wearable device to pick up the first voice, wherein the button is generated by the mobile application;

[0156] The generative artificial intelligence big language model is configured on the model server, and the first text is translated according to the translation prompt by the local or cloud-based generative artificial intelligence big language model to obtain a second text in a second language, including:

[0157] When the user releases the button, or receives a notification message sent by the smart wearable device, the first text is sent to the translation server through the mobile application, so that the translation server generates the translation prompt and sends the first text and the translation prompt to the model server, wherein the notification message is sent by the smart wearable device when the user of the smart wearable device releases the virtual button on the temple of the smart wearable device;

[0158] The second text forwarded by the translation server is received, wherein the second text is sent to the translation server by the model server, and the model server translates the first text by utilizing the generative artificial intelligence large language model according to the translation prompt.

[0159] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and the method further includes:

[0160] generating a translation record according to the second text, or the first text and the second text, through the mobile application, and displaying the translation record on a screen of the smart mobile terminal; and

[0161] The translation record is stored in the smart mobile terminal or storage server through the mobile application.

[0162] Optionally, in other embodiments of the present application, the method further includes: when detecting a user operation on a share button in the user interface of the mobile application, sharing the translation record through the mobile application in accordance with the sharing method indicated by the operation.

[0163] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, and before sending the second voice to the smart wearable device for playing, the method further includes:

[0164] Through the mobile application, the playback speed of the second voice is adjusted to a preset playback speed.

[0165] Optionally, in other embodiments of the present application, a mobile application is installed on the smart mobile terminal, the generative artificial intelligence large language model is configured on the model server, the smart wearable device performs a voice pickup operation, and sends the picked-up voice to the smart mobile terminal in real time via Bluetooth in the form of a data stream, and the method further includes:

[0166] Setting the second language according to a setting operation performed by the user through the mobile application;

[0167] Using the speech-to-text engine to detect whether the language of the received speech is the first language;

[0168] When the language is the first language, a translation start instruction is sent to the smart wearable device through the Bluetooth to instruct the smart wearable device to enter a translation mode, and the received voice is used as the first voice to perform the operation of converting the first voice into a first text using a local or cloud-based voice-to-text engine, and generating a translation prompt; using a local or cloud-based generative artificial intelligence large language model, according to the translation prompt, translating the first text to obtain a second text in a second language, wherein the translation prompt includes information of the first language and the second language; and

[0169] When silence of a preset length of time is detected, a stop translation instruction is sent to the smart wearable device via the Bluetooth to instruct the smart wearable device to exit the translation mode and end the voice pickup operation.

[0170] Optionally, in other embodiments of the present application, the first voice is a voice from a user of the smart wearable device, and the method further includes:

[0171] The second voice is sent to the target terminal, so that the target terminal sends the second voice to the target smart wearable device for playback.

[0172] Optionally, in other embodiments of the present application, a mobile application is installed in the smart mobile terminal, and the method further includes:

[0173] When the smart wearable device as a master smart wearable device completes pairing with a plurality of slave smart wearable devices, a translation prompt is generated through the mobile application;

[0174] By means of the generative artificial intelligence large language model, according to the translation prompt generated by the smart wearable device, the first text is translated to obtain the plurality of third texts, wherein the language of each of the third texts corresponds to the language used by the user of each subordinate smart wearable device;

[0175] Each of the third texts is distributed to the corresponding slave smart wearable device, so that the corresponding slave smart wearable device uses a text-to-speech engine to convert the received third text into a third voice and play the third voice.

[0176] In the above translation method provided in this application, the smart wearable device can have multiple translation modes, which can specifically include: monitoring mode, speaking mode, pairing mode and one-to-many mode. The one-to-many mode can be applied to application scenarios such as tour guides and meetings.

[0177] As an application example, take smart glasses and mobile phones as an example, in scenario A of monitoring mode:

[0178] Aa, users wearing smart glasses can use smart glasses to listen to the other party's foreign language voice, and let the smart glasses translate the voice into the local language and play it;

[0179] Ab, users can set the translated foreign and local languages ​​through the built-in application of smart glasses;

[0180] Ac, users can set the playback speed of local language voice;

[0181] Ad, users can start the translation function through the mobile application in their mobile phone to trigger the smart glasses to enter the translation mode;

[0182] Ae, the user presses the virtual button on the smart glasses, or the user presses the screen button on the UI interface of the mobile application on the screen of the mobile phone, and the microphone of the smart glasses starts to listen to the foreign language voice of the other party;

[0183] Af, the smart glasses send the foreign language speech in audio format to the mobile application in real time via Bluetooth;

[0184] Ag, the mobile app runs a speech-to-text engine (cloud or local device) that converts speech into foreign language text;

[0185] Ah, at the same time, the mobile application displays transcripts in foreign languages, where transcripts in foreign languages ​​can be shared or saved;

[0186] Ai, when a virtual button on the smart glasses is released, or a screen button on the mobile application is released on the screen of the mobile phone, the mobile application sends the foreign language text to the translation server;

[0187] Aj, the translation server sends the foreign language text along with the translation hint (generated on the translation server) to GAILLM (cloud or local);

[0188] Ak, GAILLM translates the text format into the local language and sends it back to the translation server;

[0189] Al, the translation server sends the local language back to the mobile application in text format;

[0190] Am, a mobile app that converts local language text to speech via a text-to-speech engine (cloud or local device);

[0191] An, the mobile application displays the translated local language transcript on the screen, wherein the translated local language transcript can be shared or saved in the mobile phone;

[0192] Ao, a mobile app that sends the translated local language speech back to the smart glasses in audio format at the desired playback speed;

[0193] Ap, the smart glasses play the translated local language speech through the speakers.

[0194] As an application example, take smart glasses and mobile phones as an example, in scenario B of monitoring mode:

[0195] Ba, similar to scenario A, but users do not need to set a foreign language, they only need to send the local language;

[0196] Bb, the speech-to-text engine is able to automatically detect which foreign language.

[0197] As an application example, take smart glasses and mobile phones as an example, in scenario C of monitoring mode:

[0198] Ca, extending to scenario B, the user does not need to use the virtual button on the smart glasses or the screen button on the mobile application to start or stop listening.

[0199] Cb, when the speech-to-text engine detects that the speech is in a foreign language, it automatically starts translating the picked-up speech and stops translating after detecting a period of silence.

[0200] As an application example, take smart glasses and mobile phones as an example, in scene D of monitoring mode:

[0201] Da, similar to scenarios A, B, or C above, but without the mobile app.

[0202] Db, the smart glasses directly send the foreign language speech to the translation server, and play the local language speech received by the translation server directly on the smart glasses.

[0203] As an application example, take smart glasses and mobile phones as an example, in scenario E of speaking mode:

[0204] Ea, users wearing smart glasses can use smart glasses to listen to their own local language voice and let the smart glasses play the translated foreign language voice through the attached front speakers;

[0205] Eb, users can set local and foreign languages ​​through the built-in application of smart glasses;

[0206] Ec, users can set the playback speed of foreign language voice;

[0207] Ed, the user starts the translation function on the mobile application in the phone;

[0208] Ee, the user presses the virtual button on the smart glasses, or presses the screen button in the UI interface of the mobile application on the screen of the mobile phone, and the smart glasses microphone starts to listen to the user's local language voice;

[0209] Ef, the smart glasses send the local language speech in audio format to the mobile phone in real time via Bluetooth;

[0210] E.g., the mobile application runs a speech-to-text engine (cloud or local device) to convert the speech into text in the local language;

[0211] Eh, at the same time, the mobile application displays the transcript in the local language, wherein the transcript in the local language can be shared or saved;

[0212] Ei, when a virtual button on the smart glasses is released, or a screen button on the mobile application is released, the mobile application sends the local language in text form to the translation server;

[0213] Ej, the translation server sends the local language text along with the translation hint (generated on the translation server) to the generative GAILLM (cloud or local);

[0214] Ek, GAILLM translates the local language text in text format into a foreign language in text format and sends it back to the translation server;

[0215] El, the translation server sends the foreign language in text format back to the mobile application;

[0216] Em, the mobile application converts the foreign language text into corresponding foreign language speech through a text-to-speech engine (cloud or local device);

[0217] En, the mobile application displays a foreign language transcript on a screen, wherein the foreign language transcript can be shared or saved;

[0218] Eo, the mobile app sends the foreign language speech back to the smart glasses in audio format at the desired playback speed;

[0219] Ep, the smart glasses play the foreign language speech through the front speakers so that the person talking can listen.

[0220] As an application example, take smart glasses and mobile phones as an example, in scene F of speaking mode:

[0221] Fa, similar to scenario D, the foreign language speech is played through the mobile application in the phone instead of through the front speakers on the smart glasses or the external speakers paired with the smart glasses.

[0222] As an application example, take smart glasses and mobile phones as an example, in the speaking mode scene G:

[0223] Ga, similar to scenario E or F, but the user does not need to release the virtual button on the glasses or the screen button on the mobile application to indicate the end of the speech;

[0224] Gb, users press and immediately release a virtual button on the smart glasses or an on-screen button on the mobile app to start listening;

[0225] Gc, the smart glasses can automatically detect whether the user's voice has ended through the voice activity detection algorithm and / or the IMU signal.

[0226] As an application example, taking smart glasses and mobile phones as an example, in scene H of the speaking mode, the following operations can also be performed:

[0227] Ha, similar to scenario G, the user does not need to use the on-screen button in the mobile app or the virtual button on the smart glasses to start listening;

[0228] Hb, when the speech-to-text engine detects that the picked-up speech is in the local language, it will automatically start translating the picked-up speech.

[0229] As an application example, take smart glasses and mobile phones as an example, in the speaking mode scenario I:

[0230] Ia, similar to scenario E, G, or H, but without the mobile app;

[0231] Ib, the smart glasses directly send the local language speech to the translation server, and play the translated foreign language speech directly through the front speakers on the smart glasses or the external speakers paired with the smart glasses.

[0232] As an application example, take smart glasses and mobile phones as an example, in scenario J of pairing mode:

[0233] Ja, users A and B wear smart glasses A and B, respectively, and use languages ​​A and B, respectively. Smart glasses A and B correspond to mobile phones A and B, respectively, and these smart glasses are paired for translation mode;

[0234] Jb, user A presses the virtual button on smart glasses A, or presses the screen button on mobile application A in mobile phone A, and the microphone of smart glasses A starts to listen to the voice of user A in language A;

[0235] Jc, smart glasses A sends the speech of language A in audio format to mobile phone A in real time via Bluetooth;

[0236] Jd, mobile application A utilizes a speech-to-text engine (cloud or local device) to convert speech in language A into a first text in language A;

[0237] Je, when user A releases the virtual button on smart glasses A, or releases the screen button on mobile application A, mobile application A sends the first text in language A in text form to the translation server;

[0238] Jf, the translation server sends the first text in language A together with the translation prompt in text form to GAILLM (cloud or local);

[0239] Jg, GAILLMS translates the first text in the text format of language A into the second text in the text format of language B, and then sends it back to the translation server;

[0240] Jh, the translation server sends the second text in language B to the mobile application B in mobile phone B in text format;

[0241] Ji, mobile application B converts the second text in language B into speech in language B through a text-to-speech engine (in the cloud or on a local device);

[0242] Jj, mobile application B sends the speech in language B to smart glasses B in audio format;

[0243] Jk, smart glasses B play the speech in language B through the speaker so that user B can listen;

[0244] Jl, the transcripts of languages ​​A and B are displayed on mobile applications A and B and can be shared and saved;

[0245] Jm, on the contrary, when user B wants to communicate with A, he only needs to repeat steps Jb to JI, and user B's spoken voice will be translated into language A's voice and played on user A's smart glasses A.

[0246] As an application example, take smart glasses and mobile phones as an example, in the one-to-many mode scenario K:

[0247] Ka, users A, B, C and D wear smart glasses A, B, C and D respectively, and use languages ​​A, B, C and D respectively. Smart glasses A, B, C and D are paired for translation mode. Smart glasses A, B, C and D interact with mobile phones A, B, C and D via Bluetooth respectively. Mobile applications A, B, C and D are installed in mobile phones A, B, C and D respectively.

[0248] Kb, user A presses the right temple of smart glasses A and speaks, the microphone of smart glasses A picks up the voice A of user A in language A, and sends it to the translation server in text form through mobile application A, the translation server translates the text in language A into text in languages ​​B, C, and D and sends the translated texts to the corresponding smart glasses B, C, and D respectively; smart glasses B, C, and D each translate the received text into corresponding voices through a voice-to-text engine, and then play the converted voices to their respective users through their respective speakers;

[0249] Kc, when user B wants to communicate with user A, user B presses the right temple of smart glasses B to speak, the microphone of smart glasses B picks up the speech of user B and converts the picked up speech in language B into text in language B through the speech engine, the text in language B is sent to the translation server by the smart glasses, the translation server translates the text in language B into text in language A, and then sends the text in language A to smart glasses A; smart glasses A convert the text in language A into speech in language A through the text-to-speech engine, and then play the speech in language A through the speaker of smart glasses A;

[0250] Kd, users C and D communicate with user A in different languages ​​through step Kc.

[0251] To further illustrate the above four translation modes, Figures 11 to 14 Provide explanation.

[0252] like Fig.11 As shown, in the monitoring mode, user A does not wear smart glasses, and user B wears smart glasses. The smart glasses worn by user B establish a data connection with the mobile APP on user B's mobile phone via Bluetooth. User B can set the A language used by user A and the B language used by user B through the mobile APP, and switch the current translation mode to the monitoring mode. In this monitoring mode, the A language used by user A is the first language (i.e., the source language), and the B language used by user B is the second language (i.e., the target language).

[0253] When user B presses the virtual button on the right temple of the smart glasses, user A starts to speak, and the smart glasses pick up user A's voice through the microphone and send the picked up user A's voice in language A to the mobile APP. When user B releases the virtual button, the mobile APP uses GAILLM to translate the received voice into language B and sends it to the smart glasses, so that the smart glasses play the voice in language B for user B through the built-in speakers.

[0254] At the same time, the mobile APP can display the original text in language A and the translated text in language B corresponding to the voice of user A on the mobile phone screen of user B.

[0255] like Fig.12 As shown, in the speaking mode, user A does not wear smart glasses, and user B wears smart glasses. The smart glasses worn by user B establish a data connection with the mobile APP on user B's mobile phone via Bluetooth. User B can set the language used by user A and the language used by user B through the mobile APP, and switch the current translation mode to the speaking mode. In the speaking mode, the B language used by user B is the first language (i.e., the source language), and the A language used by user A is the second language (i.e., the target language).

[0256] When user B presses the virtual button on the right temple of the smart glasses, user B starts to speak, and the smart glasses pick up the voice of user B through the microphone and send the picked up voice of user B in language B to the mobile APP. When user B releases the virtual button, the mobile APP uses GAILLM to translate the received voice into language A and sends it to the smart glasses, so that the smart glasses play the voice in language A for user A through the front speakers.

[0257] At the same time, the mobile APP can display the original text in language B corresponding to the voice of user B and the translated text in language A on the mobile phone screen of user B.

[0258] Furthermore, Fig.11 The monitoring mode shown can be Fig.12 Specifically, user A does not wear smart glasses, and user B wears smart glasses. The smart glasses worn by user B establish a data connection with the mobile APP on user B's mobile phone via Bluetooth.

[0259] On the one hand, when user B presses and holds the virtual button on the smart glasses he is wearing, the smart glasses enter the monitoring mode, and the smart glasses pick up the voice of user A through the microphone and send the picked-up voice of user A in language A to the mobile APP.

[0260] When user B releases the virtual button, the smart glasses use GAILLM through the mobile APP to translate the speech in language A into speech in language B, and play the speech in language B for user B through the built-in speaker. At the same time, the mobile APP can display the original text in language A and the translated text in language B corresponding to the speech in language A on the screen of user B's mobile phone.

[0261] On the other hand, when the smart glasses detect that user B starts speaking using a voice activity detection algorithm and / or the IMU sensor on the smart glasses, the smart glasses switch the current translation mode from a listening mode to a speaking mode, pick up the voice of user B through a microphone, and send the picked up voice of user B in language B to the mobile APP.

[0262] When the smart glasses detect that user B stops speaking using the voice activity detection algorithm and / or the IMU sensor on the smart glasses, the smart glasses use GAILLM through the mobile APP to translate the speech in language B into speech in language A, and play the speech in language A for user A through the front speakers of the smart glasses, or external speakers, or the mobile phone. At the same time, the mobile APP can display the original text in language B corresponding to the speech in language B and the translated text in language A on the mobile phone screen of user B.

[0263] like Fig.13 As shown, in pairing mode, user A wears smart glasses A, and user B wears smart glasses B. Smart glasses A worn by user A establish a data connection with the mobile APP on user A's mobile phone A via Bluetooth. Smart glasses B worn by user B establish a data connection with the mobile APP on user B's mobile phone B via Bluetooth. Smart glasses A and smart glasses B are paired through mobile phones A and B.

[0264] User A can set the language used by user A and the language used by user B through the mobile APP in mobile phone A, and switch the current translation mode to pairing mode. In this pairing mode, in mobile phone A, the language A used by user A is the first language (i.e., the source language), and the language B used by user B is the second language (i.e., the target language).

[0265] User B can set the language used by user B and the language used by user A through the mobile APP in mobile phone B, and switch the current translation mode to pairing mode. In this pairing mode, in mobile phone B, the language B used by user B is the first language (i.e., the source language), and the language A used by user A is the second language (i.e., the target language).

[0266] When user B presses the virtual button on the right temple of the smart glasses B, user B starts to speak, and the smart glasses B pick up the voice of user B through the microphone and send the picked up voice in language B to the mobile APP in mobile phone B. When user B releases the virtual button, the mobile APP in mobile phone B uses GAILLM to translate the voice in language B into text in language A and send it to the mobile APP in mobile phone A. The mobile APP in mobile phone A converts the text in language A into the corresponding voice in language A, and sends the voice in language A to smart glasses A for playback.

[0267] When user A presses the virtual button on the right temple of the smart glasses A, the smart glasses A and mobile phone A use a similar method to user B to translate the picked-up speech of user A in language A into text in language B and send it to the mobile app on mobile phone B. The mobile app in mobile phone B converts the text in language B into the corresponding speech in language B and sends the speech in language B to smart glasses B for playback.

[0268] Furthermore, the mobile APP in mobile phone A can display the original text and translation processed by itself on the screen of mobile phone A. The mobile APP in mobile phone B can display the original text and translation processed by itself on the screen of mobile phone B.

[0269] Furthermore, mobile phones A and B can also display the original text and translation processed by the other party on their respective screens in a sharing manner.

[0270] like Fig.14 As shown in the figure, in the one-to-many mode of the tour guide, users A, B, C and D wear smart glasses A, B, C and D respectively. Among them, user A is a tour guide, and the smart glasses A he wears are the server side (i.e., the master glasses). Users B, C and D are tourists, and the smart glasses B, C and D they wear are the clients (i.e., the slave glasses). Smart glasses A, B, C and D establish data connections with mobile phones A, B, C and D respectively.

[0271] When user A presses the virtual button on smart glasses A, smart glasses A pick up the voice of user A in language A, and then use the translation method in the above embodiment to translate the voice in language A into voice in language B, language C and language D through data interaction with mobile phone A and send them to smart glasses B, C and D respectively.

[0272] When user B presses the virtual button on smart glasses B, smart glasses B pick up the voice of user B in language B, and then use the translation method in the above embodiment to translate the voice in language B into language A and send it to smart glasses A through data interaction with mobile phone B.

[0273] Furthermore, the terminal responsible for translation can also generate a translation record based on the original text and the translation, and distribute the translation record to each smart glass and / or each mobile phone for sharing.

[0274] For details about the translation method not covered in this embodiment, please refer to Figures 1 to 9 The relevant description in the illustrated embodiment will not be repeated here.

[0275] In this embodiment, by using a smart mobile terminal combined with a generative artificial intelligence big language model, smart translation based on smart glasses is realized, thereby improving the practicality, interactivity and intelligence of smart glasses and increasing product stickiness. In addition, due to the scalability and self-creativity of the generative artificial intelligence big language model, the accuracy of the translation can be further improved.

[0276] The present application also provides a non-transitory computer-readable storage medium, which may be disposed in the smart glasses or smart wearable devices in the above embodiments. The non-transitory computer-readable storage medium may be the above Figure 8 The memory 204 in the illustrated embodiment. The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the artificial intelligence-based translation method described in the aforementioned embodiments is implemented. Furthermore, the computer-storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0277] In the several embodiments provided in the present application, it should be understood that the disclosed smart glasses, translation system and translation method can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the connection or direct connection or communication connection between each other shown or discussed can be an indirect connection or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0278] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0279] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0280] The above is a description of the smart glasses, translation system and translation method provided in this application. For technicians in this field, according to the ideas of the embodiments of this application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this application.

Claims

1. A translation system based on artificial intelligence, characterized in that: include: First smart glasses, a first smart mobile terminal and a cloud server, wherein the cloud server is configured with a generative artificial intelligence large language model; The first smart glasses are used to pick up a first voice in a first language from a preset direction, and send the first voice to the first smart mobile terminal via Bluetooth; The first smart mobile terminal is used to convert the first speech into a first text using a speech-to-text engine, and send the first text to the cloud server, wherein the speech-to-text engine is arranged in the first smart mobile terminal or the cloud; The cloud server is used to translate the first text through the generative artificial intelligence large language model according to the translation prompt to obtain a second text in a second language, and send the second text to the first smart mobile terminal; The first smart mobile terminal is further used to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses via Bluetooth, wherein the text-to-speech engine is arranged in the first smart mobile terminal or in the cloud; The first smart glasses are further configured to play the second voice.

2. The translation system according to claim 1, characterized in that The first intelligent mobile terminal has a mobile application installed thereon; The first smart mobile terminal is further configured to send a translation instruction to the first smart glasses in response to a preset operation of the user through the mobile application; The first smart glasses are further configured to enter a translation mode in response to the translation instruction, and pick up the first voice from the preset direction in the translation mode.

3. The translation system according to claim 1, wherein: The cloud server includes a translation server and a model server, a mobile application is installed on the first smart mobile terminal, and the generative artificial intelligence large language model is configured in the model server; The first smart mobile terminal is further configured to send a voice pickup instruction to the first smart glasses through the mobile application when a user presses a button on a screen of the first smart mobile terminal, wherein the button is generated by the mobile application; The first smart glasses are further configured to pick up the first voice from the preset direction in response to the voice pickup instruction; The first intelligent mobile terminal is further configured to send the first text to the translation server through the mobile application when the user releases the button; The translation server is used to generate the translation prompt and send the first text and the translation prompt to the model server; The model server is used to translate the first text to obtain the second text according to the translation prompt by using the generative artificial intelligence large language model, and send the second text to the translation server; The translation server is further configured to send the second text to the first smart mobile terminal.

4. The translation system according to claim 1, wherein: The cloud server includes a translation server and a model server, a mobile application is installed on the first smart mobile terminal, and the generative artificial intelligence large language model is configured in the model server; The first smart glasses are further configured to pick up the first voice from the preset direction when the user presses a virtual button on the temple of the first smart glasses; The first smart glasses are further configured to send notification information to the mobile application via Bluetooth when the user releases the virtual button; The first intelligent mobile terminal is further configured to send the first text to the translation server through the mobile application according to the notification information; The translation server is used to generate the translation prompt and send the first text and the translation prompt to the model server; The model server is used to translate the first text to obtain the second text according to the translation prompt by using the generative artificial intelligence large language model, and send the second text to the translation server; The translation server is further configured to send the second text to the first smart mobile terminal.

5. The translation system according to claim 1, wherein: The first smart mobile terminal has a mobile application installed thereon, and the first smart mobile terminal is further used for: generating a translation record according to the second text, or the first text and the second text, through the mobile application, and displaying the translation record on the screen of the first smart mobile terminal; The translation record is stored in the first smart mobile terminal or storage server through the mobile application.

6. The translation system according to claim 5, characterized in that The first intelligent mobile terminal is further configured to share the translation record through the mobile application in a sharing manner indicated by the operation when detecting that the user operates a share button in the user interface of the mobile application.

7. The translation system according to claim 1, wherein: The first smart mobile terminal has a mobile application installed thereon, and the first smart mobile terminal is further used for: The second voice with a preset playing speed is generated through the mobile application, and the second voice with the preset playing speed is sent to the first smart glasses through the Bluetooth for playing.

8. The translation system according to claim 1, wherein: The first intelligent mobile terminal has a mobile application installed thereon. The first intelligent mobile terminal is further used to set the second language according to a setting operation performed by a user through the mobile application; The first smart glasses are further used to perform a voice pickup operation and send the picked-up voice to the first smart mobile terminal in real time in the form of a data stream via Bluetooth; The first intelligent mobile terminal is further used for: Using the speech-to-text engine to detect whether the language of the received speech is the first language; When the language is the first language, sending a translation start instruction to the first smart glasses via the Bluetooth, converting the received speech as the first speech into a first text via the speech-to-text engine, and sending the first text and the translation prompt to the model server, wherein the translation prompt includes information of the first language and the second language; as well as When silence of a preset duration is detected, a stop translation instruction is sent to the first smart glasses via the Bluetooth; The first smart glasses are further configured to enter a translation mode in response to the start translation instruction, and to exit the translation mode and end the voice pickup operation in response to the stop translation instruction.

9. The translation system according to claim 1, wherein: The cloud server includes a translation server and a model server, and the generative artificial intelligence large language model is configured in the model server; The first smart glasses are further used to send the first voice to the translation server via a wireless network; The translation server is used to convert the first speech into a first text using a speech-to-text engine, generate a translation prompt, and send the first text and the generated translation prompt to the model server; The model server is used to translate the first text to obtain the second text according to the translation prompt by using the generative artificial intelligence large language model, and send the second text to the translation server; The translation server is further configured to convert the second text into a second voice using a text-to-speech engine, and send the second voice to the first smart glasses via the wireless network.

10. The translation system according to claim 1, wherein: The preset direction points to the front of the user of the first smart glasses or the mouth of the user of the first smart glasses.

11. The translation system according to claim 1, wherein: The preset direction points to the mouth of the user of the first smart glasses, and the translation system further includes a second pair of smart glasses and a second smart mobile terminal; The first smart mobile terminal is further configured to send the second voice to the second smart mobile terminal; The second smart mobile terminal is used to send the second voice to the second smart glasses; The second smart glasses are used to play the second voice.

12. The translation system according to claim 11, characterized in that The first smart mobile terminal is further used to generate a translation record in real time according to the first text and the second text through a mobile application on the first smart mobile terminal, save the translation record in the first smart mobile terminal, display the translation record, and synchronize the translation record to the second smart mobile terminal; The second smart mobile terminal is further used to save the translation record in the second smart mobile terminal and display the translation record through a mobile application on the second smart mobile terminal.

13. The translation system according to claim 1, wherein: The cloud server includes a translation server, the translation system also includes a plurality of second smart glasses, and the generative artificial intelligence large language model is configured in the translation server; The first smart glasses are further configured to pick up a first voice of a user of the first smart mobile terminal and send the first voice to the first smart mobile terminal via Bluetooth when the user of the first smart glasses presses a button on the temple of the first smart glasses; The first smart mobile terminal is further used to convert the first speech into the first text by using the speech-to-text engine through a mobile application in the first smart mobile terminal, and send the first text to the translation server; The translation server is used to generate a first translation prompt, translate the first text according to the first translation prompt through the generative artificial intelligence large language model to obtain multiple third texts, and distribute each of the third texts to the corresponding second smart glasses, wherein the language of each of the third texts corresponds to the language used by the user of each of the second smart glasses; The second smart glasses are used to convert the received third text into a third voice using a text-to-speech engine, and play the third voice.

14. The translation system according to claim 13, wherein: The second smart glasses are further configured to pick up a fourth voice of the user of the second smart glasses when the user of the second smart glasses presses a virtual button on the temple of the second smart glasses, convert the fourth voice into a fourth text through a speech-to-text engine, and send the fourth text to the translation server; The translation server is configured to generate a second translation prompt, translate the fourth text according to the second translation prompt through the generative artificial intelligence large language model to obtain a fifth text, and send the fifth text to the first smart glasses, wherein the language of the fifth text corresponds to the language used by the user of the first smart glasses; The first smart glasses are further configured to utilize a text-to-speech engine to convert the fifth text into a fifth voice and play the fifth voice.

15. The translation system according to claim 1, wherein: The cloud server includes a model server, and the generative artificial intelligence large language model is configured in the model server.

16. The translation system according to claim 1, wherein: The first intelligent mobile terminal is further used to generate the translation prompt and send the translation prompt to the cloud server.

17. A kind of smart glasses for translation based on artificial intelligence, characterized in that: include: A sound pickup device, an output device, a processor and a memory, wherein the processor is electrically connected to the sound pickup device, the output device and the memory; The memory stores one or more programs executable by the processor, wherein the one or more programs include multiple instructions, and the multiple instructions are used to: Picking up a first speech of a first language from a preset direction by the sound pickup device; The first speech is translated into a second speech in a second language by using a generative artificial intelligence large language model locally or in the cloud; as well as The second speech is outputted through the output device.

18. The smart glasses according to claim 17, wherein: The preset direction points to the front of the user.

19. The smart glasses according to claim 17, wherein: The smart glasses are installed with a client program, and the plurality of instructions are further used for: The first language and / or the second language is set according to a first preset operation performed by the user in the client program.

20. The smart glasses according to claim 19, wherein: The multiple instructions are also used to set the playback speed of the second voice according to a second preset operation performed by the user in the client program.

21. The smart glasses according to claim 17, wherein: The smart glasses further include a Bluetooth component electrically connected to the processor, and the plurality of instructions are further used to: The translation instruction sent by the mobile application on the smart mobile terminal is received through the Bluetooth component, a translation mode is entered in response to the translation instruction, and in the translation mode, the first voice from the preset direction is picked up by the sound pickup device.

22. The smart glasses according to claim 21, wherein: The smart glasses further include: temples and virtual buttons based on touch sensors disposed on the temples and electrically connected to the processor, and the plurality of instructions are further used to: When the user presses the virtual button, controlling the sound pickup device to pick up the first voice; When the user releases the virtual button, the first voice is sent to the smart mobile terminal through the Bluetooth component, so that the smart mobile terminal translates the first voice into the second voice by using a generative artificial intelligence language model set on the smart mobile terminal or the cloud server; and The second voice sent by the smart mobile terminal is received through the Bluetooth component.

23. The smart glasses according to claim 17, wherein: The plurality of instructions are also used to: Performing language detection on the first speech to determine whether the first language is a preset language by a local or cloud-based speech-to-text engine; When the first language is the preset language, entering a translation mode, and converting the first speech into a first text through the speech-to-text engine; Generate a translation prompt, translate the first text through the generative artificial intelligence large language model to obtain a second text in the second language, wherein the translation prompt includes information of the first language and the second language; Converting the second text into the second voice through a local or cloud-based voice-to-text engine; as well as When silence of a preset duration is detected by the sound pickup device, the translation mode is exited.

24. The smart glasses according to claim 17, wherein: The smart glasses further include a wireless communication component electrically connected to the processor, and the plurality of instructions are further used to: Sending the first speech to a translation server through the wireless communication component, so that the translation server translates the first speech into the second speech by using a generative artificial intelligence large language model set on the translation server or the model server; as well as The second voice sent by the translation server is received through the wireless communication component.

25. The smart glasses according to claim 17, wherein: The output device includes a front speaker, the preset direction points to the user's mouth, and the multiple instructions are also used to play the second voice through the front speaker.

26. The smart glasses according to claim 17, wherein: The smart glasses also include a Bluetooth component electrically connected to the processor, the preset direction points to the user's mouth, and the multiple instructions are also used to send the second voice to an external player through the Bluetooth component for playback.

27. The smart glasses according to claim 17, wherein: The smart glasses further include: an inertial sensor electrically connected to the processor, and the plurality of instructions are further used to: detecting whether the first speech has ended using a voice activity detection algorithm and / or sensing data from the inertial sensor; and When it is detected that the first speech ends, the first speech is translated by the generative artificial intelligence large language model to obtain the second speech.

28. A translation method based on artificial intelligence, characterized in that: Applied to a smart mobile terminal, the method comprises: Receiving a first voice in a first language sent by the smart wearable device, converting the first voice into a first text using a local or cloud-based voice-to-text engine, and obtaining a translation prompt; Translating the first text to obtain a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence large language model; The second text is converted into a second voice using a local or cloud-based text-to-speech engine, and the second voice is sent to the smart wearable device for playback.

29. The translation method according to claim 28, characterized in that: The smart mobile terminal has a mobile application installed on it, and the method further includes: The mobile application sends a translation instruction to the smart wearable device in response to a preset operation of the user, so that the smart wearable device enters a translation mode in response to the translation instruction and picks up the first voice in the translation mode.

30. The translation method according to claim 28, characterized in that: The smart mobile terminal has a mobile application installed thereon. Before receiving the first voice in the first language sent by the smart wearable device, the method further includes: When a user presses a button on the screen of the smart mobile terminal, a voice pickup instruction is sent to the smart wearable device through the mobile application to instruct the smart wearable device to pick up the first voice, wherein the button is generated by the mobile application; The generative artificial intelligence big language model is configured on the model server, and the first text is translated according to the translation prompt by the local or cloud-based generative artificial intelligence big language model to obtain a second text in a second language, including: When the user releases the button, or receives a notification message sent by the smart wearable device, the first text is sent to the translation server through the mobile application, so that the translation server generates the translation prompt and sends the first text and the translation prompt to the model server, wherein the notification message is sent by the smart wearable device when the user of the smart wearable device releases the virtual button on the temple of the smart wearable device; The second text forwarded by the translation server is received, wherein the second text is sent by the model server to the translation server, and the model server translates the first text by utilizing the generative artificial intelligence large language model according to the translation prompt.

31. The translation method according to claim 28, characterized in that: The smart mobile terminal has a mobile application installed on it, and the method further includes: generating a translation record according to the second text, or the first text and the second text, through the mobile application, and displaying the translation record on the screen of the smart mobile terminal; The translation record is stored in the smart mobile terminal or storage server through the mobile application.

32. The translation method according to claim 31, characterized in that: The method further comprises: When it is detected that the user operates a share button in the user interface of the mobile application, the translation record is shared through the mobile application in a sharing manner indicated by the operation.

33. The translation method according to claim 28, characterized in that: The smart mobile terminal has a mobile application installed thereon, and before sending the second voice to the smart wearable device for playing, the method further includes: Through the mobile application, the playback speed of the second voice is adjusted to a preset playback speed.

34. The translation method according to claim 28, characterized in that: The smart mobile terminal is installed with a mobile application, the generative artificial intelligence large language model is configured on the model server, the smart wearable device performs a voice pickup operation, and sends the picked-up voice to the smart mobile terminal in real time via Bluetooth in the form of a data stream, and the method further includes: Setting the second language according to a setting operation performed by the user through the mobile application; Using the speech-to-text engine to detect whether the language of the received speech is the first language; When the language is the first language, sending a translation start instruction to the smart wearable device through the Bluetooth to instruct the smart wearable device to enter a translation mode, and using the received voice as the first voice to perform the operations of converting the first voice into a first text using a local or cloud-based voice-to-text engine, and generating a translation prompt; translating the first text into a second text in a second language according to the translation prompt using a local or cloud-based generative artificial intelligence large language model, wherein the translation prompt includes information of the first language and the second language; and When silence of a preset duration is detected, a stop translation instruction is sent to the smart wearable device via the Bluetooth to instruct the smart wearable device to exit the translation mode and end the voice pickup operation.

35. The translation method according to claim 28, characterized in that: The first voice is a voice from a user of the smart wearable device, and the method further includes: The second voice is sent to the target terminal, so that the target terminal sends the second voice to the target smart wearable device for playback.

36. The translation method according to claim 28, characterized in that: The smart mobile terminal has a mobile application installed in it, and the method further includes: When the smart wearable device as a master smart wearable device completes pairing with a plurality of slave smart wearable devices, a translation prompt is generated through the mobile application; By means of the generative artificial intelligence large language model, according to the translation prompt generated by the smart wearable device, the first text is translated to obtain the plurality of third texts, wherein the language of each of the third texts corresponds to the language used by the user of each subordinate smart wearable device; Distribute each of the third texts to the corresponding slave smart wearable device so that the corresponding slave smart wearable device uses a text-to-speech engine to convert the received third text into a third voice and play the third voice.

Images