Multi-party cross-language interaction method and system based on large language model and intelligent terminal

Abstract

The invention discloses a multi-party cross-language interaction method and system based on a large language model and an intelligent terminal. The system comprises a master intelligent terminal and a plurality of slave intelligent terminals. The master intelligent terminal obtains first to-be-translated data of a first user, translates the first to-be-translated data into at least one piece of first data through the first large language model according to the first translation prompt, and distributes the first data to the corresponding slave intelligent terminal for display; wherein the language of each first data corresponds to the language used by the user of each corresponding slave intelligent terminal. Second to-be-translated data of a second user is obtained from the intelligent terminal, the second to-be-translated data is translated into second data through a second large language model according to the second translation prompt, the second data is sent to the main intelligent terminal to be displayed, and the language of the second data is the language used by the user of the main intelligent terminal. According to the method and the device, multi-party cross-language interaction based on the large language model and collaborated by multiple intelligent terminals is realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of smart glasses, and in particular, to a multi-party cross-language interaction method, system, and smart terminal based on a large language model. Background Art

[0002] With the development of computer technology, smart terminals such as smart glasses and smart phones have become more and more popular. However, the existing smart terminals are expensive and usually only have functions of the smart terminal itself, such as listening to music, making or answering calls, browsing the web, etc. The smart terminals are independent of each other and cannot work collaboratively. Summary of the Invention

[0003] The embodiments of the present application provide a multi-party cross-language interaction method, system, and smart terminal based on a large language model, which are used to realize multi-party cross-language interaction based on a large language model for multi-smart-terminal collaboration, thereby improving the practicability, interactivity, and intelligence of smart terminals, and increasing product viscosity.

[0004] On the one hand, the embodiments of the present application provide a multi-party cross-language interaction system based on a large language model, including: a main smart terminal and multiple slave smart terminals;

[0005] The main smart terminal is used to obtain first to-be-translated data of a first user, and through a first large language model, according to a first translation hint, translate the first to-be-translated data into at least one first data and distribute it to corresponding slave smart terminals for display, where the language of each first data corresponds to the language used by the user of each corresponding slave smart terminal, and the first large language model is configured in the main smart terminal or a cloud server;

[0006] The slave smart terminal is used to obtain second to-be-translated data of a second user, and through a second large language model, according to a second translation hint, translate the second to-be-translated data into second data and send it to the main smart terminal for display, where the language of the second data is the language used by the user of the main smart terminal, and the second large language model is configured in the slave smart terminal or the cloud server.

[0007] On the one hand, the embodiments of the present application also provide a smart terminal based on a large language model, including: an input device, a processor, a wireless communication component, and a memory, where the processor is electrically connected to the input device, the wireless communication component, and the memory;

[0008] One or more programs executable by the processor are stored in the memory, and the one or more programs include a plurality of instructions, and the plurality of instructions are used for:

[0009] In response to the first configuration instruction, configure the intelligent terminal as the host;

[0010] When the intelligent terminal serves as the host, obtain the first data to be translated through the input device and send it to the cloud server, so that the cloud server uses the large language model in the cloud server to translate the first data to be translated into at least one first data according to the first translation hint and distribute it to at least one slave intelligent terminal, where the first data to be translated includes the first text to be translated or the first speech from the user, and the languages of the first data respectively correspond to the languages used by the users of the slave intelligent terminals;

[0011] In response to the second configuration instruction, configure the intelligent terminal as a slave;

[0012] When the intelligent terminal serves as the slave, obtain the second data to be translated through the input device and send it to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into a second data according to the second translation hint and send it to the master intelligent terminal, where the second data to be translated includes the second text to be translated or the second speech from the user, and the language of the second data corresponds to the language used by the user of the master intelligent terminal.

[0013] An embodiment of the present application also provides a multi-party cross-language interaction method based on a large language model, which is applied to an intelligent mobile terminal. The method includes:

[0014] In response to the first configuration instruction, configure the intelligent mobile terminal as the host;

[0015] As the host, obtain the first data to be translated and send it to the cloud server, so that the cloud server uses the large language model in the cloud server to translate the first data to be translated into at least one first data according to the first translation hint and distribute it to at least one slave intelligent mobile terminal, where the first data to be translated includes the first text to be translated or the first speech from the user, and the languages of the first data respectively correspond to the languages used by the users of the slave intelligent mobile terminals;

[0016] In response to the second configuration instruction, configure the intelligent mobile terminal as a slave;

[0017] As the slave device, obtain the second data to be translated and send it to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into the second data according to the second translation prompt and send it to the main intelligent mobile terminal, where the second data to be translated includes the second text to be translated or the second voice from the user, and the language of the second data corresponds to the language used by the user of the main intelligent mobile terminal.

[0018] In each embodiment of the present application, by combining multiple intelligent terminals with a large language model, multi-intelligent terminal collaboration-based multi-party cross-lingual interaction based on the large language model is realized, thereby improving the practicality, interactivity and intelligence of the intelligent terminal, and increasing product viscosity. Moreover, due to the scalability and self-creation of the large language model, the accuracy of translation can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic structural diagram of a multi-party cross-lingual interaction system based on a large language model provided by an embodiment of the present application;

[0021] Figure 2 It is a schematic structural diagram of a multi-party cross-lingual interaction system based on a large language model provided by another embodiment of the present application;

[0022] Figure 3 It is a schematic internal structure diagram of an intelligent terminal based on a large language model provided by an embodiment of the present application;

[0023] Figure 4 It is a schematic internal structure diagram of an intelligent terminal based on a large language model provided by another embodiment of the present application;

[0024] Figure 5 It is a flowchart of a multi-party cross-lingual interaction method based on a large language model provided by an embodiment of the present application;

[0025] Figure 6 For Figure 5 a schematic diagram of an application example of the method shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the accompanying drawings in the embodiments of this application. Apparently, the described embodiments are some but not all of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0027] In the following text, the terms "comprising", "having", and their cognates that can be used in various embodiments of this application are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be construed as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or precluding the possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0028] In addition, the terms "first", "second", "third", etc. are only used for descriptive distinction and should not be construed as indicating or implying relative importance.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the various embodiments of this application belong. The terms (such as those defined in a commonly used dictionary) will be interpreted as having the same meaning as their contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of this application.

[0030] See Figure 1 , Figure 1 which is a schematic structural diagram of a multi-party cross-lingual interaction system based on a large language model provided for an embodiment of this application. As Figure 1 shown, the multi-party cross-lingual interaction system 100 includes: a main intelligent terminal 110 and multiple slave intelligent terminals 120. Among them, the multiple slave intelligent terminals 120 are affiliated with the main intelligent terminal 110. Their relationship can be, but is not limited to, for example, a tour guide and tourists of different nationalities (languages) in a tour guide scenario, a host and other participants of different nationalities (languages) in a conference scenario, and a host and exhibitors of different nationalities (languages) in an exhibition scenario.

[0031] The main intelligent terminal 110 is used to obtain the first data to be translated of the first user, and through the first large language model (LLM), according to the first translation prompt, translate the first data to be translated into at least one first data and distribute it to the corresponding slave intelligent terminal 120 for display. Among them, the language of each of the first data corresponds to the language used by the user of each corresponding slave intelligent terminal 120. The first large language model is configured in the main intelligent terminal 110 or the cloud server. The first user is the user of the main intelligent terminal 110.

[0032] The slave intelligent terminal 120 is used to obtain the second data to be translated of the second user, and through the second large language model, according to the second translation prompt, translate the second data to be translated into the second data and send it to the main intelligent terminal 110 for display. Among them, the language of the second data is the language used by the user of the main intelligent terminal 110. The second large language model is configured in the slave intelligent terminal 120 or the cloud server. The second user is the user of the slave intelligent terminal 120.

[0033] Optionally, in other embodiments of the present application, the main intelligent terminal 110 is further used to: determine whether the language used by the user of at least one terminal among the multiple slave intelligent terminals 120 is different from the language used by the first user; if not, distribute the first data to be translated to each slave intelligent terminal 120 for display; if so, send the first data to be translated to at least one first terminal among the multiple slave intelligent terminals 120 for display, and based on at least one second terminal among the multiple slave intelligent terminals 120, perform the operation of translating the first data to be translated into at least one first data according to the first translation prompt and distributing it to the corresponding slave intelligent terminal 120 for display, where the language of each of the first data corresponds to the language used by the user of each of the second terminals, the language used by the user of the first terminal is the same as the language used by the first user, and the language used by the user of the second terminal is different from the language used by the first user.

[0034] Specifically, it is determined whether the language used by the users of at least one of the multiple slave intelligent terminals 120 is different from the language used by the user of the master intelligent terminal 110. If not, that is, the languages used by the users of the multiple slave intelligent terminals 120 are all the same as the language used by the user of the master intelligent terminal 110, the master intelligent terminal 110 does not translate the first data to be translated, but directly distributes the first data to be translated to each slave intelligent terminal 120 for display. If so, that is, the language used by the users of at least one of the multiple slave intelligent terminals 120 is different from the language used by the user of the master intelligent terminal 110, on the one hand, the first data to be translated is directly sent to at least one first terminal among the multiple slave intelligent terminals 120 for display, where the language used by the user of the first terminal is the same as the language used by the first user; on the other hand, according to the first translation prompt, the first data to be translated is translated into at least one first data and distributed to at least one second terminal among the corresponding multiple slave intelligent terminals 120 for display. Among them, the languages of the respective first data correspond to the languages used by the users of the respective second terminals, and the language used by the user of the second terminal is different from the language used by the first user.

[0035] For example, if the user 1 of the master intelligent terminal 110 uses Chinese, the user 2 of the slave intelligent terminal 120A uses English, the user 3 of the slave intelligent terminal 120B uses Japanese, and the user 4 of the slave intelligent terminal 120C uses Chinese, the master intelligent terminal 110 directly sends the first data to be translated from user 1 to the slave intelligent terminal 120C, and at the same time translates the first data to be translated into the first data in English and the first data in Japanese, and sends the first data in English to the slave intelligent terminal 120A and the first data in Japanese to the slave intelligent terminal 120B.

[0036] Optionally, in other embodiments of the present application, the slave intelligent terminal 120 is further configured to: determine whether the language used by the second user is the same as the language used by the first user; if the same, send the second data to be translated to the master intelligent terminal 110 for display; if not, perform the operation of translating the second data to be translated into the second data according to the second translation prompt by the second large language model and sending it to the master intelligent terminal 110 for display.

[0037] Optionally, each piece of data to be translated described above can be text input by the user or the user's picked-up voice. The form of the translated data can be the same as or different from the data to be translated. For example, text can be translated into text, or text can be translated into voice with the corresponding translated content. Text can be translated into voice, or voice can be translated into text with the corresponding translated content. The above display can be playing the voice through a speaker or displaying the text in the screen. The text and the user's voice are based on natural language, and the translation by the large language model is also a translation based on natural language.

[0038] Optionally, when the large language model is configured on the cloud server, the cloud server can also perform operations such as the above language judgment and determining whether to directly send the data to be translated according to the judgment result.

[0039] Optionally, in other embodiments of the present application, the main intelligent terminal 110 includes: a main intelligent wearable device 111 and / or a main intelligent mobile terminal 112, and the slave intelligent terminal 120 includes: a slave intelligent wearable device 121 and / or a slave intelligent mobile terminal 122; the first large language model and the second large language model include: a generative artificial intelligence large language model (Generative Artificial Intelligence Large Language Model, GAILLM) or a multimodal large language model (Multimodal Large Language Model, MLLM).

[0040] Among them, in this embodiment, each intelligent wearable device can but is not limited to include: an intelligent safety helmet, intelligent earphones, intelligent earrings, a smart watch, smart glasses, and other wearable intelligent devices. Each intelligent mobile terminal can but is not limited to include: a cellular phone, a smart phone, other wireless communication devices, a personal digital assistant, an audio player, other media players, a music recorder, a video recorder, a camera, other media recorders, a smart radio, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) player, a digital camera, and other intelligent devices that can process data while moving. An Android, iOS, or other operating system is also installed on the intelligent mobile terminal.

[0041] The generative artificial intelligence large language model can be, for example but not limited to: ChatGPT of OpenAI, Bard of Google, and other models with similar functions. The multimodal large language model can be, for example but not limited to: BLIP-2, LLaVA, MiniGPT-4, mPLUG-Owl, LLaMA-Adapter-v2, Otter, Multimodal-GPT, InstructBLIP, VisualGLM-6B, PandaGPT, LaVIN, and other models with similar functions.

[0042] The above first large language model and second large language model can be the same model, or two same models located on different servers respectively, or two different large language models. For example, both the first large language model and the second large language model can be GAILLM or both can be MLLM, or alternatively, one can be GAILLM and the other can be MLLM.

[0043] Optionally, as Figure 2 shown, in other embodiments of the present application, the main intelligent terminal 110 includes a main intelligent wearable device 111 and a main intelligent mobile terminal 112, the system 100 further includes a management server 130, and the first data to be translated includes text or voice from the first user;

[0044] The main intelligent wearable device 111 is configured to obtain the first data to be translated and send the first data to be translated to the main intelligent mobile terminal 112.

[0045] The main intelligent mobile terminal 112 is configured to send the first data to be translated to the management server 130.

[0046] The management server 130 is configured to: generate the first translation prompt; use a speech-to-text engine to convert the voice in the first data to be translated into a first text to be translated, where the speech-to-text engine is configured in the management server 130 or a speech-to-text server; through the first large language model, according to the first translation prompt, translate the first text to be translated or the text in the first data to be translated into at least one first text data, where the first large language model is configured in the management server 130 or a model server; use a text-to-speech engine to convert the at least one first text data into at least one first voice data, where the text-to-speech engine is configured in the management server 130 or a text-to-speech server; and distribute the at least one first text data and / or the at least one first voice data as the at least one first data to their respective corresponding slave intelligent terminals 120 for display.

[0047] Optionally, in other embodiments of the present application, after the second data is displayed, the main intelligent wearable device 111 is further configured to obtain third data to be translated and send the third data to be translated to the main intelligent mobile terminal 112. The third data to be translated includes: voice or text from the first user.

[0048] The main intelligent mobile terminal 112 is further configured to determine at least one first target language according to the conversation mode, determine at least one first target terminal from the plurality of slave intelligent terminals, and send the information of the at least one first target language, the information of the at least one first target terminal, and the third data to be translated to the management server 130.

[0049] The management server 130 is further configured to: use the speech-to-text engine to convert the voice in the third data to be translated into a third text to be translated; generate a third translation prompt according to the information of the at least one first target language; use the first large language model to translate the third text to be translated or the text in the third data to be translated into at least one third text data according to the third translation prompt; use the text-to-speech engine to convert the at least one third text data into at least one third voice data, and distribute the at least one third text data and / or the at least one third voice data to the at least one first target terminal.

[0050] It can be understood that if the data to be translated is text, the management server 130 can directly translate the text without performing the operation of converting voice to text.

[0051] Optionally, in other embodiments of the present application, the conversation mode includes: private chat mode, group mode, and sharing mode. The main intelligent mobile terminal 112 is further configured to:

[0052] When the conversation mode is the private chat mode, determine the language of the second user as the first target language, and determine the slave intelligent terminal of the second user as the first target terminal;

[0053] When the conversation mode is the group mode, determine at least one language corresponding to the group associated with the second user as the first target language, and determine each slave intelligent terminal in the group as the first target terminal;

[0054] When the conversation mode is the sharing mode, determine the languages of the users of all slave intelligent terminals as the first target languages, and determine all slave intelligent terminals as the target slave intelligent terminals.

[0055] For example, if the user 1 of the main intelligent terminal 110 uses Chinese, the user 2 of the slave intelligent terminal 120A uses English, the user 3 of the slave intelligent terminal 120B uses Japanese, and the user 4 of the slave intelligent terminal 120C uses English. Among them, the user 2 of the slave intelligent terminal 120A and the user 4 of the slave intelligent terminal 120C are members of the same group. After the main intelligent terminal 110 plays the translated voice of the user 2 from the slave intelligent terminal 120A, it obtains the voice from the user 1 as the third data to be translated. If the current conversation mode is the private chat mode, the main intelligent terminal 110 translates the third data to be translated into the third voice data in English and sends it to the slave intelligent terminal 120A of the user 2. If the current conversation mode is the group mode, the main intelligent terminal 110 translates the third data to be translated into the third voice data in English and sends it to the slave intelligent terminal 120A of the user 2 and the slave intelligent terminal 120C of the user 4. If the current conversation mode is the sharing mode, the main intelligent terminal 110 translates the third data to be translated into the third voice data in English and the third voice data in Japanese respectively, sends the third voice data in English to the slave intelligent terminal 120A of the user 2 and the slave intelligent terminal 120C of the user 4, and sends the third voice data in Japanese to the slave intelligent terminal 120B of the user 3.

[0056] Optionally, a mobile application (APP) or a virtual assistant program can be installed on the main intelligent terminal 110 and the slave intelligent terminals 120. The user can switch different conversation modes through the interactive interface of the mobile APP for configuring the conversation mode. Alternatively, the main intelligent terminal 110 or the slave intelligent terminals 120 can also switch different conversation modes according to the user voice commands obtained by the virtual assistant program.

[0057] Optionally, the user can also select the form of the translated data through the mobile APP, such as translating text into voice, or translating text into text, or translating voice into text, or translating voice into voice, or translating voice into text and voice.

[0058] Each intelligent terminal can report the configuration information corresponding to the operations of the user on the mobile APP or the user voice commands issued through the virtual assistant program (such as the determined conversation mode, the selected form of the translated data, etc.) to the management server 130 for the management server 130 to use for subsequent translation.

[0059] Optionally, as Figure 2As shown, in other embodiments of the present application, the slave intelligent terminal 120 includes: a slave intelligent wearable device 121 and a slave intelligent mobile terminal 122. A part of the slave intelligent mobile terminal 122 is associated with the slave intelligent wearable device 121. The second data to be translated includes text or voice from the second user. A part of the slave intelligent mobile terminal 122 is associated with the slave intelligent wearable device 121. That is to say, the slave intelligent mobile terminal 122 is not in one-to-one correspondence with the slave intelligent wearable device 121. Some users may only use the slave intelligent mobile terminal 122 (in this case, the slave intelligent mobile terminal 122 can have the functions of the above-mentioned slave intelligent wearable device 121), and some users may use both the slave intelligent wearable device 121 and the slave intelligent mobile terminal 122. In other embodiments, some users may only use the slave intelligent wearable device 121, and at this time, the functions of the slave intelligent mobile terminal 122 are realized by the slave intelligent wearable device 121 or the management server 130.

[0060] The slave intelligent wearable device 121 is used to obtain the second data to be translated and send the second data to be translated to the associated slave intelligent mobile terminal.

[0061] The slave intelligent mobile terminal 122 is used to send the second data to be translated from the associated slave intelligent wearable device to the management server 130.

[0062] The management server 130 is further used to: use the speech-to-text engine to convert the speech in the second data to be translated into a second text to be translated; generate the second translation prompt; through the second large language model, according to the second translation prompt, translate the second text to be translated or the text in the second data to be translated into second text data, and the second large language model is configured in the management server 130 or the model server; use the text-to-speech engine to convert the second text data into second speech data; and send the second text data and / or the second speech data as the second data to the master intelligent wearable device 111 for display, or send the second text data and / or the second speech data as the second data to the master intelligent mobile terminal 112 to be forwarded by the master intelligent mobile terminal 112 to the master intelligent wearable device 111 for display.

[0063] Optionally, in other embodiments of the present application, the management server 130 is further used to distribute the at least one first data to at least one corresponding slave intelligent wearable device and / or corresponding slave intelligent mobile terminal. The slave intelligent mobile terminal 122 is further used to display the received first data, or send the speech data in the received first data to the associated slave intelligent wearable device for display.

[0064] Further, in other embodiments of the present application, the management server 130 is further configured to determine at least one corresponding slave smart wearable device and / or corresponding slave smart mobile terminal, and at least one target slave smart wearable device and / or target slave smart mobile terminal according to a preset language relationship mapping table, wherein the language relationship mapping table includes the languages corresponding to the master smart terminal and each slave smart terminal respectively, the languages corresponding to the at least one corresponding slave smart wearable device and / or corresponding slave smart mobile terminal are different from the language corresponding to the master smart terminal, and the languages corresponding to the at least one target slave smart wearable device and / or target slave smart mobile terminal are the same as the language corresponding to the master smart terminal.

[0065] The management server 130 is further configured to distribute the at least one first data to the at least one corresponding slave smart wearable device and / or corresponding slave smart mobile terminal, and distribute the first data to be translated to the at least one target slave smart wearable device and / or target slave smart mobile terminal.

[0066] The slave smart mobile terminal 120 is further configured to display the received translated data (such as: the first data or the third data) or the data to be translated (such as the first data to be translated or the third data to be translated), or send the voice data or the data to be translated in the received translated data to the associated slave smart wearable device for playback.

[0067] The management server 130 is further configured to: determine whether the language corresponding to the slave smart wearable device is the same as the language corresponding to the master smart wearable device according to the language relationship mapping table; when the language corresponding to the slave smart wearable device is the same as the language corresponding to the master smart wearable device, send the second data to be translated to the master smart wearable device for display, or send the second data to be translated to the master smart mobile terminal for forwarding to the master smart wearable device for display by the master smart mobile terminal; when the language corresponding to the slave smart wearable device is different from the language corresponding to the master smart wearable device, perform the above operation of converting the voice in the second data to be translated into the second text to be translated by using the voice-to-text engine and subsequent operations.

[0068] Specifically, a language relationship mapping table is preset in the management server 130. The information stored in the language relationship mapping table includes: the identification information and corresponding languages of the main intelligent terminal and each slave intelligent terminal, as well as the identity marks corresponding to each terminal (for example: the identity mark of the main intelligent terminal can be 1, and the identity mark of the slave intelligent terminal can be 0. This is only an example, and it is not limited to this in actual applications). The language corresponding to the main intelligent terminal is the language used by the user of the main intelligent terminal, and the languages corresponding to each slave intelligent terminal are the languages used by the users of each slave intelligent terminal. The identification information of the main intelligent terminal can be the device identification information of the main intelligent terminal or the nickname of the user of the preset main intelligent terminal, and the identification information of each slave intelligent terminal can be the device identification information of each slave intelligent terminal or the nickname of the preset each slave intelligent terminal.

[0069] An application program (APP) or a virtual assistant program can be installed on the main intelligent terminal and each slave intelligent terminal. When the user sets the languages corresponding to the main intelligent terminal and each slave intelligent terminal through the APP or the virtual assistant program, the main intelligent terminal and each slave intelligent terminal send their respective identification information and the languages set by the user to the management server 130. Alternatively, the main intelligent terminal and each slave intelligent terminal can also send their respective identification information and the preset corresponding languages to the management server 130 when joining a translation group (or session group), so that the management server 130 can set the languages corresponding to the main intelligent terminal and each slave intelligent terminal in the language relationship mapping table.

[0070] Furthermore, the nickname can also be set by the user through the above APP or virtual assistant program and reported to the management server 130.

[0071] Before each translation operation is performed by the large language model (regardless of which dialogue mode or working mode), the management server 130 can determine whether the received data to be translated needs to be translated and into which languages according to the language relationship mapping table, and select to send the data to be translated or the translated data to the corresponding terminal according to the determination result.

[0072] Optionally, in other embodiments of the present application, the main intelligent terminal 110 includes the main intelligent wearable device or the main intelligent mobile terminal 112. The system 100 further includes a management server 130. The first data to be translated includes text or voice from the first user.

[0073] The main intelligent terminal 110 is configured to obtain the first data to be translated and send the first data to be translated to the management server.

[0074] The management server 130 is configured to: generate the first translation prompt; use a speech-to-text engine to convert the speech in the first data to be translated into a first text to be translated, where the speech-to-text engine is configured in the management server 130 or a speech-to-text server; through the first large language model, according to the first translation prompt, translate the first text to be translated or the text in the first data to be translated into at least one first text data, where the first large language model is configured in the management server 130 or a model server; use a text-to-speech engine to convert the at least one first text data into the at least one first speech data, where the text-to-speech engine is configured in the management server 130 or a text-to-speech server; and distribute the at least one first text data and / or the at least one first speech data as the at least one first data to their respective corresponding slave intelligent terminals for display.

[0075] Optionally, in other embodiments of the present application, after the second data is displayed, the master intelligent terminal 110 is further configured to: obtain third data to be translated, where the third data to be translated includes: speech or text from the first user; and according to the conversation mode, determine at least one first target language and determine at least one first target terminal from multiple slave intelligent terminals 120, and send the information of the at least one first target language and the at least one first target terminal and the third data to be translated to the management server 130.

[0076] The management server 130 is further configured to: use the speech-to-text engine to convert the speech in the third data to be translated into a third text to be translated; generate a third translation prompt according to the information of the at least one first target language; through the first large language model, according to the third translation prompt, translate the third text to be translated or the text in the third data to be translated into at least one third text data; and use the text-to-speech engine to convert the at least one third text data into at least one third speech data, and distribute the at least one third text data and / or the at least one third speech data to the at least one first target terminal.

[0077] Optionally, in other embodiments of the present application, the slave intelligent terminal 120 includes: a slave smart wearable device 121 or a slave smart mobile terminal 122, and a part of the slave smart mobile terminal 122 is associated with the slave smart wearable device 121, and the second data to be translated includes text or speech from the second user.

[0078] The slave intelligent terminal 120 is configured to obtain the second data to be translated and send the second data to be translated to the management server 130.

[0079] The management server 130 is further configured to: utilize the speech-to-text engine to convert the speech in the second data to be translated into a second text to be translated; generate the second translation prompt; through the second large language model, according to the second translation prompt, translate the second text to be translated or the text in the second data to be translated into second text data, where the second large language model is configured in the management server 130 or the model server; and utilize the text-to-speech engine to convert the second text data into second speech data, and send the second text data and / or the second speech data as the second data to the main intelligent terminal 110 for display.

[0080] Optionally, in other embodiments of the present application, the system 100 further includes a management server 130, and the second large language model is configured in the management server 130.

[0081] The slave intelligent mobile terminal 122 is further configured to, in response to a first switching instruction, switch the working mode to a conference mode, and in the conference mode, determine at least one second target language according to at least one second target terminal pointed to by the user's selection operation, and send the information of the at least one second target terminal and the at least one second target language and the second data to be translated to the management server 130.

[0082] The management server 130 is configured to generate the second translation prompt according to the information of the at least one second target language, and through the second large language model, according to the second translation prompt, translate the second data to be translated into at least one second data corresponding to the at least one second target language, and distribute the at least one second data to the at least one second target terminal for display according to the information of the at least one second target terminal.

[0083] The slave intelligent mobile terminal 122 is further configured to, in response to a second switching instruction, switch the working mode to a tour guide mode, and in the tour guide mode, send the second data to be translated and the language information of the first user to the management server 130.

[0084] The management server 130 is further configured to generate the second translation prompt according to the language information of the first user, and through the second large language model, according to the second translation prompt, translate the second data to be translated into second data corresponding to the language of the first user, and send the second data to the main intelligent terminal 110 for display.

[0085] Optionally, the user can also select, through the mobile APP on the main intelligent terminal 110 or the slave intelligent terminal 120, to set the working mode to the tour guide mode or the conference mode, and configure the identities of the main intelligent terminal 110 and the slave intelligent terminal 120 in each mode. Further, in the conference mode, the user can also select, through the mobile APP, to whom the data to be translated is to be played or shown.

[0086] Optionally, the main intelligent terminal 110 can also initiate a session, create a session group, and generate a shared link for joining the session or generate a QR code containing the shared link according to the operations of the user on the interaction interface of the mobile APP or the voice commands issued by the user through the virtual assistant program. Alternatively, when the main intelligent terminal 110 is a smart glasses, the user can also initiate a session by pressing a physical or virtual button on the smart glasses for initiating a session. Among them, the session can be a conference-based session or a tour guide-based session. The user can make a selection on the interaction interface of the APP, or specify which type of session through a voice command, or make a selection by pressing a selection button on the smart glasses.

[0087] Further, the main intelligent terminal 110 can initiate the session through the session server. At this time, the shared link or the QR code can also be generated by the session server.

[0088] The slave intelligent terminal 120 (such as a smart glasses or a smart phone as a slave) can scan the QR code or open the shared link through a Web application running on a browser to join the session.

[0089] Further, when the user of the main intelligent terminal 110 performs a preset operation to initiate a session on the interaction interface of the APP (for example, clicks the button on the interaction interface for initiating a session), a prompt message will be displayed on the interaction interface to prompt the user to select the source language and / or the target language before the session. If the user makes a selection, the user's selection will be saved for subsequent translation operations. If the user does not make a selection, language automatic detection will be enabled. For example, if the user does not select the source language, the language of the user's speech can be detected through a speech-to-text engine, and the detected language will be used as the source language. Further, if the user of the main intelligent terminal 110 does not select the target language, the main intelligent terminal 110 or the management server 130 can request information about the language used by the user of the slave intelligent terminal 120 from the slave intelligent terminal 120 (the slave intelligent terminal 120 can reply to the main intelligent terminal 110 or the management server 130 with the source language preset by the user of the slave intelligent terminal 120 on the APP of the slave intelligent terminal 120 or the system language of the slave intelligent terminal 120), and use the language returned by the slave intelligent terminal 120 as the target language. Further, if the slave intelligent terminal 120 does not return information about the language used by the user of the slave intelligent terminal 120, a preset default language, such as English, can also be used as the target language.

[0090] Optionally, an APP or a virtual assistant program can also be installed on the slave intelligent terminal 120 (such as an intelligent glasses or a smart phone serving as a slave device). The user of the main intelligent terminal 110 or the slave intelligent terminal 120 can also trigger the main intelligent terminal 110 or the slave intelligent terminal 120 to start picking up the user's speech by pressing a virtual button on the terminal (such as a virtual button based on a touch sensor on the temple of the intelligent glasses) or a virtual button for speaking on the interaction interface of the APP or by sending a voice command similar to "I'm about to speak" through the virtual assistant program. When the user releases the virtual button or the microphone is idle for more than a preset duration, the picking up of the user's speech will stop. Alternatively, the main intelligent terminal 110 or the slave intelligent terminal 120 can also detect the time points when the user starts speaking and stops speaking through voice activity detection (VAD).

[0091] Preferably, in this embodiment, the main intelligent terminal 110 and the slave intelligent terminal 120 can perform the picking up and translation operations of the user's speech to be translated synchronously through multiple threads, thereby reducing the translation delay.

[0092] For the details of the multi-party cross-lingual interaction system based on the large language model in this embodiment that are not described in detail, reference can also be made to the relevant descriptions in the following Figures 3 to 6 illustrated embodiments, which will not be elaborated here.

[0093] In this embodiment, by combining multiple intelligent terminals with a large language model, multi-party cross-language interaction based on the large language model with multi-intelligent terminal collaboration is achieved, thereby improving the practicability, interactivity, and intelligence of the intelligent terminals, as well as increasing product stickiness. Moreover, due to the scalability and self-creativity of the large language model, the accuracy of translation can be further improved.

[0094] See Figure 3 , the internal structural schematic diagram of an intelligent terminal based on a large language model provided by an embodiment of the present application. As Figure 3 shown, the intelligent terminal 300 includes: an input device 301, a processor 302, a wireless communication component 303, and a memory 304, wherein the processor 304 is electrically connected to the input device 301, the wireless communication component 303, and the memory 304;

[0095] One or more programs executable by the processor 302 are stored in the memory 304, and the one or more programs include multiple instructions for:

[0096] In response to a first configuration instruction, configure the intelligent terminal 300 as a host;

[0097] When the intelligent terminal 300 serves as the host, obtain first data to be translated through the input device 301 and send it to the cloud server through the wireless communication component 303, so that the cloud server uses the large language model in the cloud server to translate the first data to be translated into at least one first data according to a first translation hint and distribute it to at least one slave intelligent terminal, wherein the first data to be translated includes a first text to be translated or a first voice from a user, and the language of each first data corresponds to the language used by the user of each slave intelligent terminal;

[0098] In response to a second configuration instruction, configure the intelligent terminal 300 as a slave;

[0099] When the intelligent terminal 300 serves as the slave, obtain second data to be translated through the input device 301 and send it to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into a second data according to a second translation hint and send it to the master intelligent terminal, wherein the second data to be translated includes a second text to be translated or a second voice from the user, and the language of the second data corresponds to the language used by the user of the master intelligent terminal.

[0100] Optionally, in other embodiments of the present application, the intelligent terminal 300 is an intelligent mobile terminal or an intelligent wearable device.

[0101] Optionally, in other embodiments of the present application, the plurality of instructions are further configured to: determine whether the language used by at least one user of the plurality of slave intelligent terminals is different from the language used by the user; if not, distribute the first data to be translated to each of the slave intelligent terminals for display; if so, send the first data to be translated to at least one first terminal among the plurality of slave intelligent terminals for display, and send the information of the language used by the user of at least one second terminal among the plurality of slave intelligent terminals and the first data to be translated to the cloud server, so that the cloud server uses the large language model to translate the first data to be translated into the at least one first data according to the first translation hint and the information of the language used by the user of the at least one second terminal, and distribute the at least one first data to the corresponding second terminals, where the language of each of the first data corresponds to the language used by the user of each of the second terminals, the language used by the user of the first terminal is the same as the language used by the user, and the language used by the user of the second terminal is different from the language used by the user.

[0102] Optionally, in other embodiments of the present application, the plurality of instructions are further configured to: determine whether the language used by the user is the same as the language used by the user of the master intelligent terminal; if the same, send the second data to be translated to the master intelligent terminal for display; if different, send the second data to be translated to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into a second data according to the second translation hint and send it to the master intelligent terminal.

[0103] Optionally, in other embodiments of the present application, an application program is configured on the intelligent terminal 300, and the plurality of instructions are further configured to: through the application program, in response to an initiation instruction, create a meeting through a conference server and configure the intelligent wearable device as the host; and through the conference server, according to a first access request, add the terminal that sends the first access request as a slave intelligent terminal to the meeting.

[0104] The plurality of instructions are further configured to: after configuring the intelligent terminal 300 as a slave, send a second access request to the conference server according to a preset shared link or the shared link obtained by scanning a QR code to join the meeting initiated by the master intelligent terminal.

[0105] Preferably, the intelligent terminal 300 is an intelligent wearable device.

[0106] Optionally, as Figure 4As shown, in other embodiments of the present application, the intelligent terminal 300 further includes a Bluetooth component 405 electrically connected to the processor 302. The plurality of instructions are further configured to: send the first data to be translated to the intelligent mobile terminal through the Bluetooth component 405, so that the intelligent mobile terminal sends the first data to be translated to the cloud server; and send the second data to be translated to the intelligent mobile terminal through the Bluetooth component 405, so that the intelligent mobile terminal sends the second data to be translated to the cloud server.

[0107] Optionally, as Figure 4 shown, in other embodiments of the present application, the intelligent terminal 300 further includes a speaker 406 electrically connected to the processor 302. The plurality of instructions are further configured to:

[0108] After configuring the intelligent terminal 300 as the host, receive the voice sent by the cloud server and play it through the speaker 406;

[0109] Obtain third data to be translated through the input device 301, and determine at least one first target language according to the conversation mode and at least one first target terminal from the associated plurality of slave intelligent terminals. The third data to be translated includes third voice to be translated or third text to be translated from the user;

[0110] Send the information of the at least one first target language, the at least one first target terminal, and the third data to be translated to the cloud server through the wireless communication component 303, so that the cloud server translates the third data to be translated into at least one third data according to the third translation hint and the information through the large language model and distributes it to the at least one first target terminal for display.

[0111] Optionally, in other embodiments of the present application, the plurality of instructions are further configured to:

[0112] After configuring the intelligent terminal 300 as a slave, in response to a first switching instruction, switch the working mode to a conference mode, and in the conference mode, determine at least one second target language according to at least one second target terminal pointed to by the user's selection operation, and send the information of the at least one second target terminal, the at least one second target language, and the second data to be translated to the cloud server, so that the cloud server translates the second data to be translated into at least one second data corresponding to the at least one second target language according to the information and the second translation hint through the large language model and distributes it to the at least one second target terminal for display;

[0113] In response to the second switching instruction, switch the working mode to the tour guide mode. In this tour guide mode, send the second data to be translated and the language information of the user of the main intelligent terminal to the cloud server, so that the cloud server, based on the language information of the user of the main intelligent terminal and the second translation hint, translates the second data to be translated into the second data corresponding to the language of the user of the main intelligent terminal through the large language model, and sends the second data to the main intelligent terminal for display.

[0114] For the details of the intelligent terminal based on the large language model in this embodiment, reference can also be made to the above Figures 1 to 2 and the following Figure 5 and Figure 6 and the relevant descriptions in the embodiments shown. Details will not be repeated here.

[0115] In this embodiment, by combining multiple intelligent terminals with the large language model, multi-party cross-language interaction based on the large language model with multi-intelligent terminal collaboration is achieved, thereby improving the practicability, interactivity and intelligence of the intelligent terminal, and increasing product viscosity. Moreover, due to the scalability and self-creation of the large language model, the accuracy of translation can be further improved.

[0116] See Figure 5 , the flowchart of the multi-party cross-language interaction method based on the large language model provided by an embodiment of the present application. This method is applied to an intelligent mobile terminal, which may include but is not limited to: cellular phones, smart phones, other wireless communication devices, personal digital assistants, audio players, other media players, music recorders, video recorders, cameras, other media recorders, smart radios, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) players, digital cameras, smart wearable devices (such as smart watches, smart bracelets, etc.) and other intelligent devices that can process data while moving. An Android, IOS or other operating system is also installed on the intelligent mobile terminal.

[0117] As Figure 5 shown, this method includes:

[0118] S501. In response to the first configuration instruction, configure the intelligent mobile terminal as the host;

[0119] S502. As the host, obtain the first data to be translated and send it to the cloud server, so that the cloud server uses the large language model in the cloud server to translate the first data to be translated into at least one first data according to the first translation prompt and distribute it to at least one slave intelligent mobile terminal. The first data to be translated includes the first text to be translated or the first voice from the user, and the languages of the first data respectively correspond to the languages used by the users of the slave intelligent mobile terminals.

[0120] S503. In response to the second configuration instruction, configure the intelligent mobile terminal as a slave.

[0121] S504. As the slave, obtain the second data to be translated and send it to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into second data according to the second translation prompt and send it to the master intelligent mobile terminal. The second data to be translated includes the second text to be translated or the second voice from the user, and the language of the second data corresponds to the language used by the user of the master intelligent mobile terminal.

[0122] Specifically, a mobile application (APP) or a virtual assistant program is installed on the intelligent mobile terminal. The user can trigger the first configuration instruction or the second configuration instruction by operating on the interaction interface of the mobile APP or the virtual assistant program (for example: clicking on a virtual button preset on the interaction interface for configuring the intelligent mobile terminal as the host or the slave), or by issuing the first configuration instruction or the second configuration instruction by voice.

[0123] A language relationship mapping table is also configured on the intelligent mobile terminal. The information stored in the language relationship mapping table includes: the identification information of the intelligent mobile terminal as the host and the corresponding language, the identification information of each slave intelligent mobile terminal as the slave and the corresponding language, and the identity markers corresponding to each terminal (for example: the identity marker of the host can be 1, and the identity marker of the slave can be 0. This is only an example, and it is not limited to this in actual applications).

[0124] Optionally, in other embodiments of the present application, when the intelligent mobile terminal is used as the host, it can respond to the initiation instruction, initiate a session, create a session group, add the sender of the access request as a slave to the session group according to the access request, generate the language relationship mapping table, and synchronize the language relationship mapping table to all terminals in the session group. Further, the intelligent mobile terminal can also synchronize the language relationship mapping table to the cloud server. The session can be a session for a meeting or a session for a tour guide. The initiation instruction can be triggered by the user through a virtual button for initiating a session on the interaction interface of the mobile APP or the virtual assistant program, or by the user through voice.

[0125] Further, the intelligent mobile terminal can obtain the shared link therein by scanning the QR code, and send the access request according to the shared link. The QR code can be generated by the host initiating the session.

[0126] Further, the user can set the languages corresponding to the host and / or each slave through the language configuration menu on the interaction interface of the mobile APP or the virtual assistant program.

[0127] Further, the languages corresponding to each slave can also be reported to the host by each slave after access, and recorded by the host in the language relationship mapping table.

[0128] Further, when the intelligent mobile terminal initiates a session, the first configuration instruction can also be automatically triggered. When the intelligent mobile terminal sends the access request, the second configuration instruction can also be automatically triggered.

[0129] Further, the host can act as a hotspot and each slave accesses the host. Or, the host and each slave can be located in the same WIFI network. Or, the host and some slaves are located in the same WIFI network, and the remaining slaves join the session group through the cellular network, that is, the remaining slaves conduct a remote session with. Or, the host and each slave join the session group through the cellular network respectively.

[0130] The specific process of the cloud server using the large language model in the cloud server to perform translation according to the translation prompt can refer to the relevant content in the above Figures 1 to 4 illustrated embodiments and will not be elaborated here.

[0131] Optionally, in other embodiments of the present application, after configuring the intelligent mobile terminal as the host, the method further includes:

[0132] Receiving the voice sent by the cloud server and playing it;

[0133] Obtaining third data to be translated, and determining at least one first target language according to the conversation mode and determining at least one first target terminal from multiple associated slave intelligent terminals. The third data to be translated includes the third text to be translated or the third voice to be translated from the user;

[0134] Sending the at least one first target language, the information of the at least one first target terminal, and the third data to be translated to the cloud server, so that the cloud server, through the large language model, according to the third translation prompt and the information, translates the third data to be translated into at least one third data and distributes it to the at least one first target terminal for display.

[0135] Specifically, when the intelligent mobile terminal acts as a host, it also receives and plays the voice sent by the cloud server. The original voice comes from a user of a slave intelligent mobile terminal, and the cloud server translates the original voice and sends it to the intelligent mobile terminal.

[0136] Optionally, the conversation mode includes: private chat mode, group mode, and sharing mode. After obtaining the third data to be translated, when the conversation mode is the private chat mode, the intelligent mobile terminal determines the language of the target user corresponding to the played voice as the first target language, and determines the slave intelligent mobile terminal of the target user as the first target terminal; when the conversation mode is the group mode, determines at least one language corresponding to the group associated with the target user as the first target language, and determines each slave intelligent mobile terminal in the group as the first target terminal; when the conversation mode is the sharing mode, determines the languages of all users of the slave intelligent mobile terminals as the first target language, and determines all slave intelligent mobile terminals as the target slave intelligent mobile terminals.

[0137] Optionally, in other embodiments of the present application, the obtaining the second data to be translated and sending it to the cloud server includes:

[0138] In the meeting mode, determine at least one second target language according to at least one second target terminal pointed to by the user's selection operation, and send the information of the at least one second target terminal and the at least one second target language and the second data to be translated to the cloud server, so that the cloud server, according to the information and the second translation hint, through the large language model, translates the second data to be translated into at least one second data corresponding to the at least one second target language, and distributes it to the at least one second target terminal for display;

[0139] In the tour guide mode, send the second data to be translated and the language information of the user of the master intelligent terminal to the cloud server, so that the cloud server, according to the language information of the user of the master intelligent terminal and the second translation hint, through the large language model, translates the second data to be translated into the second data corresponding to the language of the user of the master intelligent terminal, and sends the second data to the master intelligent terminal for display.

[0140] Optionally, in other embodiments of the present application, after configuring the intelligent mobile terminal as a host in response to the first configuration instruction, the method further includes: determining whether there is at least one terminal among the multiple slave intelligent mobile terminals whose user uses a language different from the language used by the user.

[0141] Send the first data to be translated to the cloud server, including: If not, send the first data to be translated to the cloud server, and instruct the cloud server to distribute the first data to be translated to each slave intelligent mobile terminal for display; If so, send the first data to be translated to the cloud server, and instruct the cloud server to send the first data to be translated to at least one first terminal among the multiple slave intelligent mobile terminals for display. At the same time, use the large language model to translate the first data to be translated into the at least one first data according to the first translation prompt and distribute it to the second terminal among the corresponding multiple slave intelligent mobile terminals, where the language of each first data corresponds to the language used by the user of each second terminal, the language used by the user of the first terminal is the same as the language used by the user, and the language used by the user of the second terminal is different from the language used by the user.

[0142] Optionally, in other embodiments of the present application, after configuring the intelligent mobile terminal as a slave in response to the second configuration instruction, the method further includes: determining whether the language used by the user is the same as the language used by the user of the master intelligent mobile terminal.

[0143] Send the second data to be translated to the cloud server, including: If the same, send the second data to be translated to the cloud server, and instruct the cloud server to send the second data to be translated to the master intelligent mobile terminal for display; If different, send the second data to be translated to the cloud server, and instruct the cloud server to use the large language model to translate the second data to be translated into the second data according to the second translation prompt and send it to the master intelligent mobile terminal.

[0144] The following will be combined with Figure 6 , to illustrate the above method by way of example. As Figure 6 shown, in a practical application example, the tour guide initiates a conversation based on the tour guide through the APP in the master mobile phone A2, creates a tour guide conversation group, and generates a corresponding language relationship mapping table according to the identity identification information of the intelligent terminals joining the conversation. The tour guide conversation group may include the master intelligent glasses A1 and the master mobile phone A2 used by the tour guide, the slave intelligent glasses B1 and the slave mobile phone B2 used by the tourist X, the slave mobile phone C used by the tourist Y, and the slave intelligent glasses D used by the tourist Z. At the same time as initiating the conversation, under the guidance of the language setting prompt on the APP, the tour guide sets the language used by himself (e.g., language A), the language used by the tourist X (e.g., language B), the language used by the tourist Y (e.g., language C), and the language used by the tourist Z (e.g., language A) respectively.

[0145] When the tour guide is speaking, the main smart glasses A1 acquire the voice of the tour guide and send the acquired voice as the voice to be translated to the main mobile phone A2 via Bluetooth. The main mobile phone A2 sends the languages set by the tour guide for the tour guide himself / herself, tourist X, tourist Y, and tourist Z, the identity information of the slave mobile phone B2, the slave mobile phone C, and the slave smart glasses D, as well as the voice to be translated to the management server.

[0146] Alternatively, the main mobile phone A2 can also mark the corresponding languages in the language relationship mapping table while the tour guide is making language settings, and synchronize the language relationship mapping table to the management server for the management server to use it for subsequent translations.

[0147] The management server compares the language used by the tour guide with the languages used by tourist X, tourist Y, and tourist Z, and generates a translation prompt containing information on the source language (Language A) and the target languages (Language B and Language C) and indication information for indicating translation according to the comparison result. At the same time, since the language used by tourist Z is the same as the language used by the tour guide, the management server directly sends the voice to be translated to the slave smart glasses D for playback according to the identity information of the slave smart glasses D.

[0148] Furthermore, the management server converts the voice to be translated into text in Language A through the speech-to-text engine on the conversion server, and then sends the text in Language A and the translation prompt to the model server.

[0149] The model server translates the text in Language A into text in Language B and text in Language C according to the translation prompt through the large language model, and sends the text in Language B and the text in Language C to the management server.

[0150] The management server converts the text in Language B and the text in Language C into corresponding voices, namely the translated voice in Language B and the translated voice in Language C, through the text-to-speech engine in the conversion server. Then, according to the identity information of the slave mobile phone B2 and the slave mobile phone C, the management server sends the translated voice in Language B to the slave mobile phone B2 and the translated voice in Language C to the slave mobile phone C for playback. The slave mobile phone B2 sends the received translated voice in Language B to the slave smart glasses B1 via Bluetooth for playback.

[0151] Optionally, the management server can also send the text in Language B to the slave mobile phone B2 for display on the screen of the slave mobile phone B2 and send the text in Language C to the slave mobile phone C for display on the screen of the slave mobile phone C while sending the translated voice.

[0152] It can be understood that mobile phones or smart glasses as the host and slave can have various combinations to apply different scenarios. Taking a tour guide as an example, in Scenario 1, both the tour guide and tourists can use mobile phones; or, in Scenario 2, both the tour guide and tourists can use smart glasses; or, in Scenario 3, both the tour guide and tourists can use mobile phones and smart glasses simultaneously; or, in Scenario 4, the tour guide can use a mobile phone or smart glasses, while all tourists can use mobile phones and smart glasses simultaneously; or, in Scenario 5, the tour guide can use a mobile phone or smart glasses, while some tourists can use mobile phones, some can use smart glasses, and some can use both mobile phones and smart glasses simultaneously; or, in Scenario 6, the tour guide can use both a mobile phone and smart glasses, and some tourists can use mobile phones and some can use smart glasses.

[0153] Moreover, in each of the above scenarios, even in the same scenario, the data to be translated by each party can include only text, only speech, or any combination of text and speech. For example: the data to be translated by the host is text, and the data to be translated by the slave is speech. Another example is that the data to be translated by any one or both of the host or slave can be speech at the current moment and text at the next moment to meet the requirements of different translation occasions, such as occasions where some words are not convenient to say in public.

[0154] Furthermore, when playing speech, the speaker corresponding to the currently played speech can be displayed on the mobile phone screen simultaneously.

[0155] For the details not covered in the multi-party cross-lingual interaction method based on the large language model in this embodiment, reference can also be made to Figures 1 to 4 the relevant descriptions in the illustrated embodiments, which will not be elaborated here.

[0156] In this embodiment, by combining multiple intelligent terminals with the large language model, multi-intelligent terminal collaboration-based multi-party cross-lingual interaction based on the large language model is achieved, thereby improving the practicability, interactivity, and intelligence of intelligent terminals, as well as increasing product viscosity. And due to the scalability and self-creativity of the large language model, the accuracy of translation can be further improved.

[0157] This application embodiment also provides a non-transitory computer-readable storage medium, which can be set in the smart glasses or smart wearable devices in the above embodiments. The non-transitory computer-readable storage medium can be the aforementioned Figure 3 or Figure 4The memory 304 in the illustrated embodiment. A computer program is stored on the computer-readable storage medium, and when the program is executed by a processor, it implements the multi-party cross-language interaction method based on the large language model described in the foregoing embodiments. Further, the computer-readable storage medium may also be various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disc that can store program codes.

[0158] In several embodiments provided in the present application, it should be understood that the disclosed multi-party cross-language interaction method, system, and intelligent terminal based on the large language model can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed direct connection or communication connection between each other can be an indirect connection or communication connection through some interfaces, devices, or modules, and can be in an electrical, mechanical, or other form.

[0159] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0160] In the above embodiments, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0161] The above is the description of the multi-party cross-language interaction method, system, and intelligent terminal based on the large language model provided in the present application. For those skilled in the art, according to the idea of the embodiments of the present application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A multi-party cross-lingual interaction system based on large language models, characterized in that, Including: A main intelligent terminal and multiple slave intelligent terminals; The main intelligent terminal is configured to obtain first to-be-translated data of a first user, and translate the first to-be-translated data into at least one first data according to a first translation prompt by using a first large language model, and distribute the first data to corresponding slave intelligent terminals for display. Among them, the languages of the first data respectively correspond to the languages used by the users of the corresponding slave intelligent terminals. The first large language model is configured in the main intelligent terminal or a cloud server; The slave intelligent terminal is configured to obtain second to-be-translated data of a second user, and translate the second to-be-translated data into second data according to a second translation prompt by using a second large language model, and send the second data to the main intelligent terminal for display. Among them, the language of the second data is the language used by the user of the main intelligent terminal. The second large language model is configured in the slave intelligent terminal or the cloud server.

2. The system according to claim 1, wherein: The main intelligent terminal is further configured to: determine whether the languages used by the users of at least one of the multiple slave intelligent terminals are different from the language used by the first user; if not, distribute the first to-be-translated data to each of the slave intelligent terminals for display; if so, send the first to-be-translated data to at least one first terminal among the multiple slave intelligent terminals for display, and based on at least one second terminal among the multiple slave intelligent terminals, perform the operation of translating the first to-be-translated data into at least one first data according to the first translation prompt and distributing the first data to corresponding slave intelligent terminals for display. Among them, the languages of the first data respectively correspond to the languages used by the users of the second terminals. The language used by the user of the first terminal is the same as the language used by the first user, and the language used by the user of the second terminal is different from the language used by the first user; The slave intelligent terminal is further configured to: determine whether the language used by the second user is the same as the language used by the first user; if the same, send the second to-be-translated data to the main intelligent terminal for display; if different, perform the operation of translating the second to-be-translated data into second data according to the second translation prompt by using the second large language model and sending the second data to the main intelligent terminal for display.

3. The system according to claim 1, characterized in that, The main intelligent terminal includes: a main intelligent wearable device and / or a main intelligent mobile terminal. The slave intelligent terminal includes: a slave intelligent wearable device and / or a slave intelligent mobile terminal. The first large language model and the second large language model include: a generative artificial intelligence large language model and / or a multi-modal large language model.

4. The system according to claim 3, wherein The main intelligent terminal includes the main intelligent wearable device and the main intelligent mobile terminal. The system further includes a management server. The first to-be-translated data includes text or voice from the first user; The main intelligent wearable device is configured to obtain the first to-be-translated data and send the first to-be-translated data to the main intelligent mobile terminal; The main intelligent mobile terminal is used to send the first data to be translated to the management server; The management server is used for: Generating the first translation prompt; Using a speech-to-text engine to convert the speech in the first data to be translated into a first text to be translated, where the speech-to-text engine is configured in the management server or the speech-to-text server; Through the first large language model, according to the first translation prompt, translating the text in the first text to be translated or the first data to be translated into at least one first text data, where the first large language model is configured in the management server or the model server; Using a text-to-speech engine to convert the at least one first text data into at least one first speech data, where the text-to-speech engine is configured in the management server or the text-to-speech server; And Distributing the at least one first text data and / or the at least one first speech data as the at least one first data to the corresponding slave intelligent terminals for display.

5. The system according to claim 4, wherein After the second data is displayed, The main intelligent wearable device is further used to obtain third data to be translated and send the third data to be translated to the main intelligent mobile terminal, where the third data to be translated includes: speech or text from the first user; The main intelligent mobile terminal is further used to determine at least one first target language according to the conversation mode and determine at least one first target terminal from the multiple slave intelligent terminals, and send the information of the at least one first target language and the at least one first target terminal and the third data to be translated to the management server; The management server is further used for: Using the speech-to-text engine to convert the speech in the third data to be translated into a third text to be translated; Generating a third translation prompt according to the information of the at least one first target language; Through the first large language model, according to the third translation prompt, translating the text in the third text to be translated or the third data to be translated into at least one third text data; Using the text-to-speech engine to convert the at least one third text data into at least one third speech data and distributing the at least one third text data and / or the at least one third speech data to the at least one first target terminal.

6. The system according to claim 5, wherein The conversation mode includes: private chat mode, group mode and sharing mode. The main intelligent mobile terminal is further used for: When the conversation mode is the private chat mode, determining the language of the second user as the first target language and determining the slave intelligent terminal of the second user as the first target terminal; When the conversation mode is the group mode, determining at least one language corresponding to the group associated with the second user as the first target language and determining each slave intelligent terminal in the group as the first target terminal; When the conversation mode is the sharing mode, determining the languages of all users of the slave intelligent terminals as the first target language and determining all slave intelligent terminals as the target slave intelligent terminals.

7. The system according to claim 4, characterized in that The slave intelligent terminal includes: the slave intelligent wearable device and the slave intelligent mobile terminal, a part of the slave intelligent mobile terminal is associated with the slave intelligent wearable device, and the second data to be translated includes text or voice from the second user; The slave intelligent wearable device is configured to obtain the second data to be translated and send the second data to be translated to the associated slave intelligent mobile terminal; The slave intelligent mobile terminal is configured to send the second data to be translated from the associated slave intelligent wearable device to the management server; The management server is further configured to: Use the speech-to-text engine to convert the voice in the second data to be translated into a second text to be translated; Generate the second translation prompt; Through the second large language model, according to the second translation prompt, translate the second text to be translated or the text in the second data to be translated into second text data, where the second large language model is configured in the management server or the model server; Use the text-to-speech engine to convert the second text data into second voice data; and Send the second text data and / or the second voice data as the second data to the master intelligent wearable device for display, or send the second text data and / or the second voice data as the second data to the master intelligent mobile terminal for forwarding to the master intelligent wearable device by the master intelligent mobile terminal for display.

8. The system according to claim 7, wherein The management server is further configured to determine at least one corresponding slave intelligent wearable device and / or corresponding slave intelligent mobile terminal, and at least one target slave intelligent wearable device and / or target slave intelligent mobile terminal according to a preset language relationship mapping table, wherein the language relationship mapping table includes the languages corresponding to the master intelligent terminal and each slave intelligent terminal, the languages corresponding to the at least one corresponding slave intelligent wearable device and / or corresponding slave intelligent mobile terminal are different from the language corresponding to the master intelligent terminal, and the languages corresponding to the at least one target slave intelligent wearable device and / or target slave intelligent mobile terminal are the same as the language corresponding to the master intelligent terminal; The management server is further configured to distribute the at least one first data to the at least one corresponding slave intelligent wearable device and / or corresponding slave intelligent mobile terminal, and distribute the first data to be translated to the at least one target slave intelligent wearable device and / or target slave intelligent mobile terminal; The slave intelligent mobile terminal is further configured to display the received translated data or data to be translated, or send the voice data or data to be translated in the received translated data to the associated slave intelligent wearable device for playback; The management server is further configured to: determine whether the language corresponding to the slave intelligent wearable device is the same as the language corresponding to the master intelligent wearable device according to the language relationship mapping table; when the language corresponding to the slave intelligent wearable device is the same as the language corresponding to the master intelligent wearable device, send the second data to be translated to the master intelligent wearable device for display, or send the second data to be translated to the master intelligent mobile terminal for forwarding to the master intelligent wearable device for display by the master intelligent mobile terminal; when the language corresponding to the slave intelligent wearable device is different from the language corresponding to the master intelligent wearable device, perform the operation of converting the voice in the second data to be translated into a second text to be translated by using the voice-to-text engine.

9. The system according to claim 3, characterized in that The master intelligent terminal includes the master intelligent wearable device or the master intelligent mobile terminal, the system further includes a management server, and the first data to be translated includes text or voice from the first user. The master intelligent terminal is configured to obtain the first data to be translated and send the first data to be translated to the management server. The management server is configured to: Generate the first translation prompt. Use a voice-to-text engine to convert the voice in the first data to be translated into a first text to be translated, where the voice-to-text engine is configured in the management server or the voice-to-text server. Translate the first text to be translated or the text in the first data to be translated into at least one first text data according to the first translation prompt through the first large language model, where the first large language model is configured in the management server or the model server. Use a text-to-voice engine to convert the at least one first text data into at least one first voice data, where the text-to-voice engine is configured in the management server or the text-to-voice server. And Distribute the at least one first text data and / or the at least one first voice data as the at least one first data to the corresponding slave intelligent terminals for display.

10. The system according to claim 9, wherein, After the second data is displayed, The master intelligent terminal is further configured to: Obtain third data to be translated, where the third data to be translated includes: voice or text from the first user; and Determine at least one first target language according to the conversation mode and determine at least one first target terminal from the plurality of slave intelligent terminals, and send the information of the at least one first target language and the at least one first target terminal and the third data to be translated to the management server. The management server is further configured to: Use the voice-to-text engine to convert the voice in the third data to be translated into a third text to be translated. Generate a third translation prompt according to the information of the at least one first target language. Translate the third text to be translated or the text in the third data to be translated into at least one third text data according to the third translation prompt through the first large language model. Using the text-to-speech engine, convert the at least one third text data into at least one third speech data, and distribute the at least one third text data and / or the at least one third speech data to the at least one first target terminal.

11. The system according to claim 9, wherein The slave intelligent terminal includes: the slave intelligent wearable device or the slave intelligent mobile terminal, a part of the slave intelligent mobile terminal is associated with the slave intelligent wearable device, and the second data to be translated includes text or speech from the second user; The slave intelligent terminal is configured to obtain the second data to be translated and send the second data to be translated to the management server; The management server is further configured to: Using the speech-to-text engine, convert the speech in the second data to be translated into a second text to be translated; Generate the second translation prompt; Through the second large language model, according to the second translation prompt, translate the second text to be translated or the text in the second data to be translated into second text data, and the second large language model is configured in the management server or the model server; and Using the text-to-speech engine, convert the second text data into second speech data, and send the second text data and / or the second speech data as the second data to the master intelligent terminal for display.

12. The system according to claim 3, wherein The system further includes a management server, and the second large language model is configured in the management server, The slave intelligent mobile terminal is further configured to, in response to a first switching instruction, switch the working mode to a conference mode, and in the conference mode, determine at least one second target language according to at least one second target terminal pointed to by the user's selection operation, and send the information of the at least one second target terminal and the at least one second target language and the second data to be translated to the management server; The management server is configured to generate the second translation prompt according to the information of the at least one second target language, and through the second large language model, according to the second translation prompt, translate the second data to be translated into at least one second data corresponding to the at least one second target language, and according to the information of the at least one second target terminal, distribute the at least one second data to the at least one second target terminal for display; The slave intelligent mobile terminal is further configured to, in response to a second switching instruction, switch the working mode to a tour guide mode, and in the tour guide mode, send the second data to be translated and the language information of the first user to the management server; The management server is further configured to generate the second translation prompt according to the language information of the first user, and through the second large language model, according to the second translation prompt, translate the second data to be translated into second data corresponding to the language of the first user, and send the second data to the master intelligent terminal for display.

13. An intelligent terminal based on a large language model, characterized in that, Includes: An input device, a processor, a wireless communication component, and a memory, wherein the processor is electrically connected to the input device, the wireless communication component, and the memory; One or more programs executable by the processor are stored in the memory, and the one or more programs include a plurality of instructions for: In response to a first configuration instruction, configuring the smart terminal as a host; When the smart terminal serves as the host, obtaining first data to be translated through the input device and sending it to the cloud server through the wireless communication component, so that the cloud server uses the large language model in the cloud server to translate the first data to be translated into at least one first data according to a first translation prompt and distribute it to at least one slave smart terminal, wherein the first data to be translated includes a first text to be translated or a first voice from a user, and the languages of the respective first data correspond to the languages used by the users of the respective slave smart terminals; In response to a second configuration instruction, configuring the smart terminal as a slave; When the smart terminal serves as the slave, obtaining second data to be translated through the input device and sending it to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into a second data according to a second translation prompt and send it to the master smart terminal, wherein the second data to be translated includes a second text to be translated or a second voice from the user, and the language of the second data corresponds to the language used by the user of the master smart terminal.

14. The intelligent terminal according to claim 13, wherein The smart terminal is a smart mobile terminal or a smart wearable device.

15. The smart terminal according to claim 13, wherein The plurality of instructions are further used for: determining whether there is at least one terminal among the plurality of slave smart terminals whose user uses a language different from the language used by the user; if not, distributing the first data to be translated to each of the slave smart terminals for display; if so, sending the first data to be translated to at least one first terminal among the plurality of slave smart terminals for display, and sending the information of the language used by the user of at least one second terminal among the plurality of slave smart terminals and the first data to be translated to the cloud server, so that the cloud server uses the large language model to translate the first data to be translated into the at least one first data according to the first translation prompt and the information of the language used by the user of the at least one second terminal and distribute it to the corresponding second terminal, wherein the languages of the respective first data correspond to the languages used by the users of the respective second terminals, the language used by the user of the first terminal is the same as the language used by the user, and the language used by the user of the second terminal is different from the language used by the user; The multiple instructions are further used to: determine whether the language used by the user is the same as the language used by the user of the main intelligent terminal; if the same, send the second data to be translated to the main intelligent terminal for display; if not the same, send the second data to be translated to the cloud server.

16. The intelligent terminal according to claim 14, wherein An application is configured on the intelligent terminal, and the multiple instructions are further used to: Through the application, in response to an initiation instruction, create a meeting through a conference server and configure the intelligent wearable device as the host; and Through the conference server, according to a first access request, add the terminal that sends the first access request as a slave intelligent terminal to the meeting; The multiple instructions are further used to: After configuring the intelligent terminal as a slave, send a second access request to the conference server according to a preset shared link or the shared link obtained by scanning a QR code to join the meeting initiated by the main intelligent terminal.

17. The smart terminal according to claim 16, wherein The intelligent terminal is an intelligent wearable device, and the intelligent terminal further includes a Bluetooth component electrically connected to the processor. The multiple instructions are further used to: Through the Bluetooth component, send the first data to be translated to an intelligent mobile terminal, so that the intelligent mobile terminal sends the first data to be translated to the cloud server; Through the Bluetooth component, send the second data to be translated to the intelligent mobile terminal, so that the intelligent mobile terminal sends the second data to be translated to the cloud server.

18. The intelligent terminal according to claim 14, wherein The intelligent terminal further includes a speaker electrically connected to the processor. The multiple instructions are further used to: After configuring the intelligent terminal as the host, receive the voice sent by the cloud server and play it through the speaker; Obtain third data to be translated through the input device, and determine at least one first target language according to the conversation mode and determine at least one first target terminal from multiple associated slave intelligent terminals. The third data to be translated includes third voice to be translated or third text to be translated from the user; Through the wireless communication component, send the at least one first target language, the information of the at least one first target terminal, and the third data to be translated to the cloud server, so that the cloud server, through the large language model, according to the third translation prompt and the information, translates the third data to be translated into at least one third data and distributes it to the at least one first target terminal for display.

19. The intelligent terminal according to claim 14, wherein The multiple instructions are further used to: After configuring the intelligent terminal as a slave, in response to a first switching instruction, switch the working mode to the conference mode, and in the conference mode, determine at least one second target language according to at least one second target terminal pointed to by the user's selection operation, and send the information of the at least one second target terminal, the at least one second target language, and the second data to be translated to the cloud server, so that the cloud server, according to the information and the second translation hint, uses the large language model to translate the second data to be translated into at least one second data corresponding to the at least one second target language, and distributes it to the at least one second target terminal for display; In response to a second switching instruction, switch the working mode to the tour guide mode, and in the tour guide mode, send the second data to be translated and the language information of the user of the main intelligent terminal to the cloud server, so that the cloud server, according to the language information of the user of the main intelligent terminal and the second translation hint, uses the large language model to translate the second data to be translated into a second data corresponding to the language of the user of the main intelligent terminal, and sends the second data to the main intelligent terminal for display.

20. A multi-party cross-lingual interaction method based on a large language model, characterized in that, Applied to an intelligent mobile terminal, the method includes: In response to a first configuration instruction, configure the intelligent mobile terminal as a host; As the host, obtain first data to be translated and send it to the cloud server, so that the cloud server uses the large language model in the cloud server to translate the first data to be translated into at least one first data according to a first translation hint and distribute it to at least one slave intelligent mobile terminal, where the first data to be translated includes a first text to be translated or a first voice from a user, and the language of each first data corresponds to the language used by the user of each slave intelligent mobile terminal; In response to a second configuration instruction, configure the intelligent mobile terminal as a slave; As the slave, obtain second data to be translated and send it to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into a second data according to a second translation hint and send it to the main intelligent mobile terminal, where the second data to be translated includes a second text to be translated or a second voice from the user, and the language of the second data corresponds to the language used by the user of the main intelligent mobile terminal.

21. The method according to claim 20, wherein After configuring the intelligent mobile terminal as the host, the method further includes: Receive the voice sent by the cloud server and play it; Obtain third data to be translated, and according to the conversation mode, determine at least one first target language and determine at least one first target terminal from a plurality of associated slave intelligent terminals, where the third data to be translated includes a third text to be translated or a third voice from the user; Send the information of the at least one first target language, the at least one first target terminal, and the third data to be translated to the cloud server, so that the cloud server uses the large language model to translate the third data to be translated into at least one third data according to the third translation prompt and the information, and distribute it to the at least one first target terminal for display.

22. The method according to claim 20, wherein The obtaining and sending the second data to be translated to the cloud server includes: In the conference mode, determine at least one second target language according to at least one second target terminal pointed to by the user's selection operation, and send the information of the at least one second target terminal, the at least one second target language, and the second data to be translated to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into at least one second data corresponding to the at least one second target language according to the information and the second translation prompt, and distribute it to the at least one second target terminal for display; In the tour guide mode, send the second data to be translated and the language information of the user of the main intelligent terminal to the cloud server, so that the cloud server uses the large language model to translate the second data to be translated into a second data corresponding to the language of the user of the main intelligent terminal according to the language information of the user of the main intelligent terminal and the second translation prompt, and send the second data to the main intelligent terminal for display.

23. The method according to claim 20, wherein, After the intelligent mobile terminal is configured as the host in response to the first configuration instruction, the method further includes: Determine whether the language used by the user of at least one terminal among the multiple slave intelligent mobile terminals is different from the language used by the user; Sending the first data to be translated to the cloud server includes: If not, send the first data to be translated to the cloud server, and instruct the cloud server to distribute the first data to be translated to each slave intelligent mobile terminal for display; If so, send the first data to be translated to the cloud server, and instruct the cloud server to send the first data to be translated to at least one first terminal among the multiple slave intelligent mobile terminals for display, and at the same time use the large language model to translate the first data to be translated into at least one first data according to the first translation prompt and distribute it to the corresponding second terminals among the multiple slave intelligent mobile terminals, where the language of each first data corresponds to the language used by the user of each second terminal, the language used by the user of the first terminal is the same as the language used by the user, and the language used by the user of the second terminal is different from the language used by the user; After the intelligent mobile terminal is configured as a slave in response to the second configuration instruction, the method further includes: Determine whether the language used by the user is the same as the language used by the user of the main intelligent mobile terminal; The sending the second data to be translated to the cloud server includes: If they are the same, send the second data to be translated to the cloud server and instruct the cloud server to send the second data to be translated to the main intelligent mobile terminal for display; If they are different, send the second data to be translated to the cloud server and instruct the cloud server to use the large language model to translate the second data to be translated into the second data according to the second translation prompt and send it to the main intelligent mobile terminal.

Images