Smart glasses based on generative artificial intelligence large language model, system and control method
By integrating a generative artificial intelligence large language model in smart glasses to realize the chat function, the problem of single and low intelligence of smart glasses is solved, improving its intelligence and interactivity, and reducing hardware requirements and manufacturing costs.
Patent Information
- Application Number
- PCT/CN2024/092782
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-21
AI Technical Summary
The existing smart glasses have single functions, low intelligence, expensive, and lack versatility.
Adopt a smart glasses system based on a generative artificial intelligence large language model, realizes chat functions through microphone and speakers, combines intelligent mobile terminals and model servers, and uses a generative artificial intelligence large language model to process user voice and generate replies to improve the interaction of smart glasses Sex and intelligence.
The multi-function chat function of smart glasses is realized, improving its intelligence and interactivity, and reducing hardware requirements and manufacturing costs through scalability and self-creation.
Smart Images

Figure CN2024092782_21112024_PF_FP_ABST
Abstract
Description
Smart glasses, system and control method based on generative artificial intelligence large language model
[0001] This application claims priority to the Chinese patent application filed with the Patent Office of the State Intellectual Property Office of China on May 18, 2023, with application number CN202310566940.6 and entitled “Smart glasses, systems and control methods based on generative artificial intelligence large language model”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The embodiments of the present application relate to the field of smart glasses technology, and in particular to smart glasses, systems, and control methods based on a generative artificial intelligence large language model. Background Art
[0003] With the development of computer technology, smart glasses are becoming more and more popular. However, existing smart glasses are expensive and, in addition to their own functions as smart glasses, usually only have the functions of listening to music and making or receiving calls. The functions are relatively simple and the degree of intelligence is relatively low. Technical issues
[0004] The embodiments of the present application provide smart glasses, a system, and a control method based on a generative artificial intelligence large language model, which are used to improve the intelligence and interactivity of the smart glasses. Technical Solutions
[0005] On the one hand, an embodiment of the present application provides smart glasses based on a generative artificial intelligence large language model, including: a frame, temples, a microphone, a speaker, a processor, and a memory;
[0006] The temples are connected to the frame, and the processor is electrically connected to the microphone, the speaker, and the memory;
[0007] The memory stores one or more programs executable by the processor, wherein the one or more programs include multiple instructions, and the multiple instructions are used to:
[0008] activating the chat function of the smart glasses in response to a first control instruction for activating the chat function;
[0009] Acquiring a first voice of the user through the microphone, where the first voice includes a question raised by the user;
[0010] A second voice containing an answer to the question is obtained through a generative artificial intelligence large language model, and the second voice is played through the speaker.
[0011] On one hand, an embodiment of the present application further provides a smart glasses control system based on a generative artificial intelligence large language model, the system comprising smart glasses, a smart mobile terminal, and a model server, wherein the smart glasses include a microphone, a speaker, and a Bluetooth component;
[0012] The smart glasses are configured to activate the chat function of the smart glasses in response to a first control instruction for activating the chat function, acquire a first voice of the user through the microphone, and send the first voice to the smart mobile terminal through the Bluetooth component, where the first voice includes a question raised by the user;
[0013] The smart mobile terminal is configured to convert the first speech into a first text and send the text to the model server;
[0014] The model server is configured to obtain a second text based on the first text using a generative artificial intelligence language model and send the second text to the smart mobile terminal, wherein the second text includes an answer to the question;
[0015] The smart mobile terminal is further configured to convert the second text into a second voice and send the second voice to the smart glasses;
[0016] The smart glasses are further configured to receive the second voice via the Bluetooth component and play the second voice via the speaker.
[0017] On one hand, an embodiment of the present application further provides a method for controlling a smart wearable device based on a generative artificial intelligence large language model, which is applied to a smart wearable device. The method includes:
[0018] In response to a first control instruction for activating a chat function, activating the chat function of the smart wearable device;
[0019] Acquire a first voice of the user through a built-in microphone, where the first voice includes a question raised by the user;
[0020] A second voice containing an answer to the question is obtained through a generative artificial intelligence large language model, and the second voice is played through a built-in speaker. Beneficial effects
[0021] In each embodiment of the present application, a chat function based on smart glasses is realized by using a generative artificial intelligence big language model through smart glasses, thereby making the smart glasses have more functions. Moreover, due to the scalability and self-creativity of the generative artificial intelligence big language model, the intelligence and interactivity of the smart glasses can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0023] FIG1 is a schematic diagram of the internal structure of smart glasses provided in one embodiment of the present application;
[0024] FIG2 is a schematic diagram of the external structure of smart glasses provided in one embodiment of the present application;
[0025] FIG3 is a schematic diagram of the structure of a smart glasses control system based on a generative artificial intelligence large language model according to an embodiment of the present application;
[0026] FIG4 is a schematic diagram of the structure of a smart glasses control system based on a generative artificial intelligence large language model provided by another embodiment of the present application;
[0027] FIG5 is a schematic diagram of Application Example 1 in this application;
[0028] FIG6 is a schematic diagram of Application Example 2 in this application;
[0029] FIG7 is a schematic diagram of Application Example 3 in this application;
[0030] FIG8 is a schematic diagram of Application Example 5 in this application;
[0031] FIG9 is a schematic diagram of Application Example 6 in this application;
[0032] FIG10 is a schematic diagram of a chat control user interface provided by a chat program B on a smart mobile terminal according to an embodiment of the present application;
[0033] FIG11 is a schematic diagram of a chat parameter setting user interface provided by a chat program B on a smart mobile terminal according to an embodiment of the present application;
[0034] FIG12 is a schematic diagram of displaying chat records through a chat program B on a smart mobile terminal in one embodiment of the present application;
[0035] Figure 13 is a schematic diagram of the implementation process of the smart wearable device control method based on the generative artificial intelligence large language model provided in one embodiment of the present application. Modes for Carrying Out the Invention
[0036] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0037] Referring to FIG1 , FIG1 is a schematic diagram of the internal structure of smart glasses provided in one embodiment of the present application. FIG2 is a schematic diagram of the external structure of smart glasses provided in one embodiment of the present application. For ease of explanation, only the portions relevant to the embodiment of the present application are shown in the figure. As shown in FIG1 and FIG2 , smart glasses 100 include: a frame 101, at least one temple 102, at least one microphone 103, at least one speaker 104, at least one processor 105, and at least one memory 106 (only one is shown in the figure for ease of understanding).
[0038] The frame 101 may be, for example, a front frame with a lens 101A (eg, a sunglass lens, a clear lens, or a corrective lens). The at least one temple 102 may include, for example, a right temple 102B and a left temple 102A.
[0039] The temple 102 is connected to the frame 101, and the processor 105 is electrically connected to the microphone 103, the speaker 104, and the memory 106. The microphone 103, the speaker 104, the processor 105, and the memory 106 are disposed on at least one of the temples 102 and / or the frame 101. The temple 102 is detachably connected to the frame 101.
[0040] The processor 105 includes an MCU (Microcontroller Unit) and a DSP (Digital Signal Processing). The DSP is used to process the voice data acquired by the microphone 103.
[0041] Memory 106 is a non-transitory memory, specifically including RAM (Random Access Memory) and flash memory components. It stores one or more programs executable by processor 105, each of which includes multiple instructions. These instructions are used to activate the chat function of smart glasses 100 in response to a first control instruction for activating the chat function; capture a first user voice message containing a question posed by the user via microphone 103; obtain a second voice message containing a response to the question via a Generative Artificial Intelligence Large Language Model (GAILLM) model, and play the second voice message through speaker 104. GAILLM models include, but are not limited to, Open AI's ChatGPT, Google's Bard, and other models with similar functionality.
[0042] Optionally, in another embodiment of the present application, the smart glasses 100 further include a Bluetooth component 107 electrically connected to the processor 105. The Bluetooth component 107 includes a Bluetooth signal transceiver and surrounding circuitry, and may be disposed within the inner cavity of the frame 101 and / or at least one temple 102. The Bluetooth component can be connected to a smart mobile terminal such as a smartphone or smartwatch for phone calls, music, and data communications.
[0043] The multiple instructions are further configured to: receive the first control instruction sent by the smart mobile terminal via the Bluetooth component 107; simultaneously activate the chat function and wake up the microphone 103; and receive the second control instruction sent by the smart mobile terminal to disable the chat function via the Bluetooth component 107, and disable the chat function and the microphone 103 in response to the second control instruction. The first control instruction is generated by the virtual assistant program of the smart mobile terminal after receiving a voice wake-up instruction, or is generated by the operation interface of the smart mobile terminal after detecting a wake-up operation.
[0044] Optionally, in another embodiment of the present application, the one or more programs also include a virtual assistant program, which may be, but is not limited to, for example: Siri or OK Google, and the first control instruction is a first voice instruction. The multiple instructions are also used to obtain the first voice instruction through the virtual assistant program, and the first voice instruction includes a first preset keyword for instructing activation of the chat function.
[0045] Optionally, in another embodiment of the present application, the multiple instructions are also used to obtain, through the virtual assistant program, a second voice instruction containing a second preset keyword for instructing to disable the chat function, and disable the chat function in response to the second voice instruction.
[0046] Specifically, the virtual assistant program uses the microphone 103 to obtain the user's voice. If the user's voice contains a first preset keyword (i.e., an activation word) for instructing the activation of the chat function of the smart glasses 100, the chat function of the smart glasses 100 is activated; if the user's voice contains a second preset keyword for instructing the disabling of the chat function of the smart glasses 100, the chat function of the smart glasses 100 is disabled.
[0047] Optionally, in another embodiment of the present application, the plurality of instructions are further configured to capture user speech via the virtual assistant program. If the user speech contains the first preset keyword, a voiceprint is extracted from the user speech, the user is authenticated based on the voiceprint, and the chat function is activated when the user passes the authentication. Utilizing the voiceprint for authentication before activating the chat function can avoid inactivation, thereby improving the intelligence of the smart glasses control.
[0048] Optionally, in another embodiment of the present application, the virtual assistant program can be installed in a smart mobile terminal, and the above-mentioned operations involving the virtual assistant program can also be performed by the smart mobile terminal through the virtual assistant program on the smart mobile terminal. The smart mobile terminal sends corresponding control instructions to the smart glasses 100 according to the operations of the virtual assistant program to control the smart glasses 100 to perform corresponding operations, such as: activating the chat function, disabling the chat function, outputting corresponding prompt sounds, outputting corresponding prompt information, starting to pick up user voices, stopping picking up user voices, exporting chat records, etc.
[0049] Optionally, in another embodiment of the present application, the smart glasses 100 further include an input device electrically connected to the processor 105. The input device includes a button 108. The button 108 includes a physical button and / or a virtual button based on a touch sensor. The first control instruction is triggered based on a first preset operation of the user on the button 108.
[0050] Optionally, button 108 can be a power button on the smart glasses 100.
[0051] Optionally, in another embodiment of the present application, the first control instruction may also be automatically triggered when the smart glasses 100 are turned on, that is, the multiple instructions may automatically activate the chat function of the smart glasses 100 after the smart glasses 100 are turned on.
[0052] Optionally, in another embodiment of the present application, the plurality of instructions are further configured to receive the first voice via microphone 103 after receiving a user voice message containing a third preset keyword via microphone 103, wherein the third preset keyword is used to indicate that the user has started asking a question. The third preset keyword may be, for example, but not limited to, "start asking questions," "hey solos," or other user-defined keywords.
[0053] Furthermore, the multiple instructions are specifically configured to extract a voiceprint from a user's voice when microphone 103 receives a third preset keyword; authenticate the user based on the voiceprint; and, when the user passes the authentication, capture the first voice through microphone 103. Authentication using the voiceprint before capturing questions ensures that the captured questions are those posed by the user of the smart glasses, thereby improving the accuracy of capturing the first voice.
[0054] Furthermore, the aforementioned keywords can be customized by the user. The smart mobile terminal acquires the keywords input by the user through a built-in user interface (UI) for keyword setting and transmits the acquired keywords to the smart glasses 100 so that the smart glasses 100 configure the keywords in the smart glasses 100. Alternatively, the user can also set the aforementioned keywords through the virtual assistant program.
[0055] Optionally, in another embodiment of the present application, the multiple instructions are also used to obtain the first voice through the microphone 103 in response to the user's second preset operation on the button 108, wherein the second preset operation includes: long pressing the virtual button, short pressing the virtual button, touching the virtual button, tapping the virtual button, sliding on the virtual button set on the temple 102, and pressing and holding the physical button, wherein the pressing duration of the short press is less than the pressing duration of the long press, and tapping may include but is not limited to single-clicking, double-clicking and other actions.
[0056] Optionally, in another embodiment of the present application, the plurality of instructions are further configured to wake up the microphone 103 before acquiring the first voice through the microphone 103, and output a first prompt tone through the speaker 104 to prompt the user to start asking questions. The first prompt tone may be a pure voice word without actual semantics, such as "beep", or may include a word with actual semantics, such as "start".
[0057] For example, when the user taps a virtual button on the smart glasses 100 , the processor 105 wakes up the microphone 103 and outputs a first prompt tone such as “Microphone is ready, please ask questions” through the speaker 104 to prompt the user to start asking questions.
[0058] Optionally, in another embodiment of the present application, the multiple instructions are further configured to: before obtaining a second voice message containing an answer to the question through GAILLM, terminate the operation of obtaining the first voice message and output a second prompt tone to prompt the user to end the question when any of the following events is detected: the user terminates the second preset operation; the user performs a third preset operation on the button 108 after completing the second preset operation; and silence for a first preset duration occurs; wherein the preset third operation includes any one of touching the virtual button, tapping the virtual button, and sliding on the virtual button set on the temple 102. The preset third operation may be the same as or different from the second preset operation.
[0059] Optionally, the second prompt tone is also used to ask the user whether to finish asking the question, and the multiple instructions are also used to: if the user's answer is affirmative, end the operation of obtaining the first voice; if the user's answer is negative, control the microphone 103 to continue the operation of obtaining the first voice until the user's answer is affirmative.
[0060] Furthermore, sliding on the virtual button includes the user performing a preset sliding gesture on the virtual button. Different preset operations may correspond to different sliding gestures, and the user may set the sliding gesture and corresponding functions through a UI interface for setting sliding gestures provided by the smart mobile terminal. The smart mobile terminal transmits information about the sliding gesture set by the user to the smart glasses 100 through the UI interface, so that the smart glasses 100 may set the sliding gesture in the smart glasses 100 according to the information.
[0061] Optionally, in another embodiment of the present application, the multiple instructions are also used to: receive a first configuration instruction for configuring the first preset duration sent by the smart mobile terminal through the Bluetooth component 107, and set the first preset duration to the duration pointed to by the first configuration instruction.
[0062] Optionally, in another embodiment of the present application, the smart glasses 100 further include an indicator light 109 and / or a buzzer 110 electrically connected to the processor 105, and the multiple instructions are further used to:
[0063] Prompt information is output via the indicator light 109 and / or the buzzer 110. This prompt information indicates the status of the smart glasses 100, including the active state and the idle state. The active state includes: starting voice pickup, voice pickup, ending voice pickup, and voice processing. The indicator light 109 can be an LED (Light Emitting Diode). This prompt information can be synchronized with the first and second prompt tones. For example, when a user taps a virtual button on the smart glasses 100, the processor 105 outputs a first prompt tone, such as "Microphone ready, please ask a question," through the speaker 104 to prompt the user to start asking a question, while simultaneously controlling the indicator light 109 to illuminate green. Furthermore, the processor can also control the buzzer 110 to emit a beep. This simultaneous prompting through multiple methods can prevent users from missing the first prompt tone, for example, because it is too quiet, thereby improving the effectiveness of the prompt.
[0064] Optionally, in another embodiment of the present application, the GAILLM is configured on a model server (i.e., a server configured with GAILLM, hereinafter collectively referred to as a GAILLM server), the one or more programs further include a speech-to-text engine and a text-to-speech engine, and the smart glasses 100 further include a wireless communication component 111 electrically connected to the processor 105. The second speech containing the answer to the question obtained through GAILLM includes:
[0065] Converting the first speech into a first text using the speech-to-text engine;
[0066] Sending the first text to the GAILLM server via the wireless communication component 111, and receiving a second text including the reply sent by the GAILLM server, wherein the second text is obtained by the GAILLM server inputting the first text into the GAILLM;
[0067] The second text is converted into the second speech by the text-to-speech engine.
[0068] The wireless communication component includes a wireless signal transceiver and surrounding circuitry, and may be disposed within the inner cavity of the frame 101 and / or at least one of the temples 102. The wireless signal transceiver may utilize, but is not limited to, at least one of the following protocols for data transmission: WiFi (Wireless Fidelity), NFC (Near Field Communication), ZigBee, UWB (Ultra Wideband), RFID (Radio Frequency Identification), and cellular mobile communication protocols (e.g., 3G / 4G / 5G).
[0069] Optionally, the speech-to-text engine and the text-to-speech engine may also be configured in a conversion server, and the multiple instructions are further used to: send the first speech to the conversion server via the wireless communication component 111, so that the conversion server uses the speech-to-text engine to convert the first speech into the first text; send the second text to the conversion server via the wireless communication component 111, so that the conversion server uses the text-to-speech engine to convert the second text into the second speech.
[0070] Optionally, in another embodiment of the present application, the multiple instructions are also used to: generate a chat record, and send the chat record and the first text to the GAILLM server through the wireless communication component 111, so that the GAILLM server generates the second text based on the chat record and the first text through GAILLM.
[0071] Furthermore, the multiple instructions are also used to: send the generated chat records and the user's login account to the chat record storage server for storage through the wireless communication component 111, or send the generated chat records and the user's login account to the smart mobile terminal for storage through the Bluetooth component 107; when the user voice containing the query request is obtained through the microphone 103 or the virtual assistant program, obtain the chat record corresponding to the query request through the chat record storage server or the smart mobile terminal, and output the obtained chat record through the speaker 104, or export the obtained chat record to other devices according to the preset export method.
[0072] Optionally, in another embodiment of the present application, the GAILLM is configured in a smart mobile terminal, and the multiple instructions are further used to:
[0073] Converting the first speech into the first text using a built-in speech-to-text engine;
[0074] The first text is sent to the smart mobile terminal through the Bluetooth component 107, so that the smart mobile terminal obtains the second text through GAILLM on the smart mobile terminal according to the first text and sends it to the smart glasses;
[0075] Receiving the second text sent by the smart mobile terminal through the Bluetooth component 107; and
[0076] The second text is converted into the second speech through a built-in text-to-speech engine.
[0077] Furthermore, the multiple instructions are also used to: send the first text and the chat record to the smart mobile terminal through the Bluetooth component 107, so that the smart mobile terminal obtains the second text through GAILLM on the smart mobile terminal based on the first text and the chat record.
[0078] When the GAILLM is configured in a smart mobile terminal, optionally, in another embodiment of the present application, the plurality of instructions are further used to:
[0079] The first speech is transmitted to the smart mobile terminal via the Bluetooth component 107, so that the smart mobile terminal converts the first speech into the first text via a speech-to-text engine. The second text is obtained via GAILLM on the smart mobile terminal based on the first text. The second text is then converted into the second speech via a text-to-speech engine and transmitted to the smart glasses. The speech-to-text engine and the text-to-speech engine can be installed on the smart mobile terminal or on a conversion server, and the smart mobile terminal can convert between text and speech via the conversion server.
[0080] Furthermore, the multiple instructions are also used to: send the first voice and the chat record to the smart mobile terminal through the Bluetooth component 107, so that the smart mobile terminal converts the first voice into the first text through the voice-to-text engine, obtains the second text through GAILLM on the smart mobile terminal based on the first text and the chat record, converts the second text into the second voice through the text-to-speech engine and sends it to the smart glasses.
[0081] When the GAILLM is configured in a smart mobile terminal, optionally, in another embodiment of the present application, the plurality of instructions are further used to:
[0082] Sending the first voice or the first voice and the chat record to the conversion server through the wireless communication component 111, so that the first voice is converted into the first text through the conversion server, and the first text or the first text and the chat record are sent to the smart mobile terminal, so that the smart mobile terminal obtains the second text according to the first text or the first text and the chat record through GAILLM on the smart mobile terminal and sends the second text to the conversion server, and the conversion server converts the second text into the second voice; and
[0083] The second voice sent by the conversion server is received through the wireless communication component 111.
[0084] When the GAILLM is configured in a smart mobile terminal, optionally, in another embodiment of the present application, the plurality of instructions are further used to:
[0085] Sending the first speech to a conversion server through the wireless communication component 111, so that the first speech is converted into the first text through the conversion server;
[0086] Receiving the first text sent by the conversion server through the wireless communication component 111;
[0087] Sending the first text or the first text and the chat record to the smart mobile terminal through the Bluetooth component 107, so that the smart mobile terminal obtains the second text through GAILLM on the smart mobile terminal according to the first text or the first text and the chat record;
[0088] Receiving the second text sent by the smart mobile terminal through the Bluetooth component 107;
[0089] Sending the second text to the conversion server through the wireless communication component 111, so that the second text is converted into the second voice through the conversion server;
[0090] The second voice sent by the conversion server is received through the wireless communication component 111.
[0091] Optionally, in another embodiment of the present application, the smart glasses 100 further include a data sensing component 112 electrically connected to the processor 105, and the data sensing component 112 includes at least one of a positioning component, an inertial measurement unit (IMU) sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer;
[0092] The multiple instructions are also used to obtain the sensing data of the at least one device, and send the sensing data of the at least one device and the first text to the GAILLM server through the wireless communication component 111, so that the GAILLM obtains the second text based on the sensing data and the first text.
[0093] Optionally, in another embodiment of the present application, the GAILLM is configured on a GAILLM server, the smart glasses 100 are further configured with a speech-to-text engine and a text-to-speech engine, and the smart glasses 100 further include a Bluetooth component electrically connected to the processor 105. The second voice message containing the answer to the question obtained through GAILLM includes:
[0094] Converting the first speech into a first text using the speech-to-text engine;
[0095] The first text is sent to the smart mobile terminal via the Bluetooth component 107, and a second text including the reply is received from the smart mobile terminal, wherein the second text is sent to the GAILLM server via the smart mobile terminal based on the first text and the GAILLM server.
[0096] The second text is converted into the second speech by the text-to-speech engine.
[0097] Optionally, in another embodiment of the present application, the GAILLM is configured on a GAILLM server, and the smart glasses 100 further include a Bluetooth component electrically connected to the processor 105. The second voice message containing the answer to the question obtained through the GAILLM includes:
[0098] The first voice is sent to the smart mobile terminal through the Bluetooth component 107, and the second voice sent by the smart mobile terminal is received, wherein the smart mobile terminal converts the first voice into a first text and sends it to the GAILLM server, and the GAILLM server inputs the first text into the GAILLM to obtain a second text containing the reply and sends it to the smart mobile terminal, and the smart mobile terminal converts the second text into the second voice and sends it to the smart glasses 100.
[0099] Optionally, in another embodiment of the present application, the multiple instructions are also used to: after playing the second voice, if silence of a second preset length is detected, control the smart glasses 100 to enter a standby state, disable the chat function, control the microphone 103 to enter a sleep state, and output a third prompt sound to prompt the user that the smart glasses 100 are about to enter a standby state.
[0100] Optionally, in another embodiment of the present application, the multiple instructions are also used to: receive a second configuration instruction for configuring the second preset duration sent by the smart mobile terminal through the Bluetooth component 107, and set the second preset duration to the duration pointed to by the second configuration instruction.
[0101] Optionally, in another embodiment of the present application, the smart glasses 100 further include at least one of a positioning component electrically connected to the processor 105, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer.
[0102] The positioning component may be, but is not limited to, a positioning component based on GPS (Global Positioning System) or Beidou satellite.
[0103] These multiple instructions are also used to obtain the sensing data of the at least one device after obtaining the user's first voice through the microphone 103, and send the sensing data of the at least one device to the smart mobile terminal through the Bluetooth component 107, so that the smart mobile terminal sends the sensing data, the device data of the smart mobile terminal and the first text to the GAILLM server, so that GAILLM obtains the second text based on the sensing data, the device data of the smart mobile terminal and the first text.
[0104] Optionally, in another embodiment of the present application, the plurality of instructions are further used to:
[0105] In response to the user's preset fourth operation on button 108 for controlling the volume, the second voice is played according to the volume indicated by the preset fourth operation, wherein the preset fourth operation includes: sliding on the virtual button, touching the virtual button, and pressing any one of the physical button.
[0106] Optionally, in another embodiment of the present application, the multiple instructions are also used to: receive a language setting instruction sent by the smart mobile terminal through the Bluetooth component 107, and set the user's language to the target language type pointed to by the language setting instruction, so that the speech-to-text engine in the smart glasses 100 converts the first speech into a first text according to the target speech type.
[0107] Optionally, in another embodiment of the present application, the multiple instructions are also used to: receive a language self-test configuration instruction sent by the smart mobile terminal through the Bluetooth component 107, and activate the automatic language detection function in response to the language self-test configuration instruction, so that the speech-to-text engine in the smart glasses 100 converts the first speech into the first text according to the language type obtained by self-test.
[0108] Optionally, in another embodiment of the present application, the multiple instructions are further used to: receive a playback speed control instruction sent by the smart mobile terminal through the Bluetooth component 107, and play the second voice at a rate indicated by the playback speed control instruction.
[0109] Furthermore, the smart glasses 100 also include a battery 113 for providing power support for the electronic components of the smart glasses 100, such as the microphone 103, the speaker 104, the processor 105, and the memory 106.
[0110] The electronic components of the smart glasses can be connected via a bus.
[0111] It should be noted that the relationships between the components of the aforementioned smart glasses can be either replacement or superposition. That is, a pair of smart glasses can be equipped with all of the components described in this embodiment, or, depending on the needs, selectively install a portion of the components. In the case of replacement, the smart glasses also include a peripheral connection interface, which can be, for example, at least one of a PS / 2 interface, a serial interface, a parallel interface, an IEEE1394 interface, and a USB (Universal Serial Bus) interface. The functionality of the replaced component can be implemented by a peripheral connected to the interface, such as an external speaker or sensor.
[0112] In this embodiment, the smart glasses utilize a generative artificial intelligence language model to realize a chat function based on the smart glasses, thereby making the smart glasses have more functions. Moreover, due to the scalability and self-creativity of the generative artificial intelligence language model, the intelligence and interactivity of the smart glasses can be further improved.
[0113] Referring to Figure 3 , which is a schematic diagram of the structure of a smart glasses control system based on a generative artificial intelligence large language model according to one embodiment of the present application, the control system includes: smart glasses 301 , a smart mobile terminal 302 , and a GAILLM server 303 .
[0114] The smart glasses 301 may be open smart glasses, and their specific structure may refer to the relevant descriptions in the embodiments shown in FIG. 1 and FIG. 2 .
[0115] Smart mobile terminal 302 may include, but is not limited to, cellular phones, smartphones, other wireless communication devices, personal digital assistants (PDAs), audio players, other media players, music recorders, video recorders, cameras, other media recorders, smart radios, laptop computers, personal digital assistants (PDAs), portable multimedia players (PMPs), Moving Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) players, digital cameras, and smart wearable devices (such as smart watches and smart bracelets). The smart mobile terminal 302 may be installed with an Android or iOS operating system.
[0116] The GAILLM server 303 may be a single server or a distributed server cluster consisting of multiple servers.
[0117] The smart glasses 301 are used to respond to a first control instruction for activating the chat function of the smart glasses 301, activate the chat function of the smart glasses 301, obtain the user's first voice through the built-in microphone, and send the first voice to the smart mobile terminal 302 through the built-in Bluetooth component. The first voice includes the question raised by the user.
[0118] The smart mobile terminal 302 is configured to convert the first speech into a first text and send the first text to the GAILLM server 303 .
[0119] The GAILLM server 303 is configured to obtain a second text based on the first text through the generative artificial intelligence language model GAILLM and send the second text to the smart mobile terminal 302, wherein the second text includes an answer to the question.
[0120] The smart mobile terminal 302 is further configured to convert the second text into a second voice and send the second voice to the smart glasses 301 .
[0121] The smart glasses 301 are further configured to receive the second voice via the Bluetooth component and play the second voice via the speaker.
[0122] Optionally, in another embodiment of the present application,
[0123] The smart glasses 301 are further configured to convert the first speech into the first text using a built-in speech-to-text engine, and send the first text to the GAILLM server 303;
[0124] The GAILLM server 303 is further configured to obtain the second text based on the first text sent by the smart glasses 301 through the generative artificial intelligence language model and send the second text to the smart glasses 301;
[0125] The smart glasses 301 are further configured to convert the second text into the second speech using a built-in text-to-speech engine.
[0126] Optionally, in another embodiment of the present application, the smart mobile terminal 302 is further configured to display the second text on a display screen of the smart mobile terminal 302 .
[0127] Optionally, in another embodiment of the present application, as shown in FIG4 , the system further includes a conversion server 401 .
[0128] The smart mobile terminal 302 is further configured to send the first voice to the conversion server 401;
[0129] The conversion server 401 is further configured to convert the first speech into the first text via a local speech-to-text engine, and send the first text to the GAILLM server 303;
[0130] The GAILLM server 303 is further configured to obtain the second text through GAILLM according to the first text sent by the conversion server 401 and send the second text to the conversion server 401;
[0131] The conversion server 401 is further configured to convert the second text into the second voice through a local text-to-speech engine, and send the second voice to the smart mobile terminal 302 so that the smart mobile terminal 302 forwards the second voice to the smart glasses 301 .
[0132] Optionally, in another embodiment of the present application,
[0133] The smart glasses 301 are further configured to send the first speech to the conversion server 401;
[0134] The conversion server 401 is further configured to convert the first speech into the first text via a local speech-to-text engine, and send the first text to the GAILLM server 303;
[0135] The GAILLM server 303 is further configured to obtain the second text through GAILLM according to the first text sent by the conversion server 401 and send the second text to the conversion server 401;
[0136] The conversion server 401 is further configured to convert the second text into the second speech through a local text-to-speech engine, and send the second speech to the smart glasses 301 .
[0137] Optionally, in another embodiment of the present application, the smart mobile terminal 302 is further used to adjust the playback rate of the second voice to the target rate indicated by the operation in the user interface of the smart mobile terminal 302 in response to the user's operation for adjusting the playback speed, and send the second voice with the target rate to the smart glasses 301.
[0138] Optionally, in another embodiment of the present application, as shown in FIG4 , the system further includes a chat record storage server 402 .
[0139] The smart mobile terminal 302 is further configured to generate a chat record based on the data sent by the smart glasses 301 under the chat function, associate the chat record with the user's login account, and store the chat record on the smart mobile terminal 302 or the chat record storage server 402;
[0140] The smart mobile terminal 302 is further configured to send the first text and the chat history to the GAILLM server 303;
[0141] The GAILLM server 303 is further configured to obtain the second text through GAILLM according to the first text and the chat record.
[0142] Optionally, in another embodiment of the present application, the smart mobile terminal 302 is further used to obtain the target chat record corresponding to the query operation according to the user's query operation and export it according to a preset export method.
[0143] Optionally, in another embodiment of the present application, the preset export method includes: exporting to a preset social media platform or exporting to a specified device. Optionally, the preset export method may also include: displaying the target chat record on the display screen of the smart mobile terminal 302.
[0144] Optionally, in another embodiment of the present application,
[0145] The smart glasses 301 are further configured to obtain sensing data from at least one of a built-in positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer, and transmit the sensing data and the first voice to the smart mobile terminal 302 via the Bluetooth component;
[0146] The smart mobile terminal 302 is further configured to obtain device data of the smart mobile terminal 302 and send the sensing data, the device data, and the first text to the GAILLM server 303;
[0147] The GAILLM server 303 is further configured to obtain the second text according to the sensing data, the device data, and the first text through GAILLM.
[0148] In actual applications, corresponding applications, such as chat program A and chat program B, may be installed on smart glasses 301 and smart mobile terminal 302 to implement their respective functions. Preferably, smart glasses 301 can establish a data connection and exchange data with smart mobile terminal 302 via Bluetooth. Smart glasses 301 can also establish a data connection and exchange data with GAILLM server 303, conversion server 401, and chat history storage server 402 via WiFi or cellular mobile communications (e.g., 3G / 4G / 5G).
[0149] Optionally, in another embodiment of the present application, a virtual assistant program is also installed in the smart mobile terminal 302. The smart mobile terminal 302 is also used to send corresponding control instructions to the smart glasses 301 based on the user's operations in the chat program B or the voice instructions from the user obtained by the virtual assistant program to control the smart glasses 100 to perform corresponding operations, such as: activating the chat function, disabling the chat function, outputting the corresponding prompt tone, outputting the corresponding prompt information, starting to pick up the user's voice, stopping picking up the user's voice, exporting chat records, etc.
[0150] It should be noted that the specific implementation process of the functions of the smart glasses 301, smart mobile terminal 302, GAILLM server 303, conversion server 401 and chat record storage server 402 in this embodiment can also refer to the relevant descriptions in other embodiments.
[0151] Several application examples are given below to further illustrate the functions of each device or apparatus in the above system.
[0152] Application Example 1
[0153] As shown in FIG5 , after the user starts the chat function on the smart glasses 301 through the chat control UI interface shown in FIG10 provided by the chat program B running on the smart watch 302 or through the virtual assistant such as Siri / OK Google built into the smart watch 302, if the user presses and holds the touch sensor-based virtual button on the temple of the smart glasses 301, a prompt tone will be heard to prompt that the microphone on the smart glasses 301 is ready to listen to the user's speech.
[0154] While holding down the virtual button, the user asks a question to the chat program A in the smart glasses 301 through voice, such as "Where is Hong Kong?" The smart glasses 301 pick up the voice through the built-in microphone and send the voice to the chat program B on the smartphone or smart watch 302 via Bluetooth.
[0155] When the user releases the virtual button, another notification sound will be heard, and chat program A will control the microphone to stop picking up the voice and send the picked up voice to chat program B.
[0156] Chat program B uses a built-in speech-to-text engine to convert the speech into a first text, and generates a chat record between the user and chat program A (i.e., a chat record between the user and GAILLM) based on the speech, and then sends the first text together with the chat record to the GAILLM server 303, so that the GAILLM server 303 obtains a second text containing the answer to the question based on the first text and the chat record through the generative artificial intelligence large language model GAILLM and sends it to chat program B.
[0157] After receiving the second text in reply from GAILLM, chat program B converts the second text into corresponding speech through the built-in text-to-speech engine, and sends the converted speech in audio format to smart glasses 301 via Bluetooth. Finally, smart glasses 301 play the speech through its speakers.
[0158] The user can continue to ask the next question until the user disables the chat feature.
[0159] Application Example 2
[0160] As shown in Figure 6, smart glasses 301 have built-in speech-to-text engines and text-to-speech engines. A user can press and hold a virtual button based on a touch sensor on the temple and ask a question to chat program A. Smart glasses 301 pick up the user's voice through a built-in microphone and convert it into first text within the smart glasses 301.
[0161] When the user releases the virtual button, the smart glasses 301 sends the first text to the chat program B on the smart phone or smart watch 302 via Bluetooth. The chat program B generates a chat record between the user and the chat program A based on the first text, and sends the first text and the chat record to the GAILLM server 303, so that the GAILLM server 303 obtains the second text containing the answer to the question through GAILLM based on the first text and the chat record.
[0162] After receiving the second text containing the answer to the question from the GAILLM server 303, the chat program B sends the second text back to the smart glasses 301 via Bluetooth. The smart glasses 301 converts the second text into speech through an internal text-to-speech engine and then plays the speech through its speakers.
[0163] Application Example 3
[0164] As shown in Figure 7, smart glasses 301 have a built-in 4G / 5G cellular module or WiFi module. The user can press and hold a virtual button on the temple of the glasses, which uses a touch sensor, and then ask a question to chat program A in the smart glasses 301. The smart glasses 301 use a built-in microphone to pick up the voice message containing the question and convert it internally into a first text message. When the user releases or lets go of the virtual button, the smart glasses 301 generate a chat log between the user and chat program A and transmit this first text message and chat log to the GAILLM server 303 via the 4G / 5G cellular module or WiFi module. GAILLM server 303 then retrieves a second text message containing the answer to the question based on the first text message and chat log via GAILLM.
[0165] Chat program A receives the second text containing the answer to the question from GAILLM server 303, converts the second text into corresponding speech using an internal text-to-speech engine, and then plays the speech through a speaker.
[0166] Application Example 4
[0167] Based on the above-mentioned application examples 1-3, smart glasses 301 provide more than one method for users to ask questions continuously. Specifically, this can be configured through the UI interface for configuring questioning methods in chat program B on smartphone 302, allowing the user to ask the next question after hearing the answer from chat program A. One method requires the user to long-press a virtual button to reactivate the microphone of smart glasses 301 to listen to the user's next question. Another method is to keep the microphone of smart glasses 301 active for a short period of time. Through the above-mentioned UI interface, after chat program A plays the voice message containing the answer to the question, the user can immediately ask the next question within a preset time window without having to press the virtual button again. After this time window, smart glasses 301 will play a notification sound to inform the user that smart glasses 301 have entered standby mode and the microphone is no longer activated to pick up the user's voice. The user will need to press and hold the virtual button again to reactivate the microphone to ask the next question.
[0168] Application Example 5
[0169] As shown in FIG8 , after the user starts the chat function through the chat program A on the smart glasses 301 or the chat program B on the smartphone, the microphone of the smart glasses 301 starts to automatically listen to the user's voice until the user disables the chat function through the chat program A or the chat program B on the smartphone.
[0170] When the chat function is enabled, if a user asks a question and then stops speaking and remains idle for a period of time (the idle period can be configured by chat program B), smart glasses 301 transmit the audio-formatted speech to chat program B via Bluetooth. Chat program B then converts the speech into a first text using its speech-to-text engine. This first text, along with the chat history between the user and chat program A, is then transmitted to GAILLM server 303. GAILLM server 303 then retrieves a second text containing the answer to the question based on the first text and the chat history. This chat history contains all historical content of the user's chats with GAILLM via chat program A during the current chat function activation period or a preset period. The user can set this preset period through chat program B.
[0171] After receiving the second text containing the answer to the question from GAILLM server 303, chat program B converts the second text into audio format using a text-to-speech engine. This audio format is then transmitted via Bluetooth to smart glasses 301, which then play the audio through the speakers. After the speakers of smart glasses 301 finish playing the audio, the user can proceed to ask the next question.
[0172] Application Example 6
[0173] As shown in Figure 9, users can use preset keywords (i.e., wake-up words, such as "hey solos") to start using the chat function. After enabling the chat function through chat program A on smart glasses 301 or chat program B on a smartphone, users can use the wake-up word to start using the chat function. When smart glasses 301 detect the wake-up word, the microphone on smart glasses 301 is triggered to start picking up the user's voice. At the same time, smart glasses 301 output a beep to notify the user that smart glasses 301 are listening to the user's voice. After the user finishes asking a voice question and a preset period of silence has passed (this preset period can be configured through chat program B on the smartphone), smart glasses 301 will play another prompt tone to notify the user that the voice will be sent to GAILLM.
[0174] Smart glasses 301 transmit the audio format of the voice containing the question posed by the user to chat program B via Bluetooth. Chat program B converts the voice into first text using a speech-to-text engine and generates a chat log between the user and chat program A based on the voice. Chat program B then transmits the first text together with the chat log to GAILLM server 303, so that GAILLM server 303 can obtain a second text containing the answer to the question through GAILLM based on the first text and the chat log.
[0175] After receiving the second text sent by the GAILLM server 303, the chat program B converts the second text into speech through a text-to-speech engine, and then sends the speech in audio format via Bluetooth to the smart glasses 301. The smart glasses 301 finally play the speech through their speakers.
[0176] Application Example 7
[0177] Based on the above-mentioned application example 6, the smart glasses 301 remain activated for a short period of time, and the corresponding time window can be configured through the configuration UI interface of the chat program B on the smartphone 302. After the smart glasses 301 play the voice containing the answer to the question through the speaker, the user can continue to ask the next question without using the wake-up word again. After this short activation time window, if the smart glasses 301 cannot detect the user's voice, a prompt tone will be played to notify the user that the smart glasses 301 have entered standby mode and its microphone is no longer activated. The user needs to use the wake-up function to activate the smart glasses 301 to ask the next question.
[0178] Application Example 8
[0179] Based on the above application example 6, the smart glasses 301 may also have a built-in voice biometric recognition module to perform voiceprint recognition, so that only the owner of the smart glasses 301 can wake up the smart glasses 301 and use the GAILLM-based chat function.
[0180] Application Example 9
[0181] Building on Application Examples 1 and 6 above, in addition to using a beep to indicate that the smart glasses 301 are picking up the user's voice and that the voice has been sent to GAILLM and is awaiting a response, the smart glasses 301 can also use an LED to indicate their status, such as: picking up the user's voice (e.g., green), awaiting a response from GAILLM (red), or completely idle (off). In these application examples, the LED turns off after a short window of time during which the chat function of the smart glasses 301 is activated. If the user sees this LED turn off, they understand that they can activate the smart glasses 301 by pressing and holding a virtual button or using a wake-up word to pick up the user's voice again.
[0182] Application Example 10
[0183] While receiving the first audio message containing the user's question, the smart glasses 301 also acquire their own device data, such as step count, user posture, and sensor data (e.g., IMU data, electronic compass direction, touch sensor data, etc.). After the user completes their question, the smart glasses 301 convert the first audio message into a first text message. The first text message and the device data of the smart glasses 301 are then sent to the GAILLM server 303. The GAILLM server 303 then uses the first text message and the device data of the smart glasses 301 to retrieve a second text message containing the answer to the question through GAILLM. The smart glasses 301 then receive the second text message returned by the GAILLM server 303, convert it into a second audio message, and play it back.
[0184] If smart glasses 301 are currently connected to smartphone 302, they send the first text and their device data to smartphone 302. Smartphone 302 receives the first text and the device data, uses its built-in positioning system and various sensors to obtain the device data, as well as environmental data such as GPS location, temperature, and humidity. It then sends the data, along with the first text and the device data, to GAILLM server 303. Based on the first text, the device data from smart glasses 301, the device data from smartphone 302, and the environmental data, GAILLM uses GAILLM to obtain a second text containing an answer to the user's question, thereby better responding to the user's question. For example, if a user asks, "How many steps do I need to take today to complete 10K steps?" GAILLM uses the device data from smart glasses 301 sent by smartphone 302 to correctly answer the user's question about the number of steps remaining, such as, "You need to take 2,608 steps today to reach your 10K goal."
[0185] Furthermore, if the smart glasses 301 have a camera, the smart glasses 301 transmit the image captured by the camera along with the first text to the GAILLM server 303, so that the GAILLM server 303 can obtain an accurate response to the user's question based on the image through GAILLM. For example, if the user asks "What am I looking at?", "Which direction should I go?", or "Translate this", GAILLM can provide a corresponding response based on the image.
[0186] Optionally, after acquiring device data and environmental data, smart glasses 301 and smartphone 302 only send the first text to GAILLM server 303. Based on the first text sent by smart glasses 301 or smartphone 302, GAILLM server 303 determines through GAILLM whether the answer to the question requires data collected by a built-in data acquisition device in smart glasses 301 and / or smartphone 302, as well as required target data, and then sends a description of the target data to smart glasses 301 or smartphone 302. The data acquisition device may include, but is not limited to, at least one of a positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer.
[0187] If the description information is sent to the smart glasses 301 , the smart glasses 301 determine the target data from their own device data previously acquired according to the description information, and send the target data to the GAILLM server 303 .
[0188] If the description information is sent to the smartphone 302 , the smartphone 302 determines the target data from the device data of the smart glasses 301 and its own device data and environmental data previously acquired based on the description information, and sends the target data to the GAILLM server 303 .
[0189] Application Example 11
[0190] The user can set the user's language type through the chat parameter setting UI interface shown in Figure 11 provided by the chat program B on the smartphone 302, so that the speech-to-text engine on the smartphone 302 or the smart glasses 301 converts the first speech containing the user's question into corresponding text according to the set language type.
[0191] Alternatively, the user can also start automatic language detection through the UI interface, so that the smartphone 302 or smart glasses 301 automatically detects the language type corresponding to the first voice after obtaining the first voice, and converts the first voice into corresponding text according to the automatically detected language type through the speech-to-text engine.
[0192] The user can also set the playback speed or rate of the second voice through the UI interface as shown in Figure 11, such as: normal, 1.25 times faster or slower, 1.5 times faster or slower, 2 times faster or slower, etc.
[0193] The smartphone 302 sends a playback speed control instruction to the smart glasses 301 according to the user's playback speed setting operation on the UI interface, so as to instruct the smart glasses 301 to play the second voice at the rate indicated by the setting operation.
[0194] Alternatively, the smartphone 302 may also generate a second voice with a rate indicated by the setting operation from the second text and send the second voice to the smart glasses 301 for playback.
[0195] Application Example 12
[0196] As shown in Figure 12, the smartphone 302 displays the complete chat history between the user and GAILLM in the chat control UI interface of the chat program B. The smartphone 302 can also store the chat history in the device and cloud chat history storage server of each user account.
[0197] Users can also use the export button or menu in the chat history display UI to export the complete chat history as a file to social media such as Facebook, or export it to nearby devices via AirDrop.
[0198] The chat logs can be generated by smart glasses 301 based on interaction data between the user, smartphone 302, and GAILLM server 303 during the period when the chat function is activated. Smart glasses 301 transmit the generated chat logs to smartphone 302 or, via smartphone 302, to a cloud-based chat log storage server for storage. Smart glasses 301 transmit the generated chat logs and text containing the user's questions to GAILLM server 303, so that GAILLM server 303 can retrieve the corresponding answers based on the chat logs and text.
[0199] The above chat records can also be generated by the smartphone 302 based on the interaction data between the smartphone 302 and the smart glasses 301 and the GAILLM server 303 during the activation of the chat function. The smartphone 302 saves the generated chat records locally or sends them to the cloud chat record storage server for storage.
[0200] The chat logs described above can also be generated by GAILLM server 303 based on interaction data with smart glasses 301 and / or smartphone 302. GAILLM server 303 can store the generated chat logs locally for use in deriving a second text containing the answer based on the chat logs and the first text containing the question, or can send them to smartphone 302 and / or a cloud-based chat log storage server for storage. For example, if the user's first question is "What are the 10 best rock songs of the 90s?" GAILLM server 303 sends the names of the 10 best rock songs of the 90s as the answers to the first question to smart glasses 301, generates a chat log containing the first question and its answers, and saves it. If the user then asks a second question, "What are the lyrics to the second song?" GAILLM server 303, based on the second question and the chat log, sends the lyrics of the second song from the 10 best rock songs of the 90s as the answer to the second question to smart glasses 301, and generates a chat log containing the second question and its answers, and saves it.
[0201] Application Example 13
[0202] When the smart glasses 301 are playing the second voice containing the answer to the question, the user can increase or decrease the volume by sliding a finger on the virtual button on the temple of the smart glasses 301 toward the ear.
[0203] Optionally, the user can also use other custom sliding gestures to control the volume, and the custom sliding gesture can be set by the user through the above-mentioned chat parameter setting UI interface.
[0204] In this embodiment, through the interaction between smart glasses, a smart mobile terminal, and the GAILLM server, a GAILLM-based chat function is implemented on the smart glasses, thus providing them with more functionality. The scalability and self-creativity of the generative AI language model further enhance the intelligence and interactivity of the smart glasses. Furthermore, since extensive data processing is not required on the smart glasses, the hardware requirements for the smart glasses are reduced, power consumption is lowered, and the manufacturing cost of the smart glasses is reduced.
[0205] Referring to FIG. 13 , FIG. 13 is a schematic diagram illustrating a flow chart for implementing a method for controlling a smart wearable device based on a generative artificial intelligence large language model according to an embodiment of the present application. This method can be applied to smart wearable devices. The smart wearable devices may include, but are not limited to, smart helmets, smart headphones, smart earrings, smart watches, and smart glasses as shown in FIG. 1 and FIG. 2 . As shown in FIG. 13 , the method includes the following steps:
[0206] S501, activating the chat function of the smart wearable device in response to a first control instruction for activating the chat function;
[0207] S502: Acquire a first voice of a user through a built-in microphone, where the first voice includes a question raised by the user;
[0208] S503: Obtain a second voice containing an answer to the question through a generative artificial intelligence large language model, and play the second voice through a built-in speaker.
[0209] In this embodiment, the method includes two stages. The first stage is the chat function activation stage, and the second stage is the questioning or chatting stage.
[0210] Among them, the first control instruction is used to activate the chat function, which can specifically come from a smart mobile terminal, or be automatically triggered when the smart wearable device is turned on, or be triggered based on the user's operation of the control button of the smart wearable device or obtained through a virtual assistant on the smart wearable device.
[0211] After activating the chat function, the user can directly start asking questions (or chatting), or the user can also control the start and end of asking questions through the control button or through voice control.
[0212] The generative AI language model can be configured on a smart wearable device or on a cloud server. When the model is configured on a cloud server, the smart wearable device can interact with the cloud server, or a smart mobile terminal can interact with the cloud server to obtain a second speech containing the answer to the question using the generative AI language model.
[0213] In this embodiment, the smart wearable device utilizes the generative artificial intelligence big language model to realize the chat function based on the smart wearable device, thereby enabling the smart wearable device to have more functions. Moreover, due to the scalability and self-creativity of the generative artificial intelligence big language model, the intelligence and interactivity of the smart wearable device can be further improved.
[0214] Optionally, in another embodiment of the present application, in response to the first control instruction, activating the chat function of the smart wearable device includes:
[0215] Receiving the first control instruction sent by the smart mobile terminal through the built-in Bluetooth component;
[0216] In response to the first control instruction, activating the chat function and waking up the microphone;
[0217] A second control instruction for disabling the chat function sent by the smart mobile terminal is received through the Bluetooth component, and the chat function and the microphone are disabled in response to the second control instruction.
[0218] Optionally, in another embodiment of the present application, the first control instruction is a first voice instruction, and the method further includes:
[0219] Obtaining the first voice command through a built-in virtual assistant program;
[0220] A second voice command is obtained through the virtual assistant program, and the chat function is disabled in response to the second voice command.
[0221] Optionally, in another embodiment of the present application, obtaining the first voice command through a built-in virtual assistant program includes:
[0222] obtaining user voice through the virtual assistant program;
[0223] If the user's voice contains a preset wake-up word, it is confirmed that the first voice command has been obtained.
[0224] Optionally, in another embodiment of the present application, obtaining the user's first voice through a built-in microphone includes:
[0225] When a user voice containing a preset keyword is acquired through the microphone, a voiceprint is extracted from the user voice;
[0226] The user is authenticated according to the voiceprint, and when the user passes the authentication, the first voice is acquired through the microphone, wherein the preset keyword is used to indicate that the user starts to ask a question.
[0227] Optionally, in another embodiment of the present application, the first control instruction is triggered based on a first preset operation of the user on a button on the smart wearable device, where the button includes a physical button and / or a virtual button based on a touch sensor;
[0228] The acquiring of the first voice of the user through the built-in microphone includes:
[0229] In response to a second preset operation by the user on the button, waking up the microphone and outputting a first prompt tone through the speaker to prompt the user to start asking a question, and acquiring the first voice through the microphone, wherein the second preset operation includes: long pressing the virtual button, short pressing the virtual button, touching the virtual button, tapping the virtual button, sliding on the virtual button, and pressing and holding the physical button, wherein the pressing duration of the short press is shorter than the pressing duration of the long press, and tapping may include, but is not limited to, single-clicking, double-clicking, and other actions;
[0230] Before obtaining the second speech containing the answer to the question through the generative artificial intelligence language model, the method further includes:
[0231] When it is detected that the user ends the second preset operation, or when it is detected that the user performs a third preset operation on the button after ending the second preset operation, or when silence of the first preset duration is detected, the operation of acquiring the first voice is terminated, and a second prompt sound is output to prompt the user that the question is ended, wherein the third preset operation includes any one of: touching the virtual button, tapping the virtual button, and sliding on the virtual button;
[0232] The method further includes:
[0233] The first configuration instruction sent by the smart mobile terminal is received through the built-in Bluetooth component, and the first preset duration is set to the duration indicated by the first configuration instruction.
[0234] Optionally, in another embodiment of the present application, the generative artificial intelligence language model is configured on the GAILLM server, and the second speech containing the answer to the question obtained by the generative artificial intelligence language model includes:
[0235] Converting the first speech into a first text using a built-in speech-to-text engine;
[0236] Sending the first text to the GAILLM server via a built-in wireless communication component, and receiving a second text including the reply sent by the GAILLM server, wherein the second text is obtained by the GAILLM server inputting the first text into the generative artificial intelligence large language model;
[0237] The second text is converted into the second speech through a built-in text-to-speech engine.
[0238] Optionally, in another embodiment of the present application, the method further includes:
[0239] Generate a chat record, and send the chat record and the first text to the GAILLM server through the wireless communication component, so that the generative artificial intelligence language model generates the second text according to the chat record and the first text.
[0240] Optionally, in another embodiment of the present application, the generative artificial intelligence language model is configured on the GAILLM server, and the second speech containing the answer to the question obtained by the generative artificial intelligence language model includes:
[0241] Converting the first speech into a first text using a built-in speech-to-text engine;
[0242] The first text is sent to the smart mobile terminal via a built-in Bluetooth component, and a second text containing the reply is received from the smart mobile terminal, wherein the second text is sent to the GAILLM server via the smart mobile terminal based on the first text and the generative artificial intelligence language model.
[0243] The second text is converted into the second speech through a built-in text-to-speech engine.
[0244] Optionally, in another embodiment of the present application, the generative artificial intelligence language model is configured on the GAILLM server, and the second speech containing the answer to the question obtained by the generative artificial intelligence language model includes:
[0245] The second speech including the answer to the question obtained by the generative artificial intelligence large language model includes:
[0246] Through the built-in Bluetooth component, the first voice is sent to the smart mobile terminal, and the second voice sent by the smart mobile terminal is received, wherein the smart mobile terminal converts the first voice into a first text and sends it to the GAILLM server, and the GAILLM server inputs the first text into the generative artificial intelligence language model to obtain a second text containing the reply and sends it to the smart mobile terminal, and the smart mobile terminal converts the second text into the second voice and sends it to the smart wearable device.
[0247] Optionally, in another embodiment of the present application, after playing the second voice through the speaker, the method further includes:
[0248] If silence of the second preset duration is detected, the smart wearable device is controlled to enter a standby state, the chat function is disabled, the microphone is controlled to enter a dormant state, and a third prompt tone is output to prompt the user that the smart wearable device is about to enter a standby state;
[0249] The second configuration instruction sent by the smart mobile terminal is received through the built-in Bluetooth component, and the second preset duration is set to the duration indicated by the second configuration instruction.
[0250] Optionally, in another embodiment of the present application, after acquiring the user's first voice through the microphone, the method further includes:
[0251] Acquire sensing data of at least one of a built-in positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer;
[0252] The sensing data is sent to the smart mobile terminal through the Bluetooth component, so that the smart mobile terminal sends the sensing data, the device data of the smart mobile terminal and the first text to the GAILLM server, so that the generative artificial intelligence large language model obtains the second text based on the sensing data, the device data of the smart mobile terminal and the first text.
[0253] Optionally, in another embodiment of the present application, the method further includes:
[0254] In response to a fourth preset operation performed by the user on the button for controlling playback volume, playing the second voice at a volume indicated by the fourth preset operation, wherein the fourth preset operation includes: sliding on the virtual button, touching the virtual button, and pressing the physical button;
[0255] In response to the playback speed control instruction sent by the smart mobile terminal and received by the built-in Bluetooth component, the second voice is played at a rate indicated by the playback speed control instruction.
[0256] The embodiments of the present application also provide a non-transitory computer-readable storage medium, which can be provided in the smart glasses or smart wearable devices in the above embodiments. The non-transitory computer-readable storage medium can be the memory 106 in the embodiment shown in Figure 1 above. The computer-readable storage medium stores a computer program, which, when executed by the processor, implements the smart wearable device control method based on the generative artificial intelligence large language model described in the above embodiments. Furthermore, the computer-storable medium can also be various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk, or an optical disk.
[0257] In the several embodiments provided herein, it should be understood that the disclosed smart glasses, control systems, and control methods can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be omitted or not implemented. Furthermore, the connections or direct or communicative connections shown or discussed can be indirect or communicative connections through some interface, device, or module, and can be electrical, mechanical, or other forms.
[0258] It should be noted that for the aforementioned method embodiments, for ease of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.
[0259] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0260] The above is a description of the smart glasses, control system, and control method provided in this application. For those skilled in the art, based on the ideas of the embodiments of this application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A smart glasses based on a generative artificial intelligence large language model, characterized in that: include: Frames, temples, microphones, speakers, processors, and memory; The temple is connected to the frame, and the processor is electrically connected to the microphone, the speaker and the memory; The memory stores one or more programs executable by the processor, wherein the one or more programs include multiple instructions, and the multiple instructions are used to: In response to a first control instruction for activating a chat function, activating a chat function of the smart glasses; Acquire a first voice of a user through the microphone, where the first voice includes a question raised by the user; A second voice including an answer to the question is obtained through a generative artificial intelligence large language model, and the second voice is played through the speaker.
2. The smart glasses according to claim 1, wherein: The smart glasses further include a Bluetooth component electrically connected to the processor, and the plurality of instructions are further used to: receiving the first control instruction sent by the smart mobile terminal through the Bluetooth component; the first control instruction is generated by the virtual assistant program of the mobile smart terminal after obtaining the voice wake-up instruction, or is generated by the operation interface of the mobile smart terminal after detecting the wake-up operation; While activating the chat function, waking up the microphone; A second control instruction for disabling the chat function sent by the smart mobile terminal is received through the Bluetooth component, and the chat function and the microphone are disabled in response to the second control instruction.
3. The smart glasses according to claim 1, wherein: The one or more programs further include a virtual assistant program, and the first control instruction is a first voice instruction; The plurality of instructions are further used to obtain the first voice instruction through the virtual assistant program, wherein the first voice instruction includes a preset first keyword for instructing activation of the chat function; The multiple instructions are also used to obtain, through the virtual assistant program, a second voice instruction containing a preset second keyword for instructing to disable the chat function, and to disable the chat function in response to the second voice instruction.
4. The smart glasses according to claim 1, wherein: The smart glasses also include a button electrically connected to the processor, the button includes a physical button and / or a virtual button based on a touch sensor, and the first control instruction is triggered based on a first preset operation of the user on the button.
5. The smart glasses according to claim 4, wherein: The plurality of instructions are further used to obtain the first voice through the microphone after obtaining a user voice containing a third preset keyword through the microphone, wherein the third preset keyword is used to indicate that the user starts to ask a question; or The multiple instructions are also used to obtain the first voice through the microphone in response to a second preset operation of the user on the button, wherein the second preset operation includes: long pressing the virtual button, short pressing the virtual button, touching the virtual button, tapping the virtual button, sliding on the virtual button set on the temple, and pressing and holding the physical button, wherein the pressing duration of the short press is less than the pressing duration of the long press.
6. The smart glasses according to claim 5, wherein: After the user voice including the third preset keyword is acquired through the microphone, acquiring the first voice through the microphone includes: When a user voice containing a third preset keyword is acquired through the microphone, extracting a voiceprint from the user voice; The user is authenticated according to the voiceprint, and when the user passes the identity authentication, the first voice is acquired through the microphone.
7. The smart glasses according to claim 5, wherein: The plurality of instructions are also used to: Before acquiring the first voice through the microphone, the microphone is awakened, and a first prompt sound is output through the speaker to prompt the user to start asking questions.
8. The smart glasses according to claim 5, wherein: The multiple instructions are also used to terminate the operation of acquiring the first voice and output a second prompt tone to prompt the user to end the question when any of the following events is detected before the second voice containing the answer to the question is obtained through the generative artificial intelligence large language model: The user ends the second preset operation, the user performs a third preset operation on the button after ending the second preset operation, and silence of the first preset duration occurs; The third preset operation includes any one of touching the virtual button, tapping the virtual button, and sliding on the virtual button set on the temple.
9. The smart glasses according to claim 8, wherein: The smart glasses further include a Bluetooth component electrically connected to the processor, and the plurality of instructions are further used to: A first configuration instruction for configuring the first preset duration sent by the smart mobile terminal is received through the Bluetooth component, and the first preset duration is set to the duration indicated by the first configuration instruction.
10. The smart glasses according to claim 1, wherein: The smart glasses further include an indicator light and / or a buzzer electrically connected to the processor, and the plurality of instructions are further used to: Prompt information is outputted through the indicator light and / or the buzzer, and the prompt information is used to prompt the status of the smart glasses, the status includes a working status and an idle status, and the working status includes: starting to pick up voice status, voice picking up status, ending to pick up voice status and voice processing status.
11. The smart glasses according to claim 1, wherein: The generative artificial intelligence large language model is configured on a model server, the smart glasses further include a wireless communication component electrically connected to the processor, and the second voice including the answer to the question obtained by the generative artificial intelligence large language model includes: The first speech is sent to a conversion server through the wireless communication component, so that the first speech is converted into a first text through the conversion server, and the first text is sent to the model server, so that the model server obtains a second text according to the first text through the generative artificial intelligence large language model and sends it to the conversion server, and the second text is converted into the second speech through the conversion server; Receiving, via the wireless communication component, the second voice sent by the conversion server; or, The one or more programs further include a speech-to-text engine and a text-to-speech engine, and the second speech including the answer to the question obtained by the generative artificial intelligence large language model includes: Converting the first speech into a first text by the speech-to-text engine; Sending the first text to the model server through the wireless communication component, and receiving a second text including the reply sent by the model server, wherein the second text is obtained by the model server inputting the first text into the generative artificial intelligence large language model; The second text is converted into the second speech by the text-to-speech engine.
12. The smart glasses according to claim 11, wherein: The multiple instructions are also used to generate a chat record, and send the chat record and the first text to the model server through the wireless communication component, so that the generative artificial intelligence large language model generates the second text based on the chat record and the first text.
13. The smart glasses according to claim 11, wherein: The smart glasses further include at least one of a positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer electrically connected to the processor; The multiple instructions are also used to obtain the sensing data of the at least one device, and send the sensing data of the at least one device and the first text to the model server through the wireless communication component, so that the generative artificial intelligence large language model obtains the second text based on the sensing data and the first text.
14. The smart glasses according to claim 1, wherein: The generative artificial intelligence large language model is configured on a model server, the smart glasses are further configured with a speech-to-text engine and a text-to-speech engine, the smart glasses further include a Bluetooth component electrically connected to the processor, and the second speech including the answer to the question obtained by the generative artificial intelligence large language model includes: Converting the first speech into a first text by the speech-to-text engine; The first text is sent to the smart mobile terminal through the Bluetooth component, and a second text including the reply sent by the smart mobile terminal is received, wherein the second text is sent to the model server through the smart mobile terminal by sending the first text to the model server, and is obtained by the model server based on the first text and the generative artificial intelligence large language model; The second text is converted into the second speech by the text-to-speech engine.
15. The smart glasses according to claim 1, wherein: The generative artificial intelligence large language model is configured on a model server, the smart glasses further include a Bluetooth component electrically connected to the processor, and the second voice including the answer to the question obtained by the generative artificial intelligence large language model includes: Through the Bluetooth component, the first voice is sent to the smart mobile terminal, and the second voice sent by the smart mobile terminal is received, wherein the smart mobile terminal converts the first voice into a first text and sends it to the model server, the model server inputs the first text into the generative artificial intelligence large language model to obtain a second text containing the reply and sends it to the smart mobile terminal, and the smart mobile terminal converts the second text into the second voice and sends it to the smart glasses.
16. The smart glasses according to claim 2, wherein: The multiple instructions are further used to control the smart glasses to enter a standby state, disable the chat function, control the microphone to enter a dormant state, and output a third prompt tone to prompt the user that the smart glasses are about to enter a standby state if silence of a second preset duration is detected after the second voice is played; The multiple instructions are also used to receive, through the Bluetooth component, a second configuration instruction for configuring the second preset duration sent by the smart mobile terminal, and set the second preset duration to the duration pointed to by the second configuration instruction.
17. The smart glasses according to claim 14 or 15, wherein: The smart glasses further include at least one of a positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer electrically connected to the processor; The multiple instructions are also used to obtain the sensing data of the at least one device after obtaining the user's first voice through the microphone, and send the sensing data of the at least one device to the smart mobile terminal through the Bluetooth component, so that the smart mobile terminal sends the sensing data, the device data of the smart mobile terminal and the first text to the model server, so that the generative artificial intelligence large language model obtains the second text based on the sensing data, the device data of the smart mobile terminal and the first text.
18. The smart glasses according to claim 4, wherein: The plurality of instructions are also used to: In response to a fourth preset operation of the user on the button for controlling the volume, the second voice is played at a volume indicated by the fourth preset operation, wherein the fourth preset operation includes: sliding on the virtual button, touching the virtual button, and pressing any one of the physical buttons.
19. The smart glasses according to claim 11, wherein: The smart glasses also include a Bluetooth component electrically connected to the processor; The multiple instructions are further used to receive a language setting instruction sent by the smart mobile terminal through the Bluetooth component, and set the user's language to the target language type pointed to by the language setting instruction, so that the speech-to-text engine converts the first speech into a first text according to the target speech type; The multiple instructions are also used to receive a language self-detection configuration instruction sent by the smart mobile terminal through the Bluetooth component, and activate an automatic language detection function in response to the language self-detection configuration instruction, so that the speech-to-text engine converts the first speech into the first text according to the language type obtained by self-detection.
20. The smart glasses according to claim 2, wherein: The multiple instructions are also used to receive the playback speed control instruction sent by the smart mobile terminal through the Bluetooth component, and play the second voice at the rate indicated by the playback speed control instruction.
21. A smart glasses control system based on a generative artificial intelligence large language model, characterized in that: The system includes smart glasses, a smart mobile terminal and a model server, wherein the smart glasses include a microphone, a speaker and a Bluetooth component; The smart glasses are configured to activate the chat function of the smart glasses in response to a first control instruction for activating the chat function, obtain a first voice of the user through the microphone, and send the first voice to the smart mobile terminal through the Bluetooth component, wherein the first voice includes a question raised by the user; The intelligent mobile terminal is used to convert the first speech into a first text and send the first text to the model server; The model server is used to obtain a second text based on the first text through a generative artificial intelligence large language model and send the second text to the smart mobile terminal, wherein the second text contains an answer to the question; The smart mobile terminal is further used to convert the second text into a second voice and send the second voice to the smart glasses; The smart glasses are further configured to receive the second voice through the Bluetooth component and play the second voice through the speaker.
22. The system of claim 21, wherein: The system also includes a conversion server; The intelligent mobile terminal is further used to send the first voice to the conversion server; The conversion server is further configured to convert the first speech into the first text through a speech-to-text engine, and send the first text to the model server; The model server is further configured to obtain the second text through the generative artificial intelligence large language model according to the first text sent by the conversion server and send the second text to the conversion server; The conversion server is further used to convert the second text into the second voice through a text-to-speech engine, and send the second voice to the smart mobile terminal.
23. The system of claim 21, wherein: The smart glasses are further configured to convert the first speech into the first text using a built-in speech-to-text engine, and send the first text to the model server; The model server is further configured to obtain the second text according to the first text sent by the smart glasses through the generative artificial intelligence large language model and send the second text to the smart glasses; The smart glasses are further configured to convert the second text into the second speech using a built-in text-to-speech engine; or, The system also includes a conversion server; The smart glasses are further used to send the first voice to the conversion server; The conversion server is further configured to convert the first speech into the first text through a speech-to-text engine, and send the first text to the model server; The model server is further configured to obtain the second text through the generative artificial intelligence large language model according to the first text sent by the conversion server and send the second text to the conversion server; The conversion server is also used to convert the second text into the second voice through a text-to-speech engine, and send the second voice to the smart glasses.
24. The system of claim 21, wherein: The smart mobile terminal is also used to adjust the playback rate of the second voice to the target rate indicated by the operation in the user interface of the smart mobile terminal in response to the user's operation for adjusting the playback speed, and send the second voice with the target rate to the smart glasses.
25. The system of claim 21, wherein: The system also includes a chat record storage server; The smart mobile terminal is further used to generate a chat record according to the data sent by the smart glasses under the chat function, associate the chat record with the user's login account, and store the chat record in the smart mobile terminal or the chat record storage server; The smart mobile terminal is further used to send the first text and the chat record to the model server; The model server is further configured to obtain the second text according to the first text and the chat record by using the generative artificial intelligence large language model; The smart mobile terminal is also used to obtain the target chat record corresponding to the query operation according to the user's query operation and export it according to a preset export method.
26. The system of claim 25, wherein: The preset exporting method includes: exporting to a preset social media platform or exporting to a designated device.
27. The system of claim 21, wherein: The smart glasses are further used to obtain sensing data of at least one of a built-in positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer, and send the sensing data and the first voice to the smart mobile terminal through the Bluetooth component; The smart mobile terminal is further used to obtain device data of the smart mobile terminal, and send the sensing data, the device data and the first text to the model server; The model server is also used to obtain the second text according to the sensory data, the device data and the first text through the generative artificial intelligence large language model.
28. A control method for an intelligent wearable device based on a generative artificial intelligence large language model, applied to an intelligent wearable device, characterized in that: The method comprises: In response to a first control instruction for activating a chat function, activating the chat function of the smart wearable device; Acquire a first voice of a user through a built-in microphone, where the first voice includes a question raised by the user; A second voice including an answer to the question is obtained through a generative artificial intelligence large language model, and the second voice is played through a built-in speaker.
29. The method of claim 28, wherein: In response to the first control instruction, activating the chat function of the smart wearable device includes: Receiving the first control instruction sent by the smart mobile terminal through a built-in Bluetooth component; In response to the first control instruction, activating the chat function and waking up the microphone; A second control instruction for disabling the chat function sent by the smart mobile terminal is received through the Bluetooth component, and the chat function and the microphone are disabled in response to the second control instruction.
30. The method of claim 28, wherein: The first control instruction is a first voice instruction, and the method further includes: Obtaining the first voice command through a built-in virtual assistant program; A second voice instruction is obtained through the virtual assistant program, and the chat function is disabled in response to the second voice instruction.
31. The method of claim 30, wherein: The acquiring the first voice instruction through the built-in virtual assistant program includes: Acquiring user voice through the virtual assistant program; If the user voice contains a preset wake-up word, it is confirmed that the first voice command is obtained.
32. The method of claim 28, wherein: The acquiring the first voice of the user through the built-in microphone includes: When a user voice containing a preset keyword is acquired through the microphone, a voiceprint is extracted from the user voice; The user is authenticated according to the voiceprint, and when the user passes the authentication, the first voice is acquired through the microphone, wherein the preset keyword is used to indicate that the user starts to ask a question.
33. The method of claim 28, wherein: The first control instruction is triggered based on a first preset operation of the user on a button on the smart wearable device, where the button includes a physical button and / or a virtual button based on a touch sensor; The acquiring the first voice of the user through the built-in microphone includes: In response to a second preset operation of the user on the button, waking up the microphone and outputting a first prompt sound through the speaker to prompt the user to start asking questions, and acquiring the first voice through the microphone, wherein the second preset operation includes: long pressing the virtual button, short pressing the virtual button, touching the virtual button, tapping the virtual button, sliding on the virtual button, and pressing and holding the physical button, wherein the pressing duration of the short press is shorter than the pressing duration of the long press; Before obtaining the second speech including the answer to the question through the generative artificial intelligence large language model, the method further includes: When any of the following events is detected, the operation of acquiring the first voice is terminated, and a second prompt sound is output to prompt the user to end the question: the user ends the second preset operation, the user performs a third preset operation on the button after ending the second preset operation, and silence of a first preset duration occurs, wherein the third preset operation includes: any one of touching the virtual button, tapping the virtual button, and sliding on the virtual button; The method further comprises: The first configuration instruction sent by the smart mobile terminal is received through the built-in Bluetooth component, and the first preset duration is set to the duration indicated by the first configuration instruction.
34. The method of claim 28, wherein: The generative artificial intelligence large language model is configured on a model server, and the second speech including the answer to the question obtained by the generative artificial intelligence large language model includes: Converting the first speech into a first text by using a built-in speech-to-text engine; The first text is sent to the model server through a built-in wireless communication component, and a second text including the reply is received from the model server, wherein the second text is obtained by the model server inputting the first text into the generative artificial intelligence large language model; The second text is converted into the second speech through a built-in text-to-speech engine.
35. The method of claim 34, wherein: The method further comprises: Generate a chat record, and send the chat record and the first text to the model server through the wireless communication component, so that the generative artificial intelligence large language model generates the second text according to the chat record and the first text.
36. The method of claim 28, wherein: The generative artificial intelligence large language model is configured on a model server, and the second speech including the answer to the question obtained by the generative artificial intelligence large language model includes: Converting the first speech into a first text by using a built-in speech-to-text engine; The first text is sent to the smart mobile terminal through the built-in Bluetooth component, and a second text including the reply sent by the smart mobile terminal is received, wherein the second text is sent to the model server through the smart mobile terminal by sending the first text to the model server, and is obtained by the model server based on the first text and the generative artificial intelligence large language model; Converting the second text into the second speech by using a built-in text-to-speech engine; or, The second speech including the answer to the question obtained by the generative artificial intelligence large language model includes: Through the built-in Bluetooth component, the first voice is sent to the smart mobile terminal, and the second voice sent by the smart mobile terminal is received, wherein the smart mobile terminal converts the first voice into a first text and sends it to the model server, the model server inputs the first text into the generative artificial intelligence large language model to obtain a second text containing the reply and sends it to the smart mobile terminal, and the smart mobile terminal converts the second text into the second voice and sends it to the smart wearable device.
37. The method of claim 28, wherein: After playing the second voice through the speaker, the method further includes: If silence of the second preset time length is detected, the smart wearable device is controlled to enter a standby state, the chat function is disabled, the microphone is controlled to enter a dormant state, and a third prompt tone is output to prompt the user that the smart wearable device is about to enter a standby state; The second configuration instruction sent by the smart mobile terminal is received through the built-in Bluetooth component, and the second preset duration is set to the duration indicated by the second configuration instruction.
38. The method of claim 36, wherein: After acquiring the first voice of the user through the microphone, the method further includes: Acquire sensing data of at least one of a built-in positioning component, an inertial measurement unit sensor, a temperature sensor, a proximity sensor, a humidity sensor, an electronic compass, a timer, a camera, and a pedometer; The sensing data is sent to the smart mobile terminal through the Bluetooth component, so that the smart mobile terminal sends the sensing data, the device data of the smart mobile terminal and the first text to the model server, so that the generative artificial intelligence large language model obtains the second text based on the sensing data, the device data of the smart mobile terminal and the first text.
39. The method of claim 33, wherein: The method further comprises: In response to a fourth preset operation of the user on the button for controlling the playback volume, playing the second voice at a volume indicated by the fourth preset operation, wherein the fourth preset operation includes: sliding on the virtual button, touching the virtual button, and pressing the physical button; In response to the playback speed control instruction sent by the smart mobile terminal and received by the built-in Bluetooth component, the second voice is played at a rate indicated by the playback speed control instruction.
Citation Information
Patent Citations
Dialogue system, and dialogue processing method
CN110503948A
Systems and methods for training and using chatbot
CN112189192A
Question and answer method and device
CN113204623A
Device wakeup method, related device and communication system
CN115691485A
AI intelligent voice control Bluetooth intelligent glasses system
CN212032646U
Cited By
Intelligent glasses voice interaction method and system based on large language model
CN122888977A
