Information search system based on large language model, intelligent glasses and information search method

Abstract

The invention discloses an information search system based on a large language model, intelligent glasses and an information search method. The system comprises an intelligent glasses system and a model server, and a large language model is configured on the model server. The intelligent glasses system obtains a first voice containing a to-be-queried user question, obtains prompt information, converts the first voice into a first text, and sends the prompt information and the first text to the model server. And the model server determines characteristics of the user question according to the prompt information and the first text through a large language model (LLM), obtains a second text containing question reply according to the characteristics, and sends the second text to the intelligent glasses system. And the intelligent glasses system converts the second text into a second voice and plays the second voice. Real-time information search based on LLM is realized in the intelligent glasses system.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of smart glasses, and in particular to an information search system, smart glasses, and an information search method based on a large language model. Background Art

[0002] With the development of computer technology, smart glasses are becoming more and more popular. However, existing smart glasses are expensive, and in addition to their own functions as smart glasses, they usually only have the functions of listening to music and making or receiving calls. The functions are relatively simple and the degree of intelligence is relatively low.

[0003] If search functions can be added to smart glasses, they will be more popular with users. However, how to improve the accuracy, completeness and real-time nature of search information when searching is an issue worthy of in-depth study. Summary of the invention

[0004] The embodiments of the present application aim to provide an information search system, smart glasses and information search method based on a large language model, which are used to improve the real-time, completeness, convenience and accuracy of information search based on the smart glasses system, and improve the intelligence and interactivity of the smart glasses system.

[0005] On the one hand, an embodiment of the present application provides an information search system based on a large language model, including a smart glasses system and a model server, wherein the large language model is configured on the model server;

[0006] The smart glasses system is used to obtain a first voice containing a user question to be queried, obtain prompt information, convert the first voice into a first text, and send the prompt information and the first text to a model server;

[0007] The model server is used to determine the characteristics of the user question according to the prompt information and the first text through the large language model, obtain a second text containing an answer to the question according to the characteristics, and send the second text to the smart glasses system;

[0008] The smart glasses system is also used to convert the second text into a second voice and play the second voice.

[0009] In one aspect, an embodiment of the present application further provides a smart glasses based on a large language model, the smart glasses comprising: a frame, temples, at least one microphone, at least one speaker, a processor, and a memory, wherein the temples are connected to the frame, and the processor is connected to the at least one microphone, the at least one speaker, and the memory;

[0010] The memory stores a computer program executable by the processor, wherein the computer program includes a plurality of instructions. When the plurality of instructions are executed by the processor, the processor:

[0011] Acquiring, through the at least one microphone, a first voice including a user question to be queried;

[0012] Converting the first speech into a first text by a speech-to-text engine;

[0013] Determine the characteristics of the user's question according to the prompt information and the first text through a large language model, and obtain a second text containing an answer to the question according to the characteristics;

[0014] The second text is converted into a second voice through a text-to-speech engine, and the second voice is played through the at least one speaker.

[0015] On the one hand, an embodiment of the present application further provides an information search method based on a large language model, which is applied to a smart mobile terminal. The information search method includes:

[0016] Receiving a first voice including a user question to be queried sent by the smart wearable device, and converting the first voice into a first text using a voice-to-text engine, wherein the voice-to-text engine is configured in the smart mobile terminal or the voice-to-text server;

[0017] Determining the characteristics of the user's question according to the prompt information and the first text through a large language model, and obtaining a second text containing an answer to the question according to the characteristics, wherein the large language model is configured on the smart mobile terminal or the model server;

[0018] The second text is converted into a second voice through a text-to-speech engine, and the second voice is sent to the smart wearable device for playback, wherein the text-to-speech engine is configured in the smart mobile terminal or the text-to-speech server.

[0019] The embodiments of the present application implement natural language speech-based information search in smart glasses (or smart wearable devices) by utilizing a large language model, thereby improving the real-time, completeness, convenience and accuracy of information search based on smart glasses (or smart wearable devices) or in collaboration with smart mobile terminals. Moreover, due to the scalability and self-creativity of the large language model, the intelligence and interactivity of the smart glasses (or smart wearable devices) can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of the structure of an information search system based on a large language model provided in one embodiment of the present application;

[0022] Figure 2 An architectural intent of an information search system based on a large language model provided by an embodiment of the present application;

[0023] Figure 3 Another architectural intention of an information search system based on a large language model provided in an embodiment of the present application;

[0024] Figure 4 Another architectural intention of an information search system based on a large language model provided in an embodiment of the present application;

[0025] Figure 5 Another architectural intention of an information search system based on a large language model provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the external structure of smart glasses based on a large language model provided in one embodiment of the present application;

[0027] Figure 7 for Figure 6 A schematic diagram of the internal structure of the smart glasses shown;

[0028] Figure 8 A flowchart for implementing an information search method based on a large language model provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0031] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0032] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0033] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of an information search system based on a large language model provided in an embodiment of the present application. Figure 1 As shown, the information search system 100 includes: a smart glasses system 110 and a model server 120, wherein the model server 120 is configured with a large language model (LLM);

[0034] The smart glasses system 110 is used to obtain a first voice including a user question to be queried, obtain prompt information, convert the first voice into a first text, and send the prompt information and the first text to the model server 120;

[0035] The model server 120 is used to determine the characteristics of the user's question according to the prompt information and the first text through the large language model, obtain a second text containing an answer to the question according to the characteristics, and send the second text to the smart glasses system 110;

[0036] The smart glasses system 110 is further configured to convert the second text into a second voice and play the second voice.

[0037] The characteristics of user questions refer to whether the answer to the user question is instantaneous, which may include: instantaneous (or real-time) and non-instantaneous (or non-real-time). The answer to the user question with the characteristics of instantaneity is irregular, regular or irregular, or changes instantly, such as questions related to stock prices, news, weather of the day, current time, etc. The answer to the user question with the characteristics of non-instantaneity is fixed, long-term unchanged, or not easily changed, such as questions related to humanities, geography, and history.

[0038] The above-mentioned large language model includes: a generative artificial intelligence large language model (GAILLM) and / or a multimodal large language model (MLLM).

[0039] The generative artificial intelligence large language model may be, for example, but not limited to: ChatGPT of Open AI, Bard of Google, and other models with similar functions. The multimodal large language model may be, for example, but not limited to: BLIP-2, LLaVA, MiniGPT-4, mPLUG-Owl, LLaMA-Adapter-v2, Otter, Multimodal-GPT, InstructBLIP, VisualGLM-6B, PandaGPT, LaVIN, and other models with similar functions.

[0040] The prompt information is used to prompt the large language model how to obtain an answer to the user's question.

[0041] Optionally, in other implementations of the present application, the model server 120 is further used to: extract multiple keywords indicating the user's intention from the first text through the large language model, and obtain at least one sub-question corresponding to the multiple keywords;

[0042] Determine the characteristic according to the prompt information, the multiple keywords and the at least one sub-question, and determine whether it is necessary to search through a tool according to the characteristic;

[0043] If not, searching a preset knowledge database for first answer information matching the at least one sub-question, and generating the second text based on the first answer information; and

[0044] If necessary, determine at least one corresponding search tool and search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform a search operation by using the search parameters, obtain second reply information matching the at least one sub-question based on the searched information, and generate the second text based on the second reply information.

[0045] Specifically, the answer to the question can be obtained by searching a preset knowledge database or searching from a third-party network platform using a third-party tool. According to the characteristics of the user's question, the corresponding method of obtaining the answer to the question can be determined. In addition, the model server 120 can also be configured with multiple third-party tool applications, such as but not limited to: various search engines, web crawlers, vector database retrievers, weather, clocks, etc. Various types of knowledge information are stored in the preset knowledge database, such as but not limited to information related to astronomy, geography, history, humanities, religion, etc. The knowledge database can be specifically configured in the model server 120, or it can also be configured in other cloud servers.

[0046] When the characteristic is non-instantaneous, it means that the answer to the question can be searched from a preset knowledge database, so based on the at least one sub-question, the preset knowledge database is searched to obtain first answer information matching the at least one sub-question, and the second text is generated based on the first answer information.

[0047] When the characteristic is instant, it means that the answer to the question needs to be searched from a third-party network platform through a third-party tool. Therefore, based on the prompt information, the multiple keywords and the at least one sub-question, at least one search tool and search parameters required to obtain the answer to the question are determined, and the at least one search tool is called to perform a search operation by using the search parameters. Second answer information matching the at least one sub-question is obtained based on the searched information, and the second text is generated based on the second answer information.

[0048] Optionally, in other embodiments of the present application, the model server 120 is further used to convert the searched information into text information through the large language model, cut the text information into text blocks and perform embedding calculation to generate text block vectors, and store the generated vectors in a vector database;

[0049] The model server is also used to retrieve the second reply information from the vector database through the large language model.

[0050] Specifically, the large language model can use retrieval-augmented generation (RAG) to convert the searched information into text information, and then cut the text information into text blocks and perform embedding calculations to generate text block vectors, and store the generated vectors in a vector database, and then retrieve the second reply information from the vector database.

[0051] Optionally, in other implementations of the present application, the model server 120 is further configured to generate a user's session record based on the first text and the second text and store the record in a historical session database;

[0052] The model server 120 is further configured to search the historical conversation records associated with the user from the historical conversation database by using the large language model when it is determined that a search is required by a tool, and obtain context information of the user's question according to the searched historical conversation records;

[0053] When the context information is related to at least one of the multiple keywords, obtaining the second reply information by searching the vector database;

[0054] When the context information is not relevant to the multiple keywords, the corresponding at least one search tool and the search parameter are determined based on the prompt information, the multiple keywords and the at least one sub-question, and the search operation is performed by calling the at least one search tool using the search parameter, and the second reply information is obtained based on the searched information.

[0055] Optionally, in other embodiments of the present application, after playing the second voice, the smart glasses system 110 is further configured to obtain a third voice including a user instruction, convert the third voice into a third text, and send the third text to the model server 120;

[0056] The model server 120 is further configured to execute the task indicated by the user instruction based on the question answer by using the large language model, generate a fourth text including execution result information of the task, and send the fourth text to the smart glasses system 110;

[0057] The smart glasses system 110 is further configured to convert the fourth text into a fourth voice and play the fourth voice.

[0058] Optionally, in other implementations of the present application, the information search system 100 further includes a data processing server 130 ( Figure 1 not shown);

[0059] The smart glasses system 110 is further configured to obtain the prompt information through the data processing server 130, convert the first voice into the first text, and convert the second text into the second voice;

[0060] The smart glasses system 110 is further configured to convert the third voice into the third text, and convert the fourth text into the fourth voice, through the data processing server 130 .

[0061] Optionally, in other implementations of the present application, the smart glasses system 110 may include: smart glasses 111 .

[0062] Optionally, in other embodiments of the present application, the smart glasses system 110 may include: smart glasses 111 and a smart mobile terminal 112;

[0063] The smart glasses 111 are used to obtain the first voice and send it to the smart mobile terminal 112, play the second voice, obtain the third voice and send it to the smart mobile terminal 112, and play the fourth voice;

[0064] The intelligent mobile terminal 112 is used to obtain the prompt information, convert the first voice into the first text, and send the prompt information and the first text to the model server 120, and convert the third voice into the third text and send it to the model server 120;

[0065] The model server 120 is further configured to send the second text to the smart mobile terminal 112, and send the fourth text to the smart mobile terminal 112;

[0066] The smart mobile terminal 112 is further configured to convert the second text into the second voice and send the second voice to the smart glasses 111 , and to convert the fourth text into the fourth voice and send the fourth voice to the smart glasses 111 .

[0067] The structure and functions of the smart glasses 111 may be specifically referred to as follows: Figure 6 and Figure 7 The relevant descriptions in the illustrated embodiment will not be repeated here.

[0068] The intelligent mobile terminal 112 may include, but is not limited to: a cellular phone, a smart phone, other wireless communication devices, a personal digital assistant, an audio player, other media players, a music recorder, a video recorder, a camera, other media recorders, an intelligent radio, a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), a moving picture expert group (MPEG-1 or MPEG-2) audio layer 3 (MP3) player, a digital camera, and other intelligent devices that can process data on the move. The intelligent mobile terminal may also be installed with an Android, iOS, or other operating system.

[0069] Optionally, in other embodiments of the present application, the smart glasses 111 or the smart mobile terminal 112 are also used to generate prompt information through a prompt information generator, wherein the prompt information generator is configured in the smart glasses 111 or the smart mobile terminal 112 or the data processing server 130.

[0070] Optionally, in other embodiments of the present application, the prompt information includes multiple application examples, and the large language model determines whether it is necessary to search through a tool and determines the corresponding at least one search tool and the search parameters based on the multiple keywords and the at least one sub-question by learning the multiple application examples.

[0071] The multiple application examples are application cases or examples in different application scenarios. By learning the multiple application examples, the large language model can know which operations or tasks need to be performed based on the text sent by the smart glasses system 110, which information needs to be obtained, and which information needs to be fed back to the smart glasses system 110.

[0072] Optionally, in other embodiments of the present application, the smart glasses 111 or the smart mobile terminal 112 are also used to respond to the user's configuration instructions, obtain the configuration information indicated by the configuration instructions, and generate the prompt information according to the configuration information through the prompt information generator, and the configuration information includes: the large language model, the tool, the information source, the data processing method, and at least one configuration parameter of the retrieval conditions of the vector database.

[0073] The model server 120 may be configured with multiple different candidate large language models. The model server 120 is also used to determine the large language model from the different candidate large language models based on the prompt information, input the prompt information and the first text into the large language model, and obtain the second text output by the large language model, wherein the large language model also configures its own parameters based on the configuration information.

[0074] Specifically, the present application provides extensive configuration flexibility, allowing users to configure the large language model used for interaction (i.e., users can select at least one of the preset large models such as ChatGPT 3.5turbo, ChatGPT 4turbo, Claude 3, Germini, etc.), information sources (e.g., Google, Wikipedia, or a specified website), available third-party tools (e.g., search engines, web crawlers, vector database retrievers, weather forecasts, clocks, etc.), data processing procedures (e.g., splitting, embedding, storage), and retrieval conditions of the vector database (e.g., maximum number of documents, minimum cosine similarity, or use of multiple queries) to meet the different needs of users. This flexibility ensures that the present application can adapt to application requirements of various complexities and achieve fine-tuning of optimized performance.

[0075] Optionally, the data processing server 130 may be a single server or a server cluster composed of multiple servers. When the data processing server 130 is a server cluster composed of multiple servers, in this embodiment, the data processing server 130 may specifically include: a speech-to-text server 131 configured with a speech-to-text engine, a text-to-speech server 132 configured with a text-to-speech engine, a prompt server 133 configured with the prompt information generator, a knowledge storage server 134 configured with the knowledge database, a vector server 135 configured with the vector database, and a log server 136 configured with the historical conversation database. Figure 1 (not shown)

[0076] The smart glasses system 110 is further configured to send the first speech to the speech-to-text server 131, so that the speech-to-text server 131 converts the first speech into the first text through a speech-to-text engine and sends the first text to the smart glasses system 110;

[0077] The smart glasses system 110 is further configured to send the second text to the text-to-speech server 132, so that the text-to-speech server 132 converts the second text into the second speech through the speech-to-text engine and sends the second speech to the smart glasses system 110;

[0078] The smart glasses system 110 is further configured to send a notification message to the vector server 135 after the user finishes the query;

[0079] The smart glasses system 110 is further used to obtain the prompt information through the prompt server 133;

[0080] The vector server 135 is further configured to clear all vector data associated with the user in the vector database according to the notification information.

[0081] Specifically, the above information search system can have two different system architectures. Figure 2 and Figure 3 .like Figure 2 As shown in the first system architecture, the information search system 100 includes: smart glasses 111 and a cloud server 220, wherein the smart glasses 111 and the cloud server 220 exchange data via a long-distance wireless network such as WiFi / 3 / 4 / 5G. Figure 3As shown, in the second system architecture, the information search system 100 includes: smart glasses 111, a smart mobile terminal 112 and a cloud server 330, wherein the smart glasses 111 and the smart mobile terminal 112 interact with each other via a short-range low-power wireless network such as Bluetooth, and the smart mobile terminal 112 interacts with the cloud server 330 via a long-range wireless network such as WiFi / 3 / 4 / 5G.

[0082] The cloud server 220 or 330 may be a single server that integrates a large language model, a prompt information generator, the above-mentioned various engines and the above-mentioned various databases, or may be a server cluster composed of multiple servers. Figure 2 As shown, the cloud server 220 may include: a data processing server 130 and a model server 120 .

[0083] As an extension, Figure 4 As shown, the information search system can also have a third system architecture, in which the information search system 100 includes: smart glasses 111, smart mobile terminals 112 and data processing servers 130. The data processing server 130 includes: a model server 120, a speech-to-text server 131, a text-to-speech server 132, a prompt server 133, a knowledge storage server 134, a vector server 135 and a log server 136. The smart glasses 111 and the smart mobile terminals 112 exchange data via short-range low-power wireless networks such as Bluetooth. The smart glasses 111 or the smart mobile terminals 112 exchange data with the data processing server 130 via long-range wireless networks such as WiFi / 3 / 4 / 5G.

[0084] The information search process of the above-mentioned information search system 100 will be described in detail below in conjunction with Examples 1 to 3 respectively applied to three different scenarios.

[0085] Example 1 (weather query)

[0086] In this application example 1, assuming that the user's question is "What is the weather like today in Redmond, Oregon?", the above information search system 100 will perform the following operations:

[0087] 1. The smart glasses system 110 picks up the query voice from the microphone and converts the query voice into text using a speech-to-text (STT) engine;

[0088] 2. The smart glasses system 110 sends the text and corresponding prompt information from the STT engine to the large language model LLM configured in the model server 120 to obtain the answer to the user's question;

[0089] 3. LLM performs the following operations based on the received prompt information and text to obtain a question answer to the user's question;

[0090] 3-1. Understanding the query

[0091] (1) Get the topic (keyword 1): weather;

[0092] (2) Get specific details (keyword 2): location (Redmond, Oregon) and date (today).

[0093] 3-2. Decomposition query

[0094] Get sub-questions:

[0095] (1) What is the current weather in Redmond, Oregon?

[0096] (2) What is the weather forecast for Redmond, Oregon for the rest of the day?

[0097] 3-3. Determine whether to use a tool (LLM will collect information related to the characteristics of the user's question, such as the above keywords "today", "weather", "remaining time" and other information, and decide whether to use a tool based on the collected information. In this application example 1, the question about today's weather is immediate, so a tool is needed)

[0098] (1) Select the search tool: "Weather Forecast" tool;

[0099] (2) Specify search parameters: location (Redmond, Oregon), date (today).

[0100] 3-4. Call the search tool to perform the search operation

[0101] Calls the Weather Forecast tool to get weather data (results from the Weather Forecast tool) using the specified search parameters "Redmond, Oregon, today".

[0102] 3-5. General Information

[0103] Filter and extract useful information from the weather data returned by the Weather Forecast tool, such as current weather data and weather forecast data.

[0104] (1) Current weather data and weather forecast data are combined into detailed answers to questions;

[0105] (2) Includes temperature, humidity, wind speed and major weather events.

[0106] 4. The LLM sends a text containing the answer to the question to the smart glasses system 110, wherein the answer to the question clearly and in detail presents the weather information;

[0107] 5. The smart glasses system 110 converts the detailed weather information (i.e., the answer to the question) in the received text into speech through a Text-To-Speech (TTS) engine, and then outputs it to the speaker of the smart glasses 111 for playback.

[0108] A specific answer to the above question could be, for example: "Today's weather in Redmond, Oregon is as follows: the current temperature is 25°C, the humidity is 60%, and the wind speed is 10 kilometers per hour. It is expected to be sunny most of the day, with a small chance of rain in the afternoon."

[0109] Example 2 (Stock Price Query)

[0110] In this application example 2, assuming that the user's question is "What is the current stock price of Nvidia NVDA?", the above information search system 100 will perform the following operations:

[0111] 1. The smart glasses system 110 picks up the voice used to query the above user question through the microphone of the smart glasses, and converts the voice into text using the STT engine;

[0112] 2. The smart glasses system 110 sends the text and corresponding prompt information from the STT engine to the large language model LLM configured in the model server 120 to obtain the answer to the user's question;

[0113] 3. LLM performs the following operations based on the received prompt information and text to obtain a question answer to the user's question;

[0114] 3-1. Understanding the query

[0115] (1) Obtain the main topic (keyword 1): current stock price;

[0116] (2) Get specific details (keyword 2): Company (Nvidia, NVDA).

[0117] 3-2. Decomposition query

[0118] Get sub-question: What is Nvidia's current stock price?

[0119] 3-3. Determine whether to use a tool (LLM will collect information related to the characteristics of the user's question, such as the "stock price" and "current" information in the above keywords, and decide whether to use a tool based on the collected information. In this application example 2, the question about the current stock price is immediate, so a tool is needed)

[0120] (1) Select search tools: Search engine tools

[0121] (2) Specify search parameters: query ("Nvidia's current stock price"), global relevance

[0122] 3-4. Call the search tool to perform the search operation

[0123] A search engine tool (e.g., Google API) is called with specified search parameters ("query('Nvidia's current stock price'), global relevance" above) to obtain the latest stock price data.

[0124] 3-5. Result Embedding

[0125] Search results are embedded into vectors and stored in a vector database, and documents are split and embedded into vectors and then stored in the vector database.

[0126] Specifically, the result information searched by the search engine tool is converted into text information, the text information is cut into text blocks and embedded calculation is performed to generate text block vectors, and then the generated vectors are stored in a vector database.

[0127] Preferably, because the user question is immediate, in order to save space and improve the utilization of storage space, after the conversation ends, all data related to the conversation in the vector database is released.

[0128] 3-6. Data Retrieval

[0129] (1) Extract relevant stock price information from the vector database and generate a question answer based on the extracted information, where only documents relevant to the context of the user's question are retrieved;

[0130] (2) Ensure that data is up-to-date and accurate.

[0131] 4. LLM sends a text containing the answer to the question to the smart glasses system 110, wherein the answer to the question clearly and in detail presents the stock price information about Nvidia;

[0132] Specific information may include, but is not limited to: the date and time of the last update of NVIDIA's stock price and any significant market trends.

[0133] 5. The smart glasses system 110 converts the stock price information in the received text (from LLM) into speech through the TTS engine, and then outputs it to the speakers of the smart glasses 111 for playback.

[0134] An example of a response to the above question could be: "Based on the latest data, as of 1:00 p.m. on July 29, 2024, the current stock price of Nvidia (NVDA) is $450.00. Please note that stock prices are subject to change and may vary throughout the trading day."

[0135] Furthermore, if the user raises a new demand after listening to the voice answer to the question, the smart glasses system 110 can also use LLM to perform corresponding operations according to the new demand.

[0136] For example, after playing a voice containing the searched Nvidia stock price information, if it is detected that the user has issued a voice containing a buy / sell Nvidia stock instruction, the smart glasses system 110 uses STT to convert the voice into text, and then sends the converted text to LLM. Based on the previously searched Nvidia stock price information, LLM calls a third-party stock trading software to execute the task of buying / selling Nvidia stock, generates a fourth text containing the execution result information of the task and sends it to the smart glasses system 110, so that the smart glasses system 110 uses TTS to convert the fourth text into a fourth voice and play it. Among them, the execution result information can include but is not limited to: notification information on the success or failure of buying / selling Nvidia stock, Nvidia stock buying price / selling price, buying / selling time, etc.

[0137] In another application example of the present application, the user can also query flights for a certain day through the smart glasses system 110. After playing the query results, if the user issues a voice command to book a flight, the smart glasses system 110 can also call a dedicated third-party booking tool through LLM to execute the booking task.

[0138] Example 3 (historical query)

[0139] In this application example 3, assuming that the user's question is "Why did the Roman Empire decline?", the above information search system 100 will perform the following operations:

[0140] 1. The smart glasses system 110 picks up the query voice from the microphone and converts the query voice into text using the STT engine;

[0141] 2. The smart glasses system 110 sends the text and corresponding prompt information from the STT engine to the large language model LLM configured in the model server 120 to obtain the answer to the user's question;

[0142] 3. LLM performs the following operations based on the received prompt information and text to obtain a question answer to the user's question;

[0143] 3-1. Understanding the query

[0144] Get topic (keyword 1): The decline of the Roman Empire

[0145] Get detailed information (keyword 2): historical reasons for the fall

[0146] 3-2. Decomposition query

[0147] Get sub-questions:

[0148] (1) What was the main reason for the fall of the Roman Empire?

[0149] (2) When did the Roman Empire fall?

[0150] 3-3. Determine whether to use a tool (LLM will collect information related to the characteristics of the user's question, such as the above keywords "Rome", "destruction", "decline", "history", "reasons", etc., and decide whether to use a tool based on the collected information. The question about the reasons for the decline of the Roman Empire in this application example 3 is not immediate, so there is no need to use a tool)

[0151] Be sure to leverage existing knowledge bases of historical events to get answers to questions.

[0152] 3-4. General Information

[0153] The key reasons and timeline of the decline of the Roman Empire are obtained by searching the historical event knowledge base, and the key reasons and timeline are combined into a coherent answer to the question.

[0154] 4. LLM sends a text containing the answer to the question to the smart glasses system 110, where the answer to the question clearly and in detail presents the time of the decline of the Roman Empire and the various factors that led to the decline.

[0155] The answer to the above question could be as follows: "The decline of the Roman Empire was a complex historical event with multiple contributing factors. The Western Roman Empire officially fell in 476 AD when the last Roman emperor, Romulus Augustus, was deposed by the Germanic chieftain Odoacer. The key factors that led to its decline are mainly the following: 1. Economic decline: Heavy taxation, reliance on slave labor and economic recession weakened the empire; 2. Military problems: The Roman legions faced constant threats from invading barbarian tribes and struggled to maintain their vast borders; 3. Political corruption: Poor leadership, political instability and corruption weakened the government's ability to manage the empire; 4. Social problems: Declining civic pride and public services, as well as a widening gap between the rich and the poor, led to social unrest; 5. External pressure: Continued invasions by various barbarian groups, including the Visigoths, Vandals and Huns, put tremendous pressure on the empire's defenses. These factors ultimately led to the final collapse of the Western Roman Empire, marking the end of Ancient Rome and the beginning of the Middle Ages.

[0156] 5. The smart glasses system 110 converts the received text responses (from the LLM) into speech through the TTS engine, and then outputs it to the speakers of the smart glasses 111 for playback.

[0157] For ease of understanding, the following will be Figure 5 Taking the architecture shown in the figure as an example, the above-mentioned information search system based on the large language model is further explained. Figure 5 As shown, the audio input from the user picked up by the smart glasses 111 is transmitted to the STT engine so as to be converted into text by the STT engine. Afterwards, the text and prompt information are sent by the smart glasses 111 to the data processing server 130.

[0158] The data processing server 130 may be a single server, or a distributed server cluster consisting of multiple servers such as the above-mentioned model server 120, speech-to-text server 131, text-to-speech server 132, prompt server 133, knowledge storage server 134, vector server 135, and log server 136. It can be understood that in actual applications, according to actual needs, the data processing server 130 may include some or all of the above-mentioned servers, or may also include more servers with other functions.

[0159] The data processing server 130 can provide users with personalized information search services, assign an associated account with a customized virtual assistant to the user on the server side, and use the associated account to configure different large language models, tools, information sources, knowledge databases, data processing methods, and one or more of the retrieval conditions of the vector database for different users' virtual assistants based on the user's customized operations, so as to meet the different search needs of different users, thereby making the search more targeted, flexible and accurate.

[0160] Specifically, the virtual assistant can allow each LLM to interact with an external API (Application Programming Interface), access a user-customized large language model (LLM) through, for example, an LLM Wrapper, and use the LLM to identify the user's intention based on the text and the prompt information and determine whether to retrieve the answer to the question from a preset knowledge database. Among them, by using the LLM Wrapper, many large language models can be accessed, such as: OpenAI, Cohere, Hugging Face, etc.

[0161] If the result output by the LLM is that the answer to the question can be retrieved from the knowledge database, the virtual assistant will retrieve the answer to the question from the knowledge database, which can shorten the search time and improve the response speed.

[0162] Otherwise, it is determined that an online search is required, and the virtual assistant runs the tools provided by the information search service, such as a search engine, to obtain real-time data. The retrieved real-time data will be split and embedded into vectors, and then stored in a vector database. The LLM obtains relevant information by retrieving the vector database and returns it to the virtual assistant. This process is iterated until the virtual assistant accumulates enough information to reply to the user. The result information output by the LLM may include instruction information for calling multiple tools, and the virtual assistant can run multiple tools simultaneously according to the instruction information to collect more information in parallel. The virtual assistant's reply will be provided to the user in a streaming manner to enhance the user experience.

[0163] Finally, the text-based responses are converted to speech by TTS and read aloud by the smart glasses. In addition, the chat history is saved to provide the virtual assistant with contextual awareness throughout the conversation.

[0164] In the embodiments of the present application, the smart glasses system utilizes a large language model to achieve real-time information search based on natural language speech, thereby improving the real-time, convenience and accuracy of information search based on the smart glasses system, and due to the scalability and self-creativity of the large language model, the intelligence and interactivity of the smart glasses system can be further improved.

[0165] See also Figure 6 and Figure 7 , Figure 6 A schematic diagram of the external structure of smart glasses based on a large language model provided in an embodiment of the present application, Figure 7 for Figure 6 The internal structure diagram of smart glasses is shown in FIG. Figure 6 and Figure 7 As shown, the smart glasses 111 include: a frame 1111, at least one temple 1112 (for ease of understanding, Figure 6 1114 ), at least one microphone 1113, at least one speaker 1114, a processor 1115 and a memory 1116, wherein the temple 1112 is connected to the frame 1111, and the processor 1115 is connected to at least one microphone 1113, at least one speaker 1114 and a memory 1116. It can be understood that Figure 6 and Figure 7 This is just an example. In actual applications, the smart glasses 111 may also be specifically compared to Figure 6 and Figure 7 More or fewer components or elements may be shown.

[0166] The memory 1116 stores a computer program that can be executed by the processor 1115. The computer program includes multiple instructions. When the multiple instructions are executed by the processor 1115, the processor 1115:

[0167] Acquiring, through at least one microphone 1113, a first voice including a user question to be queried;

[0168] Converting the first speech into a first text by a speech-to-text engine;

[0169] Determine the characteristics of the user's question according to the prompt information and the first text through a large language model, and obtain a second text containing an answer to the question according to the characteristics;

[0170] The second text is converted into a second voice through a text-to-speech engine, and the second voice is played through at least one speaker 1114 .

[0171] Optionally, in other embodiments of the present application, when the multiple instructions are executed by the processor 1115, the processor 1115 is also caused to: extract multiple keywords for indicating user intent from the first text through the large language model, and obtain at least one sub-question corresponding to the multiple keywords; obtain the prompt information through the prompt information generator, and determine the characteristic based on the prompt information, the multiple keywords and the at least one sub-question, and determine whether it is necessary to search through the tool based on the characteristic; if not, search for first reply information matching the at least one sub-question from a preset knowledge database, and generate the second text based on the first reply information; and if necessary, determine at least one corresponding search tool and search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform a search operation by using the search parameters, obtain second reply information matching the at least one sub-question based on the searched information, and generate the second text based on the second reply information.

[0172] Optionally, in other embodiments of the present application, when the multiple instructions are executed by the processor 1115, the processor 1115 is further enabled to, after playing the second voice through the at least one speaker 1114, obtain a third voice containing the user instruction through the at least one microphone 1113, and convert the third voice into a third text;

[0173] Based on the question answer, executing the task indicated by the user instruction through the large language model, and generating a fourth text containing execution result information of the task;

[0174] The fourth text is converted into a fourth voice, and the fourth voice is played through at least one speaker 1114 .

[0175] In other embodiments of the present application, the large language model is configured in the smart glasses 111 or the smart mobile terminal 112 or the model server 120, the speech-to-text engine is configured in the smart glasses 111 or the smart mobile terminal 112 or the speech-to-text server 131, the text-to-speech engine is configured in the smart glasses 111 or the smart mobile terminal 112 or the text-to-speech server 132, and the prompt information generator is configured in the smart glasses 111 or the smart mobile terminal 112 or the prompt server 133.

[0176] In other embodiments of the present application, the smart glasses 111 also include a Bluetooth communication component 1117. When the multiple instructions are executed by the processor 1115, the processor 1115 also sends the first voice to the smart mobile terminal 112 through the Bluetooth communication component 1117, and instructs the smart mobile terminal 112: converting the first voice into the first text through the voice-to-text engine, obtaining the second text according to the prompt information and the first text through the large language model configured in the smart mobile terminal 112 or the model server 120, and converting the second text into the second voice through the text-to-speech engine, wherein the prompt information is generated by the smart mobile terminal 112 through a prompt information generator configured in the smart mobile terminal 112 or the prompt server 133; and receiving the second voice sent by the smart mobile terminal 112 through the Bluetooth communication component 1117.

[0177] Furthermore, when the multiple instructions are executed by the processor 1115, the processor 1115 sends the third voice to the smart mobile terminal 112 through the Bluetooth communication component 1117, and instructs the smart mobile terminal 112 to convert the third voice into the third text through the speech-to-text engine, and executes the task indicated by the user instruction based on the question response through the large language model configured in the smart mobile terminal 112 or the model server 120 and generates a fourth text containing the execution result information of the task, and converts the fourth text into the fourth voice through the text-to-speech engine; and, receives the fourth voice sent by the smart mobile terminal 112 through the Bluetooth communication component 1117.

[0178] Optionally, in other embodiments of the present application, the smart glasses 111 also include a wireless communication component 1118. When the multiple instructions are executed by the processor 1115, the processor 1115 is also enabled to send the first voice to the speech-to-text server 131 through the wireless communication component 1118, so that the first voice is converted into the first text through the speech-to-text engine in the speech-to-text server; obtain the prompt information generated by the prompt information generator on the prompt server 133 from the prompt server 133 through the wireless communication component 1118; send the first text and the prompt information to the model server 120, so as to obtain the second text according to the prompt information and the first text through the large language model in the model server 120; and send the second text to the text-to-speech server 132, so as to convert the second text into the second voice through the text-to-speech engine in the text-to-speech server 132.

[0179] Furthermore, when the multiple instructions are executed by the processor 1115, the processor 1115 is also caused to send the third speech to the speech-to-text server 131 through the wireless communication component 1118, so that the third speech is converted into the third text through the speech-to-text engine in the speech-to-text server 131; send the third text to the model server 120, so that the large language model in the model server 120 executes the task indicated by the user instruction based on the question answer and generates a fourth text containing the execution result information of the task; send the fourth text to the text-to-speech server 132, so that the fourth text is converted into the fourth speech through the text-to-speech engine in the text-to-speech server 132.

[0180] Optionally, in other embodiments of the present application, when the multiple instructions are executed by the processor 1115, the processor 1115 is also enabled to convert the searched information into text information through the large language model, cut the text information into text blocks and perform embedding calculations to generate text block vectors, store the generated vectors in a vector database, and retrieve the second reply information from the vector database.

[0181] Optionally, in other embodiments of the present application, when the multiple instructions are executed by the processor 1115, the processor 1115 is also enabled to clear all vector data associated with the user in the vector database after the user ends the query.

[0182] Optionally, in other embodiments of the present application, when the multiple instructions are executed by processor 1115, the processor 1115 is also enabled to generate a session record of the user based on the first text and the second text and store it in a historical session database; when the multiple instructions are executed by processor 1115, the processor 1115 is also enabled to use the large language model: when it is determined that a search is required through a tool, search the historical session database for historical session records associated with the user, and obtain context information of the user's question based on the searched historical session records; when the context information is related to at least one of the multiple keywords, obtain the second reply information by retrieving the vector database; and when the context information is not related to the multiple keywords, determine the corresponding at least one search tool and the search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform the search operation by using the search parameters, and obtain the second reply information based on the searched information.

[0183] Optionally, in other embodiments of the present application, the prompt information includes multiple application examples, and the large language model determines whether it is necessary to search through a tool and determines the corresponding at least one search tool and the search parameters based on the multiple keywords and the at least one sub-question by learning the multiple application examples.

[0184] Optionally, in other embodiments of the present application, when the multiple instructions are executed by the processor 1115, the processor 1115 is also enabled to: respond to the user's configuration instruction, obtain the configuration information indicated by the configuration instruction; generate the prompt information according to the configuration information through the prompt information generator, and the prompt information also includes the configuration information, which includes: configuration parameters of at least one of the large language model, the tool, the information source, the data processing method, and the retrieval conditions of the vector database.

[0185] Furthermore, the smart glasses 111 also include a battery 210 ( Figure 7 ), which is used to provide power support for the electronic components of the smart glasses 111, such as the at least one microphone 1113, the at least one speaker 1114, the processor 1115, the memory 1116, the Bluetooth communication module 1117 and the wireless communication module 1118.

[0186] The electronic components of the above-mentioned smart glasses can be connected through a bus.

[0187] It should be noted that the relationship between the components of the above-mentioned smart glasses can be a replacement relationship or a superposition relationship. That is, all the components in the above-mentioned embodiment can be installed on one smart glass, or a part of the above-mentioned components can be selectively installed according to needs. When it is a replacement relationship, the smart glasses are also provided with a connection interface for peripherals, and the connection interface can be at least one of a PS / 2 interface, a serial interface, a parallel interface, an IEEE1394 interface, a USB (Universal Serial Bus) interface, etc. The function of the replaced component can be realized by a peripheral connected to the connection interface, such as: an external speaker, an external sensor, etc.

[0188] For details about the smart glasses in this embodiment, please refer to Figures 1 to 5 as well as Figure 8 The relevant description in the illustrated embodiment will not be repeated here.

[0189] In the embodiments of the present application, the large language model is used by smart glasses to realize information search based on natural language speech, thereby improving the real-time, completeness, convenience and accuracy of information search based on smart glasses, and due to the scalability and self-creativity of the large language model, the intelligence and interactivity of the smart glasses can be further improved.

[0190] See also Figure 8 , Figure 8 The flowchart of the implementation of the information search method based on the large language model provided in one embodiment of the present application is applied to the smart mobile terminal 112 in the above embodiment. The information search method includes:

[0191] S801, receiving a first voice including a user question to be queried and sent by a smart wearable device, and converting the first voice into a first text using a voice-to-text engine, wherein the voice-to-text engine is configured in the smart mobile terminal or the voice-to-text server;

[0192] S802: Determine the characteristics of the user's question according to the prompt information and the first text through a large language model, and obtain a second text containing an answer to the question according to the characteristics, wherein the large language model is configured in the smart mobile terminal or the model server;

[0193] S803. Convert the second text into a second voice through a text-to-speech engine, and send the second voice to the smart wearable device for playback, wherein the text-to-speech engine is configured in the smart mobile terminal or the text-to-speech server.

[0194] Among them, in this embodiment, the smart wearable device may include but is not limited to: a smart helmet, a smart headset, a smart earring, a smart watch, a smart glasses and other wearable smart devices.

[0195] Optionally, in other implementations of the present application, in step S802, the characteristics of the user's question are determined according to the prompt information and the first text through a large language model, and a second text containing an answer to the question is obtained according to the characteristics, specifically including:

[0196] S8021. Extracting multiple keywords indicating user intention from the first text through the large language model, and obtaining at least one sub-question corresponding to the multiple keywords;

[0197] S8022: Determine the characteristic according to the prompt information, the multiple keywords, and the at least one sub-question, and determine whether a search needs to be performed through a tool according to the characteristic;

[0198] S8023. If not, searching a preset knowledge database for first answer information matching the at least one sub-question, and generating the second text based on the first answer information, wherein the knowledge database is configured in the smart mobile terminal or the knowledge storage server; and

[0199] S8024. If necessary, determine at least one corresponding search tool and search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform a search operation by using the search parameters, obtain second reply information matching the at least one sub-question based on the searched information, and generate the second text based on the second reply information.

[0200] Optionally, in other embodiments of the present application, tasks related to the question answer can also be performed according to the user's subsequent instructions, such as: buying tickets, buying / selling stocks, sharing question answers, generating related articles and publishing them on online social platforms, etc., thereby giving the smart wearable device more functions. Specifically, after sending the second voice to the smart wearable device for playback, the information search method also includes: receiving a third voice containing user instructions sent by the smart wearable device, and using the speech-to-text engine to convert the third voice into a third text; using the large language model, based on the question answer, executing the task indicated by the user instruction, and generating a fourth text containing the execution result information of the task; and converting the fourth text into a fourth voice through the text-to-speech engine, and sending the fourth voice to the smart wearable device for playback.

[0201] Optionally, in other embodiments of the present application, in step S8024, second reply information matching the at least one sub-question is obtained based on the searched information, including: converting the searched information into text information, cutting the text information into text blocks and performing embedding calculations to generate text block vectors, and storing the generated vectors in a vector database, wherein the vector database is configured in the smart mobile terminal or the vector server; and retrieving the second reply information from the vector database.

[0202] Optionally, in other implementations of the present application, the information search method further includes: after the user ends the query, clearing all vector data associated with the user in the vector database.

[0203] Optionally, in other embodiments of the present application, the information search method further includes: generating a session record of the user based on the first text and the second text and storing it in a historical session database, wherein the historical session database is configured in the smart mobile terminal or the log server.

[0204] Before determining the corresponding at least one search tool and search parameter according to the prompt information, the multiple keywords and the at least one sub-question, the information search method further includes: searching the historical conversation records associated with the user from the historical conversation database, and obtaining the context information of the user's question according to the searched historical conversation records; when the context information is related to at least one of the multiple keywords, the second answer information is obtained by searching the vector database. Then, the step of determining the corresponding at least one search tool and search parameter according to the prompt information, the multiple keywords and the at least one sub-question includes: when the context information is not related to the multiple keywords, determining the corresponding at least one search tool and search parameter according to the prompt information, the multiple keywords and the at least one sub-question.

[0205] Optionally, in other embodiments of the present application, the prompt information includes multiple application examples, and the large language model determines whether it is necessary to search through a tool and determines the corresponding at least one search tool and the search parameters based on the multiple keywords and the at least one sub-question by learning the multiple application examples.

[0206] Optionally, in other implementations of the present application, the information search method further includes: obtaining configuration information indicated by the configuration instruction in response to the user's configuration instruction; and sending the configuration information to a prompt information generator, so that the prompt information generator generates the prompt information containing the configuration information, the configuration information including: configuration parameters of at least one of the large language model, the tool, the information source, the data processing method, and the search conditions of the vector database. The prompt information generator can be configured in a smart mobile terminal or a prompt server.

[0207] Optionally, in other embodiments of the present application, before obtaining the second text containing the answer to the question based on the prompt information and the first text through the large language model, the information search method also includes: obtaining the prompt information through the prompt information generator.

[0208] For more details about the above information search method based on large language model, please refer to Figures 1 to 7 The relevant description in the illustrated embodiment will not be repeated here.

[0209] In the embodiments of the present application, information search based on natural language speech is realized through the collaboration between the smart mobile terminal and the smart wearable device and the use of a large language model, thereby improving the real-time, completeness, convenience and accuracy of information search based on the smart mobile terminal and the smart wearable device, and due to the scalability and self-creativity of the large language model, the intelligence and interactivity of the smart wearable device can be further improved.

[0210] The present application also provides a non-transitory computer-readable storage medium, which may be disposed in the smart glasses or smart wearable devices in the above embodiments. The non-transitory computer-readable storage medium may be the above Figure 7 The memory 304 in the illustrated embodiment. The computer readable storage medium stores a computer program, and when the program is executed by the processor, the information search method based on the large language model described in the aforementioned embodiments is implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, etc., which can store program codes.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed information search system, smart glasses and information search method based on a large language model can be implemented in other ways. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the connection or direct connection or communication connection between each other shown or discussed can be an indirect connection or communication connection through some interface, device or module, which can be electrical, mechanical or other forms.

[0212] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0213] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0214] The above is a description of the information search system, smart glasses and information search method based on a large language model provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An information search system based on a large language model, characterized in that: The system includes a smart glasses system and a model server, wherein a large language model is configured on the model server; The smart glasses system is used to obtain a first voice containing a user question to be queried, obtain prompt information, convert the first voice into a first text, and send the prompt information and the first text to a model server; The model server is used to determine the characteristics of the user question according to the prompt information and the first text through the large language model, obtain a second text containing an answer to the question according to the characteristics, and send the second text to the smart glasses system; The smart glasses system is also used to convert the second text into a second voice and play the second voice.

2. The information search system according to claim 1, characterized in that: The model server is also used to: Extracting multiple keywords indicating user intention from the first text through the large language model, and obtaining at least one sub-question corresponding to the multiple keywords; Determine the characteristic according to the prompt information, the multiple keywords and the at least one sub-question, and determine whether it is necessary to search through a tool according to the characteristic; If not, searching a preset knowledge database for first answer information matching the at least one sub-question, and generating the second text based on the first answer information; as well as If necessary, determine at least one corresponding search tool and search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform a search operation by using the search parameters, obtain second reply information matching the at least one sub-question based on the searched information, and generate the second text based on the second reply information.

3. The information search system according to claim 2, characterized in that: The model server is further used to convert the searched information into text information through the large language model, cut the text information into text blocks and perform embedding calculation to generate text block vectors, and store the generated vectors in a vector database; The model server is further used to retrieve the second reply information from the vector database through the large language model.

4. The information search system according to claim 3, characterized in that: The model server is further configured to generate a user's session record based on the first text and the second text and store the record in a historical session database; The model server is further used to use the large language model, When it is determined that a search is required through a tool, searching the historical conversation records associated with the user from the historical conversation database, and obtaining context information of the user's question according to the searched historical conversation records; When the context information is related to at least one of the multiple keywords, obtaining the second reply information by searching the vector database; When the context information is not relevant to the multiple keywords, the corresponding at least one search tool and the search parameters are determined based on the prompt information, the multiple keywords and the at least one sub-question, the search operation is performed by calling the at least one search tool using the search parameters, and the second reply information is obtained based on the searched information.

5. The information search system according to claim 4, characterized in that: After playing the second voice, The smart glasses system is further used to obtain a third voice containing a user instruction, convert the third voice into a third text, and send the third text to the model server; The model server is further configured to execute the task indicated by the user instruction based on the question answer by using the large language model, generate a fourth text containing execution result information of the task, and send the fourth text to the smart glasses system; The smart glasses system is also used to convert the fourth text into a fourth voice and play the fourth voice.

6. The information search system according to claim 5, characterized in that: The information search system also includes a data processing server; The smart glasses system is further configured to obtain the prompt information through the data processing server, convert the first voice into the first text, and convert the second text into the second voice; The smart glasses system is also used to convert the third voice into the third text and convert the fourth text into the fourth voice through the data processing server.

7. The information search system according to claim 6, characterized in that: The smart glasses system comprises: smart glasses; or, The smart glasses system comprises: smart glasses and a smart mobile terminal; wherein the smart glasses are used to: Acquire the first voice and send it to the smart mobile terminal, play the second voice, acquire the third voice and send it to the smart mobile terminal, and play the fourth voice; The smart mobile terminal is used to obtain the prompt information, convert the first voice into the first text, and send the prompt information and the first text to the model server, and convert the third voice into the third text and send it to the model server; The model server is further configured to send the second text to the smart mobile terminal, and send the fourth text to the smart mobile terminal; The smart mobile terminal is further configured to convert the second text into the second voice and send the second voice to the smart glasses, and convert the fourth text into the fourth voice and send the fourth voice to the smart glasses.

8. The information search system according to claim 7, characterized in that: The smart glasses or the smart mobile terminal are further used to generate the prompt information through a prompt information generator, wherein the prompt information generator is configured in the smart glasses or the smart mobile terminal or the data processing server.

9. The information search system according to claim 4, characterized in that: The prompt information includes multiple application examples. The large language model determines whether it is necessary to search through tools and determines the corresponding at least one search tool and the search parameters based on the multiple keywords and the at least one sub-question by learning the multiple application examples.

10. The information search system according to claim 8, characterized in that: The smart glasses or the smart mobile terminal are further used to respond to a configuration instruction of a user, obtain configuration information indicated by the configuration instruction, and generate the prompt information according to the configuration information through the prompt generator, wherein the configuration information includes: configuration parameters of at least one of the large language model, the tool, the information source, the data processing method, and the search condition of the vector database; The model server is configured with multiple different candidate large language models. The model server is also used to determine the large language model from the different candidate large language models according to the prompt information, input the prompt information and the first text into the large language model, and obtain the second text output by the large language model, wherein the large language model also configures its own parameters according to the configuration information.

11. The information search system according to claim 10, characterized in that: The data processing server includes: a speech-to-text server configured with a speech-to-text engine, a text-to-speech server configured with a text-to-speech engine, a prompt server configured with the prompt information generator, a knowledge storage server configured with the knowledge database, a vector server configured with the vector database, and a log server configured with the historical conversation database; The smart glasses system is further configured to send the first speech to the speech-to-text server, so that the speech-to-text server converts the first speech into the first text through the speech-to-text engine and sends the text to the smart glasses system; The smart glasses system is further configured to send the second text to the text-to-speech server, so that the text-to-speech server converts the second text into the second speech through the speech-to-text engine and sends the second speech to the smart glasses system; The smart glasses system is further used to send notification information to the vector server after the user finishes the query; The vector server is further configured to clear all vector data associated with the user in the vector database according to the notification information.

12. A smart glasses based on a large language model, characterized in that: The smart glasses include: a frame, temples, at least one microphone, at least one speaker, a processor and a memory, wherein the temples are connected to the frame, and the processor is connected to the at least one microphone, the at least one speaker and the memory; The memory stores a computer program executable by the processor, wherein the computer program includes a plurality of instructions. When the plurality of instructions are executed by the processor, the processor: Acquiring, through the at least one microphone, a first voice including a user question to be queried; Converting the first speech into a first text by a speech-to-text engine; Determine the characteristics of the user's question according to the prompt information and the first text through a large language model, and obtain a second text containing an answer to the question according to the characteristics; The second text is converted into a second voice through a text-to-speech engine, and the second voice is played through the at least one speaker.

13. The smart glasses according to claim 12, wherein: When the plurality of instructions are executed by the processor, the processor is further caused to: Extracting multiple keywords indicating user intention from the first text through the large language model, and obtaining at least one sub-question corresponding to the multiple keywords; Acquire the prompt information through a prompt information generator, determine the characteristic according to the prompt information, the multiple keywords and the at least one sub-question, and judge whether it is necessary to search through a tool according to the characteristic; If not, searching a preset knowledge database for first answer information matching the at least one sub-question, and generating the second text based on the first answer information; as well as If necessary, determine at least one corresponding search tool and search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform a search operation by using the search parameters, obtain second reply information matching the at least one sub-question based on the searched information, and generate the second text based on the second reply information.

14. The smart glasses according to claim 13, wherein: When the plurality of instructions are executed by the processor, the processor is further enabled to, after playing the second voice through the at least one speaker, Acquire a third voice including a user instruction through the at least one microphone, and convert the third voice into a third text; Using the large language model, based on the question answer, execute the task indicated by the user instruction, and generate a fourth text containing information on the execution result of the task; The fourth text is converted into a fourth voice, and the fourth voice is played through the at least one speaker.

15. The smart glasses according to claim 14, wherein: The large language model is configured on the smart glasses or the smart mobile terminal or the model server, the speech-to-text engine is configured on the smart glasses or the smart mobile terminal or the speech-to-text server, the text-to-speech engine is configured on the smart glasses or the smart mobile terminal or the text-to-speech server, and the prompt information generator is configured on the smart glasses or the smart mobile terminal or the prompt server; The smart glasses further include a Bluetooth communication component, and when the plurality of instructions are executed by the processor, the processor is also enabled to communicate with the smart glasses through the Bluetooth communication component. The first speech is sent to the smart mobile terminal, and the smart mobile terminal is instructed to: convert the first speech into the first text through the speech-to-text engine, obtain the second text according to the prompt information and the first text through the large language model configured in the smart mobile terminal or the model server, and convert the second text into the second speech through the text-to-speech engine, wherein the prompt information is obtained by the smart mobile terminal through a prompt information generator configured in the smart mobile terminal or the prompt server; as well as Receiving the second voice sent by the smart mobile terminal; as well as When the plurality of instructions are executed by the processor, the processor is also caused to communicate via the Bluetooth component. The third speech is sent to the smart mobile terminal, and the smart mobile terminal is instructed to: convert the third speech into the third text through the speech-to-text engine, execute the task indicated by the user instruction based on the question answer through the large language model configured in the smart mobile terminal or the model server and generate a fourth text containing the execution result information of the task, and convert the fourth text into the fourth speech through the text-to-speech engine; and Receiving the fourth voice sent by the smart mobile terminal; or, The smart glasses further include a wireless communication component, and when the plurality of instructions are executed by the processor, the processor is further enabled to communicate with the smart glasses through the wireless communication component. Sending the first speech to the speech-to-text server so that the first speech is converted into the first text through a speech-to-text engine in the speech-to-text server; Acquire, from the prompt server, the prompt information generated by the prompt information generator on the prompt server; Sending the first text and the prompt information to the model server, so as to obtain the second text according to the prompt information and the first text through a large language model in the model server; as well as Sending the second text to the text-to-speech server so that the second text is converted into the second speech by a text-to-speech engine in the text-to-speech server; and When the plurality of instructions are executed by the processor, the processor is further caused to communicate with the wireless communication component. Sending the third speech to the speech-to-text server, so that the third speech is converted into the third text by the speech-to-text engine in the speech-to-text server; Sending the third text to the model server, so as to execute the task indicated by the user instruction based on the question answer through the large language model in the model server and generate a fourth text containing the execution result information of the task; as well as The fourth text is sent to the text-to-speech server, so that the fourth text is converted into the fourth speech through a text-to-speech engine in the text-to-speech server.

16. The smart glasses according to claim 15, wherein: When the multiple instructions are executed by the processor, the processor is also enabled to convert the searched information into text information through the large language model, cut the text information into text blocks and perform embedding calculations to generate text block vectors, store the generated vectors in a vector database, and retrieve the second reply information from the vector database.

17. The smart glasses according to claim 16, wherein: When the multiple instructions are executed by the processor, the processor is also caused to clear all vector data associated with the user in the vector database after the user ends the query.

18. The smart glasses according to claim 16, wherein: When the plurality of instructions are executed by the processor, the processor is further enabled to generate a session record of the user according to the first text and the second text and store the record in a historical session database; When the plurality of instructions are executed by the processor, the processor is also caused to, through the large language model: When it is determined that a search is required through a tool, searching the historical conversation records associated with the user from the historical conversation database, and obtaining context information of the user's question according to the searched historical conversation records; When the context information is related to at least one of the multiple keywords, obtaining the second reply information by searching the vector database; as well as When the context information is not relevant to the multiple keywords, the corresponding at least one search tool and the search parameters are determined based on the prompt information, the multiple keywords and the at least one sub-question, the search operation is performed by calling the at least one search tool using the search parameters, and the second reply information is obtained based on the searched information.

19. The smart glasses according to claim 18, wherein: The prompt information includes multiple application examples. The large language model determines whether it is necessary to search through tools and determines the corresponding at least one search tool and the search parameters based on the multiple keywords and the at least one sub-question by learning the multiple application examples.

20. The smart glasses according to claim 19, wherein: When the plurality of instructions are executed by the processor, the processor is further caused to: In response to a configuration instruction of a user, obtaining configuration information indicated by the configuration instruction; The prompt information is generated according to the configuration information by the prompt information generator, and the prompt information also includes the configuration information, and the configuration information includes: configuration parameters of at least one of the large language model, the tool, the information source, the data processing method and the retrieval conditions of the vector database.

21. An information search method based on a large language model, characterized in that: Applied to a smart mobile terminal, the information search method includes: Receiving a first voice including a user question to be queried sent by the smart wearable device, and converting the first voice into a first text using a voice-to-text engine, wherein the voice-to-text engine is configured in the smart mobile terminal or the voice-to-text server; Determining the characteristics of the user's question according to the prompt information and the first text through a large language model, and obtaining a second text containing an answer to the question according to the characteristics, wherein the large language model is configured on the smart mobile terminal or the model server; The second text is converted into a second voice through a text-to-speech engine, and the second voice is sent to the smart wearable device for playback, wherein the text-to-speech engine is configured in the smart mobile terminal or the text-to-speech server.

22. The information search method according to claim 21, characterized in that: The method of determining the characteristics of the user's question according to the prompt information and the first text by using the large language model, and obtaining the second text containing the answer to the question according to the characteristics, specifically includes: Extracting multiple keywords indicating user intention from the first text through the large language model, and obtaining at least one sub-question corresponding to the multiple keywords; Determine the characteristic according to the prompt information, the multiple keywords and the at least one sub-question, and determine whether it is necessary to search through a tool according to the characteristic; If not, searching a preset knowledge database for first answer information matching the at least one sub-question, and generating the second text based on the first answer information, wherein the knowledge database is configured in the smart mobile terminal or the knowledge storage server; and If necessary, determine at least one corresponding search tool and search parameters based on the prompt information, the multiple keywords and the at least one sub-question, call the at least one search tool to perform a search operation by using the search parameters, obtain second reply information matching the at least one sub-question based on the searched information, and generate the second text based on the second reply information.

23. The information search method according to claim 22, characterized in that: After sending the second voice to the smart wearable device for playing, the method further includes: Receiving a third voice including a user instruction sent by the smart wearable device, and converting the third voice into a third text using the voice-to-text engine; Using the large language model, based on the question answer, execute the task indicated by the user instruction, and generate a fourth text containing information on the execution result of the task; The fourth text is converted into a fourth voice by the text-to-speech engine, and the fourth voice is sent to the smart wearable device for playback.

24. The information search method according to claim 23, characterized in that: The obtaining, according to the searched information, second answer information matching the at least one sub-question includes: Converting the searched information into text information, cutting the text information into text blocks and performing embedding calculation to generate text block vectors, and storing the generated vectors in a vector database, wherein the vector database is configured in the smart mobile terminal or the vector server; The second reply information is retrieved from the vector database.

25. The information search method according to claim 24, characterized in that: The information search method further includes: After the user finishes the query, all vector data associated with the user are cleared from the vector database.

26. The information search method according to claim 24, characterized in that: The information search method further includes: Generate a session record of the user according to the first text and the second text and store the record in a historical session database, wherein the historical session database is configured in the smart mobile terminal or the log server; Before determining at least one corresponding search tool and search parameter according to the prompt information, the multiple keywords and the at least one sub-question, the information search method further includes: Searching the historical conversation records associated with the user from the historical conversation database, and obtaining context information of the user's question according to the searched historical conversation records; When the context information is related to at least one of the multiple keywords, obtaining the second reply information by searching the vector database; The step of determining at least one corresponding search tool and search parameter according to the prompt information, the multiple keywords, and the at least one sub-question includes: When the context information is not relevant to the multiple keywords, at least one corresponding search tool and search parameter are determined according to the prompt information, the multiple keywords and the at least one sub-question.

27. The information search method according to claim 26, characterized in that: The prompt information includes multiple application examples. The large language model determines whether it is necessary to search through tools and determines the corresponding at least one search tool and the search parameters based on the multiple keywords and the at least one sub-question by learning the multiple application examples.

28. The information search method according to claim 27, characterized in that: The information search method further includes: In response to a configuration instruction of a user, acquiring configuration information indicated by the configuration instruction; and Sending the configuration information to a prompt information generator, so that the prompt information generator generates the prompt information including the configuration information, wherein the configuration information includes: configuration parameters of at least one of the large language model, the tool, the information source, the data processing method, and the search condition of the vector database, wherein the prompt information generator is configured in the smart mobile terminal or the prompt server; Before obtaining the second text including the answer to the question according to the prompt information and the first text by using the large language model, the information search method further includes: The prompt information is obtained through the prompt information generator.

Images