Prompt word generation method and question and answer method of large language model based on intelligent glasses

Abstract

The invention provides a cue word generation method and a question answering method of a large language model based on intelligent glasses, and relates to the technical field of natural language process.The cue word generation method of the large language model based on the intelligent glasses comprises the steps that in response to a received user voice instruction, a current image shot by the intelligent glasses is obtained; performing image information detection on the current image to obtain an entity object in the current image; and on the basis of the entity object in the current image, constructing a cue word. According to the method and the device, the entity object included in the current image can be combined, the customized cue word related to the current scene of the intelligent glasses can be constructed, and more background information provided by the current image is combined, so that the information in the current environment of the intelligent glasses can be described, and the specific cue word with a clear command is obtained; therefore, on the basis of the optimized cue word, the situation that the answer result of a large language model is not uniform can be avoided.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a prompt word generation method and a question-answering method based on a large language model of smart glasses. Background Art

[0002] In the field of machine learning, especially natural language processing, prompts have a very important impact on the output of large models. Prompts can be understood as a piece of text or instructions given to the model to guide the model to generate corresponding outputs. Prompts are a key factor in controlling and guiding the content generated by large models. They not only affect the theme and content of the generated text, but also the style, tone, and emotion. By carefully designing prompts, users can effectively use the capabilities of large models to meet various needs. If the quality of prompts is not high enough, it may not be possible to obtain higher-quality large model generation results. In this regard, no better solution has been proposed yet. Summary of the invention

[0003] The embodiments of the present application provide a prompt word generation method and a question-answering method based on a large language model of smart glasses to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a method for generating prompt words based on a large language model of smart glasses, comprising: in response to receiving a user voice command, obtaining a current image taken by the smart glasses; performing image information detection on the current image to obtain entity objects in the current image; and constructing prompt words based on the entity objects in the current image.

[0005] In a second aspect, an embodiment of the present application provides a question-and-answer method, comprising: obtaining a current user voice command received by the smart glasses; determining the intention of the current user voice command, and determining a natural language processing model for processing the current voice command based on the intention; wherein the natural language processing model is a first model deployed on the smart glasses or a second model deployed on the cloud; the smart glasses are communicatively connected to the cloud; in response to the determined model being the first model, inputting the current user voice command into the first model to obtain a first question-and-answer result; in response to the determined model being the second model, generating a prompt word, inputting the prompt word into the second model to obtain a second question-and-answer result; wherein the second model is a large language model, and the prompt word is generated according to the above method.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0008] Compared with the related art, this application has the following advantages:

[0009] The present application provides a prompt word generation method and a question-answering method based on a large language model of smart glasses. The prompt word generation method based on a large language model of smart glasses includes: in response to receiving a user voice command, obtaining a current image taken by the smart glasses; performing image information detection on the current image to obtain a physical object in the current image; and constructing a prompt word based on the physical object in the current image. According to an embodiment of the present application, a customized prompt word related to the current scene of the smart glasses can be constructed in combination with the physical object included in the current image. In combination with more background information provided by the current image, the information in the current environment of the smart glasses can be described so as to obtain a prompt word with clear commands and specific and clear instructions. Based on the large language model, the more detailed the input data is within the allowed length range, the better. Therefore, based on the optimized prompt word, the situation where the answer results of the large language model are all the same can be avoided.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0012] Figure 1 A flow chart of a method for generating prompt words based on a large language model of smart glasses provided in an embodiment of the present application is shown;

[0013] Figure 2 A processing flow chart provided in an embodiment of the present application is shown;

[0014] Figure 3 A schematic diagram of a classification model network structure provided in an embodiment of the present application is shown;

[0015] Figure 4 A flowchart of a question-and-answer method provided in an embodiment of the present application is shown;

[0016] Figure 5 A structural block diagram of a prompt word generation device based on a large language model of smart glasses provided in an embodiment of the present application is shown;

[0017] Figure 6 A structural block diagram of a question-and-answer device provided in an embodiment of the present application is shown;

[0018] Figure 7 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0020] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0021] Big models are increasingly used, and prompts are very important to them. Prompts (inputs of big models) are commands that drive big language models. Prompts are the text that is input to big language models. If the content of the prompts is different, the big model may give completely different responses.

[0022] Based on this, the present application provides a prompt word generation method and a question-answering method based on a large language model of smart glasses. The prompt word generation method based on a large language model of smart glasses can obtain the current city name, current context information, and current photo recognition results based on the characteristics of the glasses, and automatically generate "background information" of the prompt words. The user's voice command is used as the "current question" and input into the large model, which can improve the accuracy of the large model's output results.

[0023] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0024] The present application embodiment provides a method for generating prompt words based on a large language model of smart glasses, which can be applied to smart glasses. Figure 1The flowchart of the method for generating prompt words based on the large language model of smart glasses according to an embodiment of the present application is shown. The method may include:

[0025] Step S101, in response to receiving a user voice command, obtaining a current image captured by the smart glasses.

[0026] In an embodiment of the present application, smart glasses may be AR (augmented reality) glasses, VR (virtual reality) glasses, MR (mixed reality) glasses, etc. The so-called smart glasses may have computing and application execution capabilities, such as a processor, a memory, and an operating system to support the operation of the application to execute the prompt word generation method of the large language model based on smart glasses of the present application; the so-called smart glasses may also only have display capabilities, and by connecting other terminals with computing and application execution capabilities, such as mobile phones, tablet computers, etc., to support the operation of the application to execute the prompt word generation method of the large language model based on smart glasses of the present application. Smart glasses are equipped with voice interaction and photo taking functions, and can obtain the current location of the smart glasses, the context of the voice interaction between the smart glasses and the user, and the photos taken by the smart glasses.

[0027] The user of the smart glasses can interact with the smart glasses through user voice commands. The user voice command can be a natural language used by the user, such as English, Chinese, French, Spanish, etc. The user voice command can be used to represent the operation that the user wants the smart glasses to perform. For example, the user voice command can be "Please play song B by singer A".

[0028] In response to receiving a user voice command, a current image captured by the smart glasses is obtained, wherein the smart glasses can automatically capture the current image, for example, capture the current image at a specified interval, or can perform the action of capturing the current image according to the user's real-time manipulation. The current image is an image of a person and / or object within the current field of view of the smart glasses.

[0029] It should be noted that large language models (LLM) refer to natural language processing (NLP) models with huge parameter amounts and huge amounts of training data. These models are usually built using deep learning techniques and trained on large-scale data sets in order to grasp the complexity and nuances of language so as to be able to perform various natural language processing tasks. The large language model can be a GPT (Generative Pre-trained Transformer) series, such as GPT-3, GPT-4 and other models, a BERT (Bidirectional Encoder Representations from Transformers) series model, or an ERNIE (a language model optimized for Chinese and other Asian languages) model, or a T5 (a unified text-to-text pre-training model that can perform a variety of NLP tasks) model, etc. In an embodiment of the present application, the type of the large language model can be selected according to actual needs, and the embodiment of the present application is not specifically limited to this.

[0030] Step S102: performing image information detection on the current image to obtain entity objects in the current image.

[0031] In an embodiment of the present application, image information detection involves extracting useful information from an image. Image information detection may include one or more of the following operations: target detection, i.e. identifying a target object in an image and determining its position (usually represented by a bounding box). Image classification, i.e. assigning the entire image to one or more categories. Instance segmentation, i.e. detecting and classifying target objects in an image, and generating pixel-level masks for each target object. Semantic segmentation, i.e. assigning each pixel in an image to a category. Face detection and recognition, i.e. identifying faces in an image and determining their positions, while face recognition further identifies specific people. Edge detection, i.e. identifying the boundaries of objects in an image.

[0032] Entity objects are objects or elements in the current image with clear boundaries and independent features. Entity objects can be natural objects (such as people, animals, trees), man-made objects (such as vehicles, buildings) or other specific elements (such as text, logos).

[0033] In this step, the specific method and algorithm of image information detection can be set according to actual needs, and the embodiment of the present application does not specifically limit this. Through image information detection, the entity object in the current image can be obtained, providing richer information that conforms to the current smart glasses usage environment for constructing prompt words.

[0034] Step S103: construct prompt words based on the entity objects in the current image.

[0035] In an embodiment of the present application, prompt words are constructed based on the entity objects in the current image, so that the constructed prompt words contain information in the current usage environment of the smart glasses. They are customized information generated for the current usage environment. Based on such prompt words, the large language model can be helped to understand and respond to user needs more accurately.

[0036] The present application provides a method for generating prompt words based on a large language model of smart glasses, the method comprising: in response to receiving a user voice command, obtaining a current image captured by the smart glasses; performing image information detection on the current image to obtain a physical object in the current image; and constructing prompt words based on the physical object in the current image. According to an embodiment of the present application, customized prompt words related to the scene currently located by the smart glasses can be constructed in combination with the physical objects included in the current image. In combination with more background information provided by the current image, it is possible to describe the information in the environment currently located by the smart glasses, so as to obtain prompt words with clear commands and specific and clear instructions. Based on the large language model, the more detailed the input data is within the allowed length range, the better. Therefore, based on the optimized prompt words, the situation where the answer results of the large language model are all the same can be avoided.

[0037] In a possible implementation, constructing a prompt word based on the entity objects in the current image can be performed according to the following steps: calculating a ratio value of an object area of ​​at least one entity object in the current image to an overall area of ​​the current image; if the ratio value is greater than a preset ratio threshold value, taking the at least one entity object as a candidate entity object; and constructing the prompt word based on the candidate entity object.

[0038] In this possible implementation, the current image may include one or more entity objects. The ratio of the object area of ​​some or all entity objects in the current image to the overall area of ​​the current image can be calculated. The object area can be a bounding box obtained after image information detection processing. The preset ratio threshold value can be set according to experience or actual needs, and the embodiment of the present application does not specifically limit this. Entity objects whose ratio values ​​are greater than the preset ratio threshold value are taken as candidate entity objects, and one or more candidate entity objects can be selected from the entity objects in the current image, that is, if the ratio value is greater than the preset ratio threshold value, the at least one entity object is taken as a candidate entity object. Finally, based on the screened candidate entity objects as background information, prompt words are constructed.

[0039] In this step, the images captured by the camera on the smart glasses can be used to detect and classify the images, and the largest and most important objects in the current field of view can be extracted. In a scene with multiple objects, usually only close-range objects can be clearly distinguished from the primary and secondary in terms of size. The ratio threshold of the object in the image can be used to determine whether the object is close or far away. If the size of the target identified in the image is relatively small relative to the entire image, it means that the target is far away from the glasses, and such a target will not be further processed. Only close-range objects are further processed.

[0040] Generally speaking, there may be multiple candidate entities in the current image, and all of these candidate entity objects may be used as background information, or only some of them may be used as background information. For example, they may be sorted from large to small according to the ratio value, and the first preset number of candidate entity objects may be used as background information. For another example, multiple candidate entities may be classified, and the candidate entity objects belonging to the most common classification may be used as background information. This application will not list them one by one.

[0041] In a possible implementation, constructing the prompt word based on the candidate entity objects can be performed according to the following steps: determining the entity object with the largest proportion value among the candidate entity objects; taking the entity object with the largest proportion value as the entity object focused on by the current image; and constructing the prompt word based on the entity object focused on by the current image.

[0042] In the embodiment of the present application, a ratio value can be calculated for each candidate entity object. The entity object with the largest ratio value is the entity object closest to the smart glasses and is more likely to become the object of concern, that is, the entity object with the largest ratio value is determined among the candidate entity objects; the entity object with the largest ratio value is used as the entity object of concern in the current image. The prompt word is constructed based on the entity object of concern in the current image, so as to obtain a prompt word that is more relevant to the current environment.

[0043] In a possible implementation, constructing prompt words based on entity objects in the current image can be performed according to the following steps: using a pre-trained scene classifier to identify the scene to which the current image belongs; wherein the scene is used to characterize the environmental features described by the current image; taking entity objects in the current image that are identical to one or more entity objects of interest corresponding to the scene as entity objects of interest to which the current image is concerned; and constructing prompt words based on the entity objects of interest to which the current image is concerned.

[0044] In this possible implementation, a scene classifier is a computer vision model used to identify and classify scene types in images. The goal of a scene classifier is to extract features from a given image and classify the image into a specific scene category, such as a forest, a beach, a city street, an indoor room, etc.

[0045] Before executing this possible implementation, multiple scenes and entity objects of interest in each scene can be pre-set. For example, for the scene "on the road", the entity objects of interest may include: cars, buildings, pedestrians, etc. Corresponding to the scene "conference room", the entity objects of interest may include: computers, projectors, desks and other office equipment. Since there are relatively few typical scenes, smart glasses, using pre-trained scene classifiers, can quickly and accurately know the current scene based on only a few pictures. For example, scenes such as conference rooms, offices, on the road, elevators, and subways can be distinguished. In this step, the image targets of interest can be constrained according to the current scene. For different scenes, the corresponding targets of interest are listed, and targets of no interest are excluded.

[0046] After identifying the scene to which the current image belongs, one or more entity objects of interest specifically corresponding to the scene to which the current image belongs are determined from a plurality of pre-set scenes. Then, entity objects in the current image that are identical to the one or more entity objects of interest corresponding to the scene are taken as entity objects of interest to which the current image is concerned; and prompt words are constructed based on the entity objects of interest to which the current image is concerned, so that prompt words that are more suitable for the current environment of the smart glasses can be obtained.

[0047] In a possible implementation, the smart glasses are provided with a camera device, which is used to collect human eye information of the user of the smart glasses; constructing prompt words based on the entity objects in the current image can be performed according to the following steps: determining the position and size of the pupil according to the human eye information, and determining the user's line of sight focus direction based on the position and size of the pupil; determining the user's area of ​​interest in the current image according to the user's line of sight focus direction, and taking the entity objects in the user's area of ​​interest as the entity objects of interest of the current image; constructing prompt words based on the entity objects of interest of the current image.

[0048] In this possible implementation, a camera device may be provided on the smart glasses, and the camera device is used to collect eye information of the user of the smart glasses, and determine the position and size of the pupil according to the eye information. By detecting the position and size of the pupil, the direction of the current eye attention is determined, so as to obtain the part of the user's current interest, and only identify the target contained in the part. For example, it is determined that the user's eyes are looking at the upper left corner, there is a cloud in the upper left corner of the current image, a lake in the middle, and a tree on the right. According to the direction of the user's eyes, the object cloud is the focused object, that is, the target of interest, that is, the entity object of the current image.

[0049] It should be noted that to determine the direction of a person's gaze, it is necessary to place cameras on the glasses facing the direction of the glasses, one for each eyeball. The center of the eyeball (pupil) is dark and the surrounding is white. Generally speaking, when looking straight ahead, the dark pupil is in the center of the eye, and the gaze direction is determined based on the position of the pupil. Due to differences in human eyes, each person's eyeball and gaze direction can be calibrated during the first use, so that subsequent judgments of gaze direction will be more accurate.

[0050] Considering that there may be multiple entity objects in the current image, and the entity object that the user is interested in is not necessarily the entity object closest to the smart glasses in the current image. Therefore, in a possible implementation, before constructing the prompt word based on the entity object in the current image, the following steps may be performed: obtaining the task subject involved in the user voice instruction; constructing the prompt word based on the entity object in the current image, including: screening the entity object associated with the task subject from the entity objects in the current image; and constructing the prompt word based on the entity object associated with the task subject.

[0051] In this possible implementation, after receiving the user voice command, the user voice command can be analyzed to obtain the task theme involved in the user voice command, so that before designing the prompt word, the specific task that the model wants to complete can be clarified. Task theme is used to characterize tasks that require a large language model to complete, such as writing poems, drawing, and writing reports. It should be noted that in order to distinguish the task themes of user voice commands, a task theme classifier can be used, which can be rule-based or neural network-based.

[0052] When the task theme involved in the user's voice command is obtained, the entity objects associated with the task theme are screened from the entity objects of the current image. Since the task theme may be any theme, different task themes have different relevance to different entity objects in the current image. For example, let the large model generate a paragraph of content describing the scenery, then the sky, clouds, lakes, mountains, trees and other scenery-related entity objects in the current image can be used as background information for the prompt words, and the large model generates more appropriate content. For another example, let the large model answer why animals wag their tails. There are many targets in the current collected pictures, including a dog. Although it is not the largest object, it is more relevant to the theme, so the prompt word is constructed based on the entity objects associated with the task theme, that is, the target dog is selected as the background information for the large model prompt word.

[0053] Filtering entity objects associated with the task subject from the entity objects of the current image can be achieved by querying a mapping table of pre-set task subjects and entity objects of interest. Table 1 shows the mapping table of task subjects and entity objects of interest:

[0054] Task Topic Entity of interest Scenery Description sky, clouds, lake, mountains, trees Animal related Dogs, cats, elephants, lions, giraffes... Architecture Siheyuan, Shikumen, office buildings, villas, etc. …… ……

[0055] Table 1

[0056] Based on Table 1, if the task theme involved in the user's voice command is related to architecture, and if the entity objects of the current image contain one or more of the entity objects of "Siheyuan, Shikumen, office building, villa" recorded in Table 1, then these entity objects that are the same as those in Table 1 are regarded as entity objects associated with the task theme.

[0057] In this possible implementation, a list of targets of interest is queried according to different task themes, and the corresponding target is searched in the current image. Even if the target is not the object with the largest proportion, it is used as background information for the prompt word.

[0058] In actual use, due to the lack of background information and context, the results generated by the large model often make the user dissatisfied. In addition, considering that the reasonable use of guiding language, such as questions or instructions, can help the model better understand the expected task type, therefore, in a possible implementation, constructing prompt words based on the entity objects in the current image can be performed according to the following steps: obtaining the positioning information of the smart glasses, converting the positioning information into a city name, and using the city name as the current location information; obtaining the topic information involved in the user voice command; constructing prompt words based on the user voice command, the entity objects in the current image, the current location information and the topic information.

[0059] In this possible implementation, the location information of the smart glasses is obtained. In order to obtain more accurate prompt words, the location information can be converted into a city name, and the city name is used as the current location information. Topic information. Refers to the content discussed in the previous round or previous rounds of voice interaction between the user and the smart glasses. It is the result of natural language processing in the glasses system and is information that can be directly obtained based on the context content. For example, previously discussing news, stocks, checking weather, navigation, etc.

[0060] Since the large language model has the characteristic that the more detailed the input data (prompt) is within the allowed length range, the better, in this step, prompt words are constructed based on the user voice command, the entity object in the current image, the current location information and the topic information, so as to obtain prompt words with detailed and rich information. By optimizing the prompt words, the large model can better understand the user's questions and generate content suitable for the user better and faster. Through accurate prompt word design, the probability of the model outputting errors or irrelevant information can be reduced. Appropriate prompt word design can help the model cope with a wider range of scenarios and tasks, and improve the versatility and flexibility of the model.

[0061] The present application provides a prompt word generation method and a question-answering method based on a large language model of smart glasses. The prompt word generation method based on a large language model of smart glasses combines the characteristics of the large language model, integrates the existing multimodal features on the glasses, generates background information, and improves the accuracy of the large model.

[0062] In the present application, when constructing prompt words based on the entity objects in the current image, some better entity objects can be screened from the current image to construct prompt words: for example, the entity objects in the current image that are closer to the smart glasses are taken as better entity objects (that is, constructing prompt words based on the entity objects in the current image includes: calculating the ratio of the object area of ​​at least one entity object in the current image to the overall area of ​​the current image; if the ratio is greater than a preset ratio threshold value, taking the at least one entity object as a candidate entity object; and constructing the prompt words based on the candidate entity objects).

[0063] After determining multiple entity objects that are relatively close to the smart glasses, one or more of the following schemes may be further adopted to further select a better entity object from the multiple entity objects that are relatively close to the smart glasses: taking the object that is closest to the smart glasses as the better entity object (i.e. constructing the prompt word based on the candidate entity objects, including: determining the entity object with the largest ratio value among the candidate entity objects; taking the entity object with the largest ratio value as the entity object of interest in the current image; and constructing the prompt word based on the entity object of interest in the current image); using a pre-trained scene classifier to further select a better entity object (i.e. constructing a prompt word based on the entity object in the current image, including: using a pre-trained scene classifier to identify the scene to which the current image belongs; wherein the scene is used to characterize the environmental features described by the current image; taking the entity object in the current image that is identical to one or more entity objects of interest corresponding to the scene as the entity object of interest in the current image; and constructing the prompt word based on the entity object of interest in the current image) ; further selecting a better entity object based on the camera device configured in the smart glasses (that is, the smart glasses are provided with a camera device, and the camera device is used to collect human eye information of the user of the smart glasses; constructing prompt words based on the entity objects in the current image, including: determining the position and size of the pupil according to the human eye information, and determining the user's line of sight focus direction based on the position and size of the pupil; determining the user's area of ​​interest in the current image according to the user's line of sight focus direction, and taking the entity objects in the user's area of ​​interest as the entity objects focused on by the current image; constructing prompt words based on the entity objects focused on by the current image); further selecting a better entity object in combination with the task subject involved in the voice instruction (that is, before constructing prompt words based on the entity objects in the current image, it also includes: obtaining the task subject involved in the user's voice instruction; constructing prompt words based on the entity objects in the current image, including: screening entity objects associated with the task subject from the entity objects in the current image; constructing the prompt words based on the entity objects associated with the task subject).

[0064] The present application embodiment also provides a question-answering method, such as Figure 4 The flowchart of the question-answering method according to an embodiment of the present application is shown. The method may include:

[0065] Step S401, obtaining the current user's voice command received by the smart glasses.

[0066] In the embodiment of the present application, the smart glasses may be AR (augmented reality) glasses, VR (virtual reality) glasses, MR (mixed reality) glasses, etc. The so-called smart glasses may have computing and application execution capabilities, such as having a processor, a memory, and an operating system to support the operation of applications to execute the method for generating prompt words based on the large language model of smart glasses in the present application; the so-called smart glasses may also only have display capabilities, and by connecting to other terminals with computing and application execution capabilities, such as mobile phones, tablet computers, etc., to support the operation of applications to execute the method for generating prompt words based on the large language model of smart glasses in the present application.

[0067] The user of the smart glasses can interact with the smart glasses through user voice commands. The user voice command can be a natural language used by the user, such as English, Chinese, French, Spanish, etc. The user voice command can be used to represent the operation that the user wants the smart glasses to perform. For example, the user voice command can be "Please play song B by singer A".

[0068] Step S402, determining the intention of the current user voice command, and determining a natural language processing model for processing the current voice command based on the intention; wherein the natural language processing model is a first model deployed on the smart glasses or a second model deployed on the cloud; the smart glasses are communicatively connected to the cloud.

[0069] In an embodiment of the present application, a pre-trained NLP (Natural Language Processing) classification model can be used to identify the intention of the current user's voice command, determine whether the intention of the current user's voice command belongs to the type corresponding to the first model or the second model, and then determine the natural language processing model used to process the current voice command based on the intention. The pre-trained NLP classification model can classify the words spoken in the user voice interaction. For example, it can be divided into two categories, suitable for a large model (second model) and not suitable for a large model (first model). The judgment is based on the intention of the current user's voice command. Generally, complex sentences / commands are suitable for a large model. Before executing this application, such as in the development stage, such a classification model can be pre-trained using a large number of samples.

[0070] In an embodiment of the present application, taking into account the computational overhead and delay of the second model, a classification model is used in advance to filter out unnecessary calculations, that is, the first model is used to perform calculations that do not necessarily require the second model to be implemented, thereby optimizing the utilization of computing resources.

[0071] In one possible implementation, the structure of the NLP classification model is as follows: Figure 3As shown. The input layer (one-hot conversion layer) is used for: the string of user input sentences is converted into index numbers of words through the input layer, and a sentence becomes an array of index numbers. It can be agreed that the maximum length is 70 words, and the excess part is truncated and discarded, because people usually speak less than 70 words at a time. Therefore, the output of the input layer is an array of up to 70 elements. The value of the array element is an integer value, representing the index of the word. Word embedding layer (word vector): Word embedding is to use a multi-dimensional floating point data to represent the meaning of each word, such as 128 dimensions, so each word is represented by a one-dimensional array of 128 elements. Word embedding is obtained during training and used directly during inference. The word index array output by the input layer is converted into a multi-dimensional word vector through the word embedding layer, so the index of the first 70 words becomes a [70,128] matrix, each element is a floating point number.

[0072] Convolution layer: The function of convolution layer is to ignore most features and amplify and extract certain features. For example, in NLP intent classification processing, 3-, 4-, and 5-word feature extraction is used. Continuous 3, 4, and 5 words are taken as interesting features and extracted for later processing. In this way, 3 to 5 words can be treated as a whole. If they are words or phrases, they are considered as a whole. If they are all single words, their relationship is considered. The input of the convolution layer is the output matrix of the previous word embedding layer, and the output is also a matrix. The size of the matrix is ​​related to the size of the convolution kernel. For example, if the convolution kernel size is [3,128], the output matrix after convolution is [68,1]. 3-byte length feature extraction corresponds to the convolution kernel [3,128], 4-byte length feature extraction corresponds to the convolution kernel [4,128], and 5-byte length feature extraction corresponds to the convolution kernel [5,128]. There are multiple convolution kernels of each size, such as 128. Pooling layer: The purpose of the pooling layer is to ignore unimportant features in the features extracted by the convolution kernel and only look at the most important features. The method is "downsampling". The so-called "downsampling" is to find the largest value in the matrix and replace the entire matrix with the maximum value. Each convolution kernel is followed by a pooling layer, and the output of the pooling layer is a maximum value in the matrix representing the convolution result.

[0073] Fusion layer: The output data of multiple pooling layers are combined to form a large one-dimensional array, such as 128 3-byte convolution pooling, 128 4-byte convolution pooling, and 128 5-byte convolution pooling. The 384 values ​​obtained form a one-dimensional array, and each element in the array is a floating point number. Fully connected layer (Dense): The fully connected layer receives the output from the fusion layer, which is a one-dimensional array of 384 elements. After passing through the fully connected layer, it becomes n floating point values, where n is equal to the preset classification category. For example, if the user has 20 intentions, then n is 20. Here, the size of each value in the n floating point numbers represents the size of the possibility of the category. The more likely the classification, the larger the corresponding value. Normalization is generally required. Normalization means to reduce the value proportionally so that the sum of all n values ​​is 1.

[0074] Output layer: Receives the output (n values) of the previous layer (fully connected layer), which is equivalent to a one-dimensional array with n elements. The subscript of each element in the array represents a classification number. The output layer converts the classification number into user-recognizable intent, such as classification type 1 represents the "singer" category, and classification type 2 represents the "song name" category. The classification number and category name are pre-defined by the user.

[0075] The first model performs better in environments with limited computing efficiency and resources, is suitable for real-time and embedded applications, and can be deployed on the smart glasses side, while the second model has advantages in processing complex tasks and pursuing high precision, but requires more computing resources and time, and can be deployed on the cloud, wherein the smart glasses are communicatively connected to the cloud.

[0076] In this step, through pre-classification filtering, that is, distinguishing the first model and the second model category, only the second model category calls the second model to reduce its calculation amount. It should be noted that if the network is disconnected, even if the natural language processing model used to process the current voice instruction is determined to be the second model according to the intention, the call of the second model is directly rejected, and the first model can be selected as the natural language processing model for the current voice instruction.

[0077] It should be noted that the second model can be a large language model (LLM). Large language models refer to natural language processing (Natural Language Processing, NLP) models with huge parameter amounts and huge amounts of training data. These models are usually built using deep learning techniques and trained on large-scale data sets in order to grasp the complexity and nuances of language, so as to be able to perform various natural language processing tasks. The large language model can be a GPT (Generative Pre-trained Transformer) series, such as GPT-3, GPT-4 and other models, a BERT (Bidirectional Encoder Representations from Transformers) series model, or an ERNIE (a language model optimized for Chinese and other Asian languages) model, or a T5 (a unified text-to-text pre-training model that can perform a variety of NLP tasks) model, etc. In an embodiment of the present application, the type of the large language model can be selected according to actual needs, and the embodiment of the present application is not specifically limited to this.

[0078] In an embodiment of the present application, intent recognition processing is performed based on the current user voice command, that is, the intention of the current user voice command is determined. Intent recognition maps the user's natural language input to a specific intent category. Common methods include but are not limited to: rule and template matching: matching user input based on predefined rules or templates. Machine learning model: Use classifiers such as SVM (Support Vector Machine), decision trees, and neural networks to classify intent. Deep learning model: Use deep neural networks such as RNN (recurrent neural network), LSTM (Long-Stort TermMemory, long short-term memory network), and Transformer (a sequence language model based on an attention mechanism) to perform intent recognition, especially on complex and diverse inputs.

[0079] Step S403, in response to the determined model being the first model, inputting the current user voice instruction into the first model to obtain a first question and answer result.

[0080] In the embodiment of the present invention, the first model can adopt a traditional neural network model, which can be deployed on smart glasses or other terminals that communicate with smart glasses, such as mobile phones. The small model can be used to perform tasks such as intent recognition, named entity recognition, and rejection.

[0081] In the embodiment of the present application, considering that the second model has a large amount of calculation, even if it is deployed in the cloud, the computing power overhead is also large. Therefore, for some tasks with less calculation amount and can be processed by the first model, the first model is used to process them, thereby reducing the calculation amount of the second model.

[0082] Step S404, in response to the determined model being the second model, generates prompt words, inputs the prompt words into the second model, and obtains a second question and answer result; wherein, the second model is a large language model, and the prompt words are generated according to the above-mentioned prompt word generation method of the large language model based on smart glasses.

[0083] In an embodiment of the present invention, the second model is a neural network model with a relatively large parameter scale and a large amount of calculation. In voice interaction, according to different tasks, the second model may include but is not limited to: a large model for voice recognition, a large model for natural language understanding, a large model for text content generation, a large model for image and video content generation, etc. Due to the limitations of computing and storage resources, the second model can currently run in the cloud. In one possible implementation, the second model uses a large model for natural language understanding and a large model for text content generation. For example, the user says: "Please help me write a resume. My basic information is as follows:..." The large model will generate a resume. The design involves natural language understanding and content generation. It can be two large models that handle content understanding and content generation respectively, or it can be one large model that handles understanding and generation.

[0084] The embodiment of the present application provides a question-answering method, which includes: obtaining a current user voice command received by the smart glasses; determining the intention of the current user voice command, and determining a natural language processing model for processing the current voice command according to the intention; wherein the natural language processing model is a first model deployed on the smart glasses or a second model deployed on the cloud; the smart glasses are connected to the cloud in communication; in response to the determined model being the first model, the current user voice command is input into the first model to obtain a first question-answering result; in response to the determined model being the second model, a prompt word is generated, and the prompt word is input into the second model to obtain a second question-answering result; wherein the second model is a large language model, and the prompt word is generated according to the above method. According to the embodiment of the present application, a customized prompt word related to the current scene of the smart glasses can be constructed in combination with the entity objects included in the current image, and in combination with more background information provided by the current image, the information in the current environment of the smart glasses can be described, so as to obtain a prompt word with clear commands and specific and clear instructions. Based on the large language model, the more detailed the input data is within the allowed length range, the better, so that the optimized prompt word can avoid the situation where the answer results of the large language model are all the same. In addition, the first model and the second model can be distinguished, so that the calculations that are not necessarily performed by the second resource can be handed over to the first model for execution, thereby optimizing the configuration of computing resources and reducing the amount of calculation of the second model.

[0085] Since the second model is deployed in the cloud, it cannot be used normally when the network is disconnected. In addition, the second model has a large processing delay, a slow processing speed, and a timeout. If it is greater than 1 second, it will affect the user experience. Therefore, in a possible implementation, if the determined model is the second model, the method can also perform the following steps: input the current user voice command into the first model to obtain an alternative result; when it is detected that the smart glasses have no network connection or the calculation time of the second question and answer result exceeds the preset time threshold, the alternative result is used as the second question and answer result.

[0086] In this step, in order to cope with various situations that cannot be handled by the large model, such as network problems or timeouts, the processing of the small model is always processed in parallel. After determining to use the second model to process the current voice command, the first model is used to process the current voice command in parallel, that is, the current user voice command is input into the first model to obtain an alternative result. When it is detected that the smart glasses have no network connection or the calculation time of the second question and answer result exceeds the preset time threshold, the alternative result is used as the second question and answer result. It should be noted that even if the second model returns the processing result on time, the first model processing is also carried out in parallel, but when the second model is in normal working condition, its result is not used. It is a backup function. In this step, in the event of a second model processing timeout or no network connection, the processing result obtained by the first model for the current user voice command can also be provided or a preset reply can be used.

[0087] See also Figure 2 A processing flow chart shown below is combined with Figure 2 A possible implementation process of this method is described.

[0088] Step S1, classification:

[0089] The NLP classification model is used to identify user intentions. For example, when a user says "I want to listen to song XX by Liu XX", the user's intention is to search for the song XX by the singer "Liu XX". Such intentions are numbered in advance. For example, number 1 represents "search for singer and song title".

[0090] Step S2, classification result score determination:

[0091] The previous step infers the scores of the current user's speech content in various fields, where the large model (second model) is a category and corresponds to a score. This step determines whether the highest score is the category corresponding to the large model. If yes, go to step S4; if not, go to step S3.

[0092] If what the user said is not suitable for processing by the large model, the large model will not be called, and the process directly goes to step S3 to call the small model (the first model).

[0093] Step S3, small model processing:

[0094] The voice assistant application on glasses can be separated from the large model and serve independently, including several small models (intent recognition, named entity recognition, rejection, etc.). Small models refer to traditional neural network models, some of which are deployed on glasses and most of which are deployed on mobile phones. In order to adapt to various large model processing exceptions, small models must be processed in parallel and the processing results of small models must be provided for use at any time.

[0095] Step S4, network status determination:

[0096] The user can walk around freely wearing smart glasses, and there is no guarantee that there will always be wireless network, such as walking to the basement, elevator, etc. This step determines whether the current network is available. If it is not available, proceed to step S3, and if it is available, proceed to step S5.

[0097] Step S5, photo image recognition:

[0098] Since smart glasses can be equipped with image sensors, and this is the future trend, after all, glasses with photo taking and video recording can do many interesting functions. A major feature of smart glasses is that the camera on the glasses always captures the picture facing the front as the face changes, that is, the captured picture is the target that the user is facing, the potential target of interest. For example, when using glasses for interview translation, facing the interviewee. This step extracts the target category from the picture, or there is no target, and outputs the result to step S6.

[0099] Step S6, generating prompt words:

[0100] Good prompt words can help the big model better understand the user's questions and generate more reasonable content. In the scenario of wearing smart glasses, the smart glasses can be used with a mobile phone to obtain the following information:

[0101] 1. Geographic location: current location, city;

[0102] 2. Physical objects in the current image: close-range targets identified in step S5, such as a car in front, a cat in front, a bowl of beef noodles on the table, etc.;

[0103] 3. Topic categories of the first three rounds of voice interaction within a maximum of 5 minutes.

[0104] By submitting this background information together with user questions to the big model, the content generated by the big model will be more suitable for users.

[0105] The resulting prompt is in the following format:

[0106] [Current location]: Shanghai;

[0107] [Object in front of me]: A cat;

[0108] [Current topic]: Animal encyclopedia knowledge;

[0109] [Question]: Please help me write a poem.

[0110] The input of the large model above is definitely better than just "help me write a poem". The large model gets the location, the object in front of it, and the topic discussed before. The generated poem is likely to carry background information, which is more suitable for the current scene. The background information of the prompt word is not fixed. If some information currently exists, it will be added. If it does not exist, it will be empty. For example, in the task of describing the scenery, the image targets of interest are the sky, clouds, lakes, mountains, and trees. Unfortunately, there are no such targets in the current collected pictures, so the prompt word does not have the "object in front of you" information.

[0111] Step S7, large model reasoning:

[0112] A large model refers to a neural network model with a relatively large parameter scale and a large amount of calculation. In voice interaction, it is divided into a voice recognition large model, a natural language understanding large model, a text content generation large model, an image and video content generation large model, etc. according to different tasks. Currently, due to the limitations of computing and storage resources, large models can only run in the cloud. The large models used in this patent include a natural language understanding large model and a text content generation large model. For example, if a user says, "Help me write a resume, my basic information is as follows:...", the large model will generate a resume. The design involves natural language understanding and content generation. It can be that two large models handle content understanding and content generation respectively, or one large model handles understanding and generation. It should be noted that background information not related to the topic may be ignored by the large model. For example, there is a mosquito in front of the smart glasses, and the task of the smart glasses is to write a resume. Because the resume and the mosquito are not closely related during training, the mosquito will be ignored. If the task of the smart glasses is to write an explanatory text for a summer evening, the mosquito will most likely be written into the answer.

[0113] Step S8, timeout judgment:

[0114] The calculation of large models is relatively large, and the delay is also relatively large. In voice interaction, if the end-to-end delay exceeds 1.5 seconds, the user can clearly perceive it, which will affect the user experience. The end-to-end delay includes voice recognition, semantic understanding, semantic behavior execution, network transmission, etc., so the time left for large model reasoning is limited. Once the large model reasoning exceeds the predetermined time T, special processing is required. The timeout threshold value T is an empirical value, usually a few hundred milliseconds. If the large model processing response times out, go to step S9, and if there is no timeout, go to step S10.

[0115] Step S9, return the small model processing result:

[0116] Entering this step means that due to network problems or timeouts, the inference results of the large model are not used, and the results of the small model are directly processed. In order to cope with various situations that the large model cannot handle, the processing of the small model is always processed in parallel. Even if the large model returns the processing results on time, the processing of the small model is also carried out in parallel, but its results are not usually used. Usually, it is used as a backup.

[0117] Step S10, return the large model result:

[0118] Entering this step means that the inference result of the big model arrives within the time limit, and the result of the big model is used. There are many types of big models, including natural language understanding big models and content generation big models. Content generation big models include generating text, voice, pictures and videos. The big model mentioned here can be any type of big model, which generally refers to various models with large parameter scale and large computational complexity.

[0119] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a prompt word generation device based on a large language model of smart glasses. Figure 5 The structure block diagram of a prompt word generation device based on a large language model of smart glasses according to an embodiment of the present application is shown, and the device may include:

[0120] The image module 501 is used to obtain the current image taken by the smart glasses in response to receiving a user voice command; the detection module 502 is used to perform image information detection on the current image to obtain the entity object in the current image; the construction module 503 is used to construct a prompt word based on the entity object in the current image.

[0121] The present application provides a prompt word generation device based on a large language model of smart glasses, and the prompt word generation device based on the large language model of smart glasses is used to: in response to receiving a user voice command, obtain the current image taken by the smart glasses; perform image information detection on the current image to obtain the entity object in the current image; and construct prompt words based on the entity object in the current image. According to the embodiment of the present application, it is possible to construct customized prompt words related to the scene currently located by the smart glasses in combination with the entity objects included in the current image, and in combination with more background information provided by the current image, it is possible to describe the information in the environment currently located by the smart glasses, so as to obtain prompt words with clear commands and specific and clear instructions. Based on the large language model, it has the characteristic that the more detailed the input data is within the allowed length range, the better, so that the optimized prompt words can avoid the situation where the answer results of the large language model are all the same.

[0122] In a possible implementation, constructing a prompt word based on the entity object in the current image includes: calculating a ratio value of an object area of ​​at least one entity object in the current image to an overall area of ​​the current image; if the ratio value is greater than a preset ratio threshold value, taking the at least one entity object as a candidate entity object; and constructing the prompt word based on the candidate entity object.

[0123] In a possible implementation, constructing the prompt word based on the candidate entity objects includes: determining the entity object with the largest proportion value among the candidate entity objects; taking the entity object with the largest proportion value as the entity object focused on by the current image; and constructing the prompt word based on the entity object focused on by the current image.

[0124] In a possible implementation, constructing prompt words based on entity objects in the current image includes: using a pre-trained scene classifier to identify the scene to which the current image belongs; wherein the scene is used to characterize environmental features described by the current image; taking entity objects in the current image that are identical to one or more entity objects of interest corresponding to the scene as entity objects of interest to which the current image is concerned; and constructing prompt words based on the entity objects of interest to which the current image is concerned.

[0125] In a possible implementation, the smart glasses are provided with a camera device, which is used to collect human eye information of the user of the smart glasses; construct prompt words based on the entity objects in the current image, including: determining the position and size of the pupil according to the human eye information, and determining the user's line of sight focus direction based on the position and size of the pupil; determining the user's area of ​​interest in the current image according to the user's line of sight focus direction, and using the entity objects in the user's area of ​​interest as the entity objects focused on by the current image; and constructing prompt words based on the entity objects focused on by the current image.

[0126] In a possible implementation, the device also includes a theme module, which, before constructing a prompt word based on the entity objects in the current image, is used to: obtain a task theme related to the user voice instruction; construct a prompt word based on the entity objects in the current image, including: screening entity objects associated with the task theme from the entity objects in the current image; and constructing the prompt word based on the entity objects associated with the task theme.

[0127] In a possible implementation, constructing a prompt word based on the entity object in the current image includes: obtaining positioning information of the smart glasses, converting the positioning information into a city name, and using the city name as current location information; obtaining topic information involved in the user voice command; and constructing a prompt word based on the user voice command, the entity object in the current image, the current location information, and the topic information.

[0128] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a question-answering method device. Figure 6 The figure is a structural block diagram of a question-answering method device according to an embodiment of the present application, and the device may include:

[0129] An acquisition module 601 is used to acquire the current user voice command received by the smart glasses; a selection module 602 is used to determine the intention of the current user voice command, and determine a natural language processing model for processing the current voice command according to the intention; wherein the natural language processing model is a first model deployed on the smart glasses or a second model deployed on the cloud; the smart glasses are communicatively connected to the cloud; a first response module 603 is used to respond to the determined model being the first model, input the current user voice command into the first model, and obtain a first question and answer result; a second response module 604 is used to respond to the determined model being the second model, generate a prompt word, input the prompt word into the second model, and obtain a second question and answer result; wherein the second model is a large language model, and the prompt word is generated according to the above-mentioned prompt word generation method based on the large language model of smart glasses.

[0130] The embodiment of the present application provides a question-answering device, which is used to: obtain the current user voice command received by the smart glasses; determine the intention of the current user voice command, and determine the natural language processing model for processing the current voice command according to the intention; wherein the natural language processing model is a first model deployed on the smart glasses or a second model deployed on the cloud; the smart glasses are connected to the cloud in communication; in response to the determined model being the first model, the current user voice command is input into the first model to obtain a first question-answering result; in response to the determined model being the second model, a prompt word is generated, and the prompt word is input into the second model to obtain a second question-answering result; wherein the second model is a large language model, and the prompt word is generated according to the above method. According to the embodiment of the present application, a customized prompt word related to the current scene of the smart glasses can be constructed in combination with the entity objects included in the current image, and in combination with more background information provided by the current image, the information in the current environment of the smart glasses can be described, so as to obtain a prompt word with clear commands and specific and clear instructions. Based on the large language model, the more detailed the input data is within the allowed length range, the better the characteristics are, so that the optimized prompt word can avoid the situation where the answer results of the large language model are stereotyped. In addition, the first model and the second model can be distinguished, so that the calculations that are not necessarily performed by the second resource can be handed over to the first model for execution, thereby optimizing the configuration of computing resources and reducing the amount of calculation of the second model.

[0131] In a possible implementation, if the determined model is the second model, the device also includes an alternative module, which is used to: input the current user voice command into the first model to obtain an alternative result; when it is detected that the smart glasses have no network connection or the calculation time of the second question and answer result exceeds a preset time threshold, use the alternative result as the second question and answer result.

[0132] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0133] Figure 7 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 7 As shown, the electronic device includes: a memory 701 and a processor 702. The memory 701 stores a computer program that can be run on the processor 702. When the processor 702 executes the computer program, the method in the above embodiment is implemented. The number of the memory 701 and the processor 702 can be one or more.

[0134] The electronic device also includes:

[0135] The communication interface 703 is used to communicate with external devices and perform data exchange transmission.

[0136] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the memory 701, the processor 702 and the communication interface 703 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0137] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0138] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0139] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0140] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0141] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0142] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0143] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0144] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0145] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0148] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0149] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0150] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0151] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0152] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating prompt words based on a large language model of smart glasses, including: In response to receiving a user voice command, acquiring a current image captured by the smart glasses; Performing image information detection on the current image to obtain entity objects in the current image; Construct prompt words based on the entity objects in the current image.

2. The method according to claim 1, wherein: Constructing a prompt word based on the entity object in the current image, including: Calculating a ratio value of an object area of ​​at least one physical object in the current image to an overall area of ​​the current image; If the ratio value is greater than a preset ratio threshold value, taking the at least one entity object as a candidate entity object; The prompt word is constructed based on the candidate entity object.

3. The method according to claim 1, wherein: Constructing a prompt word based on the entity object in the current image, including: Using a pre-trained scene classifier, identifying the scene to which the current image belongs; wherein the scene is used to characterize environmental features described by the current image; Taking the entity objects in the current image that are identical to the one or more entity objects of interest corresponding to the scene as the entity objects of interest of the current image; A prompt word is constructed based on the entity object concerned by the current image.

4. The method according to claim 1, wherein: The smart glasses are provided with a camera device, and the camera device is used to collect human eye information of the user of the smart glasses; Constructing a prompt word based on the entity object in the current image, including: Determine the position and size of the pupil according to the human eye information, and determine the direction of the user's sight based on the position and size of the pupil; Determine a user's area of ​​interest in the current image according to the user's sight line focus direction, and use the entity object in the user's area of ​​interest as the entity object focused on by the current image; A prompt word is constructed based on the entity object concerned by the current image.

5. The method according to claim 1, wherein: Before constructing the prompt word based on the entity object in the current image, the method further includes: Obtaining the task subject involved in the user voice command; Constructing a prompt word based on the entity object in the current image, including: Filtering entity objects associated with the task subject from entity objects in the current image; The prompt word is constructed based on the entity object associated with the task theme.

6. The method according to claim 1, wherein: Constructing a prompt word based on the entity object in the current image, including: Acquire the positioning information of the smart glasses, convert the positioning information into a city name, and use the city name as the current location information; Acquiring topic information involved in the user voice command; A prompt word is constructed based on the user voice instruction, the entity object in the current image, the current location information and the topic information.

7. Question and answer method, including: Get the current user's voice command received by the smart glasses; Determine the intention of the current user voice command, and determine a natural language processing model for processing the current voice command according to the intention; wherein the natural language processing model is a first model deployed on the smart glasses or a second model deployed on the cloud; the smart glasses are communicatively connected to the cloud; In response to the determined model being the first model, inputting the current user voice instruction into the first model to obtain a first question-and-answer result; In response to the determination that the model is the second model, a prompt word is generated, and the prompt word is input into the second model to obtain a second question and answer result; wherein, the second model is a large language model, and the prompt word is generated according to the method described in any one of claims 1-6.

8. The method according to claim 7, wherein: If the determined model is the second model, the method further includes: Input the current user voice command into the first model to obtain an alternative result; When it is detected that the smart glasses have no network connection or the calculation time of the second question and answer result exceeds a preset time threshold, the alternative result is used as the second question and answer result.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium having a computer program stored therein, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Images