Interaction system and method based on intelligent glasses and intelligent glasses

By introducing information preprocessing and multimodal large model technology into the smart glasses system, the problem of poor user experience in traditional smart glasses interaction systems is solved, and more efficient interactive information processing and feedback is achieved, improving user experience.

CN119045770APending Publication Date: 2024-11-29GUANGDONG UNIV OF TECH
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202411211195.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The traditional smart glasses interactive system has poor user experience, mainly due to the low efficiency of information transmission and processing, which leads to inconvenient and rich interaction behavior.

Method used

An interactive system based on smart glasses is designed, including smart glasses and service terminals, equipped with information preprocessing module, smart assistant module, application software module and multimodal processing module, and identify the received interactive information by preprocessing it. Instruct and call multimodal large model for interactive operation, and feedback response information in real time.

Benefits of technology

It improves the clarity and reliability of interactive information, realizes smarter interactive content generation, improves user experience, and provides more convenient and richer operation methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045770A_ABST
    Figure CN119045770A_ABST
Patent Text Reader

Abstract

The invention discloses an interaction system and method based on intelligent glasses and the intelligent glasses, and relates to the technical field of the intelligent glasses, and the system comprises the intelligent glasses and a service terminal; the service terminal is provided with an information preprocessing module, an intelligent assistant module, an application software module, a multi-mode processing module and a feedback module; the information preprocessing module and the feedback module are both in communication connection with the intelligent glasses, the intelligent assistant module is connected with the information preprocessing module, and the application software module and the multi-mode processing module are both connected with the intelligent assistant module and the feedback module; the definition and reliability of data are improved by preprocessing interaction information, more intelligent interaction content generation is realized by introducing a multi-modal large model technology, and an application software module and a multi-modal large model are selectively called to complete interaction by performing instruction recognition processing through the interaction information. And the response information is fed back to the intelligent glasses for display in real time, so that more convenient and abundant operations are provided for the user, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of smart glasses, and in particular to an interactive system and method based on smart glasses, and smart glasses. Background Art

[0002] Smart glasses are a type of wearable device that integrates computing, display and networking capabilities, and can realize a variety of interactive experiences such as augmented reality (AR), virtual reality (VR) and mixed reality (MR). At present, the traditional interactive system based on smart glasses mainly transmits text, images, audio and other information through communication between smart glasses and the server to complete the interactive behavior, and the user experience is poor. Summary of the invention

[0003] The present invention provides an interactive system and method based on smart glasses and smart glasses, which improve the user experience when the user uses the smart glasses for interaction.

[0004] A first aspect of the present invention provides an interactive system based on smart glasses, comprising: smart glasses and a service terminal;

[0005] The service terminal is equipped with an information preprocessing module, an intelligent assistant module, an application software module, a multimodal processing module and a feedback module;

[0006] The information preprocessing module and the feedback module are both connected to the smart glasses in communication, the smart assistant module is connected to the information preprocessing module, and the application software module and the multimodal processing module are both connected to the smart assistant module and the feedback module;

[0007] The information preprocessing module is used to preprocess the received initial interaction information sent by the smart glasses and output target interaction information;

[0008] The intelligent assistant module is used to perform instruction recognition processing according to the target interaction information, generate a recognition result, and selectively call the application software module and the multimodal large model of the multimodal processing module to perform interactive operations according to the recognition result, and output response information;

[0009] The feedback module is used to feed back the response information to the smart glasses for display.

[0010] Optionally, the smart glasses are provided with a wireless communication module, an image acquisition module, an audio input module, an audio output module, a control storage module, a wireless charging module and a display module connected via a bus;

[0011] The wireless communication module is communicatively connected with the information preprocessing module and the feedback module;

[0012] The image acquisition module is used to acquire the initial image in the initial interaction information;

[0013] The audio input module is used to collect the initial audio in the initial interaction information;

[0014] The audio output module is used to display the response audio of the response information;

[0015] The display module is used to display the response text and / or the response image of the response information;

[0016] The control storage module is used to store the initial image and the initial audio, and control the audio output module and the display module to display the response information;

[0017] The wireless charging module is used for supplying power.

[0018] Optionally, the information preprocessing module is specifically used to:

[0019] Receiving interaction information sent by the smart glasses;

[0020] If the initial interaction information is unimodal information, performing data denoising on the initial interaction information and outputting target interaction information;

[0021] If the initial interaction information is multimodal information, data alignment is performed according to the timestamp of the initial interaction information, and then data denoising is performed to output the target interaction information.

[0022] Optionally, the target interaction information includes target audio and / or target image; and the intelligent assistant module is specifically used for:

[0023] If the target interaction information contains target audio, converting the target audio into instruction text through a preset speech recognition model;

[0024] Performing lexical analysis on the instruction text based on a preset SpaCy library to generate multiple instruction phrases;

[0025] Using a preset NLTK library to perform part-of-speech tagging on each of the instruction phrases, and outputting the phrase part-of-speech of each of the instruction phrases;

[0026] Determine the dependency relationship between each of the instruction phrases based on the part of speech of each phrase through a preset AllenNLP model analysis;

[0027] According to the dependency relationship, a preset SRL model is used to perform semantic role labeling on each of the instruction phrases to determine a labeling result;

[0028] If the annotation result contains both an execution object and an execution action, the execution object and the execution action are combined to construct an operation instruction;

[0029] If the operation instruction successfully matches the preset instruction word library and the target interactive information contains a target image, calling the corresponding application software module according to the operation instruction to perform an interactive operation on the target image and outputting response information;

[0030] If the operation instruction fails to match the preset instruction word library and the target interactive information contains a target image, the operation instruction is encapsulated into a prompt text, and the multimodal large model of the multimodal processing module is called to perform an interactive operation based on the prompt text and the target image, and output response information;

[0031] If the annotation result does not contain both the execution object and the execution action and the target interaction information contains a target image, or the target interaction information contains only a target image, a prompt text is generated according to the default function instruction through a preset auxiliary model, and is sent to the multimodal large model of the multimodal processing module for processing in combination with the target image, and response information is output.

[0032] Optionally, the multimodal processing module is used to:

[0033] When the target image sent by the intelligent assistant module is received, a preset multimodal large model is called to perform image recognition on the target image to determine the image keyword;

[0034] When the prompt text sent by the intelligent assistant module is received, named entity recognition is performed on the prompt text through the multimodal large model to determine the text keywords;

[0035] The image keyword and / or the text keyword are used as target keywords, and a topic text is generated according to the target keyword based on the multimodal macro model, and multiple topic texts are output and sorted to determine a response text;

[0036] Generate a response image or / and a response audio according to the target topic text through the multimodal large model;

[0037] The response text, response image and / or response audio are output as response information.

[0038] A second aspect of the present invention provides an interaction method based on smart glasses, comprising:

[0039] When receiving the initial interaction information sent by the smart glasses, preprocessing the initial interaction information and outputting target interaction information;

[0040] Performing instruction recognition processing according to the target interaction information to generate a recognition result, and performing an interaction operation according to the recognition result to output response information;

[0041] The response information is fed back to the smart glasses for display.

[0042] A third invention of the present invention provides a pair of smart glasses, comprising a frame;

[0043] The frame is provided with a wireless communication module;

[0044] A lens is arranged in the frame, and a display module is arranged on the lens;

[0045] Both sides of the frame are pivotally connected with temples, and the temples are provided with an image acquisition module, an audio input module, a touch button, a control storage module, an audio output module and a wireless charging module;

[0046] The wireless communication module, the display module, the image acquisition module, the audio input module, the touch button, the control storage module, the audio output module and the wireless charging module are connected via a bus;

[0047] The wireless communication module is used for communication connection with the service terminal.

[0048] Optionally, the display module includes a MicroLED chip, a driving circuit and a touch film;

[0049] The MicroLED chip and the touch film are both connected to the bus through the driving circuit;

[0050] The MicroLED chip is arranged on the inner surface of the lens, the touch film is arranged on the outer surface of the lens, and the driving circuit is arranged on the lens frame.

[0051] Optionally, the wireless charging module includes a radio frequency transmitter, a radio frequency receiver and a battery connected in sequence;

[0052] The radio frequency transmitter is externally arranged on the temple;

[0053] The radio frequency receiver and the battery are both arranged on the temple.

[0054] A fourth aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the interaction method based on smart glasses as described above.

[0055] It can be seen from the above technical solutions that the present invention has the following advantages:

[0056] The first aspect of the above technical solution of the present invention provides an interactive system based on smart glasses, including: smart glasses and a service terminal; the service terminal is equipped with an information preprocessing module, an intelligent assistant module, an application software module, a multimodal processing module and a feedback module; the information preprocessing module and the feedback module are both connected to the smart glasses for communication, the intelligent assistant module is connected to the information preprocessing module, and the application software module and the multimodal processing module are both connected to the intelligent assistant module and the feedback module; the information preprocessing module is used to preprocess the initial interactive information sent by the received smart glasses and output the target interactive information; the intelligent assistant module is used to issue instructions according to the target interactive information Recognition processing is performed to generate recognition results, and according to the recognition results, the application software module and the multimodal large model of the multimodal processing module are selectively called to perform interactive operations and output response information; the feedback module is used to feed back the response information to the smart glasses for display; based on the above scheme, the clarity and reliability of the data are improved by preprocessing the interactive information, and the multimodal large model technology is introduced to achieve more intelligent interactive content generation. The interactive information is used for instruction recognition processing to selectively call the application software module and the multimodal large model to complete the interaction, and the response information is fed back to the smart glasses for display in real time, providing users with more convenient and rich operations and improving the user experience.

[0057] The second aspect of the above technical solution of the present invention provides an interactive method based on smart glasses, which preprocesses the interactive information sent by the smart glasses and performs recognition processing to generate a recognition result, performs interactive operations according to the recognition result and a preset multimodal large model, and outputs response information to the smart glasses for display; based on the above solution, the clarity and reliability of the data are improved by preprocessing the interactive information, the multimodal large model technology is introduced to achieve more intelligent interactive content generation, and the response information is fed back to the smart glasses for display in real time, thereby providing users with more convenient and rich operations and improving the user experience.

[0058] The third aspect of the above technical scheme of the present invention provides a kind of smart glasses, including a frame; a wireless communication module is arranged on the frame; a lens is arranged in the frame, and the lens is provided with a display module; temples are pivotally connected to both sides of the frame, and the temples are provided with an image acquisition module, an audio input module, a touch button, a control storage module, an audio output module and a wireless charging module; the wireless communication module, the display module, the image acquisition module, the audio input module, the touch button, the control storage module, the audio output module and the wireless charging module are connected through a bus; the wireless communication module is used to communicate with the service terminal; based on the above scheme, multiple functional modules are integrated into the smart glasses. This highly integrated design enables the smart glasses to maintain a comfortable wearing experience while being rich in functions, and pre-processes the interactive information through the connected service terminal to improve the clarity and reliability of the data, introduces multimodal large model technology to realize more intelligent interactive content generation, and displays the response information in the smart glasses in real time, thereby providing users with more convenient and rich operations and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 A schematic diagram of the structure of an interactive system based on smart glasses provided in Embodiment 1 of the present invention;

[0061] Figure 2 A flowchart of an interactive system based on smart glasses provided in Embodiment 1 of the present invention;

[0062] Figure 3 A flowchart of a method for interacting with smart glasses provided in the second embodiment of the present invention;

[0063] Figure 4 A schematic diagram of the structure of smart glasses provided in Embodiment 3 of the present invention;

[0064] Figure 5 A schematic diagram of the structure of a display module provided in Embodiment 3 of the present invention;

[0065] In the figure:

[0066] 1. Frame; 2. Lens; 3. Wireless communication module; 4. Display module; 401. MicroLED chip; 402. Driving circuit; 403. Touch film; 5. Temple; 6. Image acquisition module; 7. Audio input module; 8. Touch button; 9. Control storage module; 10. Audio output module; 11. Wireless charging module; 1101. Radio frequency receiver; 1102. Battery. DETAILED DESCRIPTION

[0067] The embodiments of the present invention provide an interactive system and method based on smart glasses, and smart glasses, which improve the user experience when the user interacts with the smart glasses.

[0068] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] See also Figure 1 and Figure 2 , Embodiment 1 of the present invention provides an interactive system based on smart glasses, including: smart glasses and a service terminal;

[0070] The service terminal is equipped with an information pre-processing module, an intelligent assistant module, an application software module, a multi-modal processing module and a feedback module;

[0071] The information preprocessing module and the feedback module are both connected to the smart glasses for communication, the smart assistant module is connected to the information preprocessing module, and the application software module and the multimodal processing module are both connected to the smart assistant module and the feedback module.

[0072] It should be noted that the service terminals include but are not limited to mobile phones, computers, smart home terminals and other terminals that can communicate and interact with smart glasses.

[0073] The information preprocessing module is used to preprocess the initial interaction information received from the smart glasses and output target interaction information.

[0074] It should be noted that the information preprocessing module can receive initial interactive information such as data or instructions transmitted between the smart glasses and the server through communication connection with the smart glasses, including but not limited to images, text, audio and other forms. After receiving the initial interactive information, in order to improve the clarity and reliability of the interactive information, it is preprocessed. The preprocessing may include data denoising, data alignment, data format conversion and other processing steps, thereby outputting the target interactive information.

[0075] The intelligent assistant module is used to perform instruction recognition and processing according to the target interaction information, generate recognition results, and selectively call the multimodal large model of the application software module and the multimodal processing module to perform interactive operations and output response information based on the recognition results.

[0076] It should be noted that the application software module can be understood as an App with corresponding functions on the service terminal, and a multimodal large model is deployed on the multimodal processing module. The multimodal large model can choose a high-performance, low-power embedded processor, such as the ARM Cortex series or a dedicated AI chip, to perform the computing tasks required by the large model. It can also use model pruning, quantization and other technologies to reduce the model size and computing requirements, and can also use frameworks such as ONNX for model optimization; a multimodal large model refers to a multimodal large model that can support multiple types of data (such as text, images, audio, etc.) as input and output, such as the GPT-4o large model, the Pangu large model and other multimodal large models. Given that GPT4o is mainly trained on English information and the Pangu large model is mainly trained on Chinese information, the above two models are not very sensitive and understandable to other languages. You can also use existing multilingual models to parse speech-to-text information or train your own multimodal multilingual large model based on Llama3 to achieve multilingual support for smart glasses. The target interaction information may contain relevant instructions issued by the user through the smart glasses. After receiving the target interaction information sent by the information preprocessing module, the intelligent assistant module performs instruction recognition processing on the target interaction information to parse the specific instruction request, thereby generating a recognition result, and distributes the corresponding interaction operation based on the recognition result, directly calls the application software module, or calls the multimodal large model of the multimodal processing module, so as to complete the interaction operation corresponding to the interaction information and output the response information of the interaction result, thereby realizing more intelligent interaction content generation.

[0077] The feedback module is used to feed back the response information to the smart glasses for display.

[0078] It should be noted that the feedback module is communicatively connected with the smart glasses, and can transmit the response information output by the application software module or the multimodal processing module to the user end where the smart glasses are located in real time, thereby improving the real-time and smoothness of the interaction, thereby enhancing the user experience.

[0079] In a specific embodiment, the smart glasses are provided with a wireless communication module, an image acquisition module, an audio input module, an audio output module, a control storage module, a wireless charging module and a display module connected via a bus;

[0080] The wireless communication module is communicatively connected with the information preprocessing module and the feedback module;

[0081] The image acquisition module is used to acquire the initial image in the initial interaction information;

[0082] The audio input module is used to collect the initial audio in the initial interaction information;

[0083] The audio output module is used to display the response audio of the response information;

[0084] The display module is used to display the response text and / or response image of the response information;

[0085] Control the storage module to store the initial image and the initial audio, and control the audio output module and the display module to display the response information;

[0086] The wireless charging module is used for power supply.

[0087] It should be noted that when the control storage module receives the initial image captured by the image acquisition module and the initial audio captured by the audio input module, it performs preliminary compression and stores it as initial interaction information. After receiving the response information fed back by the service terminal, it controls the audio output module and the display module to display the corresponding content respectively according to the information format of the response information.

[0088] In a specific implementation, the information preprocessing module is specifically used to:

[0089] Receiving initial interaction information sent by the smart glasses;

[0090] If the initial interaction information is unimodal information, data denoising is performed on the initial interaction information to output the target interaction information;

[0091] If the initial interaction information is multimodal information, data alignment is performed according to the timestamp of the initial interaction information, and then data denoising is performed to output the target interaction information.

[0092] It should be noted that the initial interaction information sent by the smart glasses includes an initial image and / or an initial audio. It can be understood that the initial image includes an initial picture or an initial video. In specific implementation, the user transmits the initial image and / or the initial audio as interaction information to the information preprocessing module through the wireless communication module according to the current environment. If the initial interaction information is unimodal information, that is, it is understood as a single type of information with only an initial image or an initial audio, then the initial interaction information is directly subjected to voice denoising or image denoising to output the target interaction information. If the initial interaction information is multimodal information, that is, it is understood as the simultaneous existence of two or more types of data, namely, the initial image and the initial audio, considering that the reception and propagation time of the initial image and the initial audio are different, in order to synchronize data from different sources and ensure the consistency of the data in time and space, it is ensured that the time when the initial audio is sent is synchronized with the time when the initial image is collected, and the initial image and the initial audio are realigned and synchronized according to the timestamps of the initial image and the initial audio, and then data denoising is performed to output the target interaction information.

[0093] Speech denoising can reduce the interference of background noise and make the audio content easier to understand. Speech denoising is the process of improving speech quality by reducing or eliminating noise in speech signals. In specific implementation, speech denoising can be achieved by using a time domain denoising method through a moving average filter. The window size is dynamically adjusted according to the signal characteristics. The process of speech denoising can be described as follows:

[0094] ;

[0095] In the formula, is the target audio, is the initial audio, is the window radius (indicates the radius forward and backward from the current point points);

[0096] In another implementation, speech denoising can use generative adversarial networks (GANs) for speech enhancement and denoising. GAN generates high-quality denoised speech through adversarial training of the generator and the discriminator, which can maintain the naturalness and clarity of the speech while removing noise. Using Wave-U-Net, an end-to-end speech denoising model based on the U-Net architecture, the speech waveform is directly operated in the time domain to perform denoising processing. It can effectively remove noise while maintaining speech quality, and is suitable for real-time speech processing;

[0097] Image denoising can significantly improve the visual quality of images by removing noise from them, making them clearer and more delicate. Image denoising can also provide a better foundation for subsequent image processing steps of multimodal large models, and improve the accuracy and robustness of the content generated by multimodal large models. In specific implementation, image denoising can adopt the spatial domain denoising method, which is implemented using Gaussian filters. The process of image denoising can be described as follows:

[0098] ;

[0099] in, ;

[0100] In the formula, is the target image, is the initial image, is the weight of the Gaussian kernel, is the standard deviation; The filter center is The offset of the direction to the neighboring point, The filter center is The offset of the direction to the neighboring point;

[0101] In another implementation, image denoising can use the image denoising models NID and RIDNet based on deep convolutional neural networks, which mainly use residual learning and attention mechanisms to improve the denoising performance, effectively capture the details in the image, and retain more edge information; the Noise2Noise model can be used to learn denoising by training only with noisy image pairs without the need for clean reference images.

[0102] In a specific implementation, the target interaction information includes target audio and / or target image; the intelligent assistant module is specifically used to:

[0103] If the target interaction information contains target audio, the target audio is converted into instruction text through a preset speech recognition model;

[0104] Perform lexical analysis on the instruction text based on the preset SpaCy library to generate multiple instruction phrases;

[0105] Use the preset NLTK library to perform part-of-speech tagging on each instruction phrase and output the part-of-speech of each instruction phrase;

[0106] Determine the dependency relationship between each instruction phrase based on the part of speech of each phrase through the preset AllenNLP model analysis;

[0107] According to the dependency relationship, the preset SRL model is used to label the semantic roles of each instruction phrase and determine the labeling results;

[0108] If both the execution object and the execution action exist in the annotation result, the execution object and the execution action are combined to construct the operation instruction;

[0109] If the operation instruction successfully matches the preset instruction word library and the target interactive information does not contain the target image, the corresponding application software module is called according to the operation instruction to perform the interactive operation and output the response information;

[0110] If the operation instruction successfully matches the preset instruction word library and the target interactive information contains the target image, the corresponding application software module is called according to the operation instruction to perform the interactive operation on the target image and output the response information;

[0111] If the operation instruction fails to match the preset instruction word library and the target interactive information does not contain the target image, the operation instruction is encapsulated into a prompt text, and the multimodal large model of the multimodal processing module is called to perform the interactive operation based on the prompt text and output the response information;

[0112] If the operation instruction does not successfully match the preset instruction vocabulary and the target interactive information has a target image, the operation instruction is encapsulated into a prompt text, and the multimodal large model of the multimodal processing module is called to perform the interactive operation based on the prompt text and the target image, and the response information is output;

[0113] If the annotation result does not contain both the execution object and the execution action and the target interaction information does not contain the target image, a preset auxiliary model generates a prompt text according to the default function instruction and sends it to the multimodal large model of the multimodal processing module for processing, and outputs the response information;

[0114] If the annotation result does not contain both the execution object and the execution action and the target interaction information contains the target image, or the target interaction information contains only the target image, a prompt text is generated according to the default function instruction through a preset auxiliary model, and is sent to the multimodal large model of the multimodal processing module for processing together with the target image, and response information is output.

[0115] It should be noted that, if the initial interaction information sent by the smart glasses includes an initial image and / or an initial audio, then correspondingly, the target interaction information includes a target audio and / or a target image; when the target interaction information only includes a target image, a default function instruction (such as image recognition) is used to generate a corresponding prompt text through a preset AR auxiliary model, and a multimodal large model is called through a multimodal processing module to perform an interactive operation based on the prompt text and the target image, and a response information is output;

[0116] When the user gives the initial audio with the command content through the audio input module, after the voice noise reduction, the target audio exists in the target interaction information, and the preset speech recognition model, such as the Whisper speech recognition model launched by OpenAI, is called to convert the target audio into Convert to command text , the process can be described as follows:

[0117] ;

[0118] After the command text is obtained through speech recognition, the preset SpaCy natural language text processing library is first used to perform lexical analysis on the command text, and the command text is segmented into several command phrases, such as "open file" is segmented into "open" and "file";

[0119] Secondly, the preset NLTK natural language processing toolkit is used to mark the part of speech of each command phrase, for example, POS("open") = verb (V), POS("file") = noun (N), thereby outputting the part of speech of each command phrase;

[0120] Next, the preset AllenNLP software package is used to analyze and determine the dependency relationship between each instruction phrase according to the part of speech of each phrase, for example, DEP ("file", "open") = direct object (dobj). Dependency analysis can help understand the structure of the sentence;

[0121] After that, the SRL model based on BiLSTM and CRF provided by AllenNLP is used to complete semantic role labeling in combination with dependency relationships. The semantic role labeling focuses on labeling execution objects and execution actions, such as SRL ("open") = action (Action), SRL ("file") = object (Object), and the labeling results are output;

[0122] Based on the above annotation results, when the annotation results of the target audio do not contain both the execution object and the execution action, it is understood that there is no specific instruction in the target audio. At this time, if the target interaction information does not contain a target image, the default function instruction (such as no instruction prompt) is used to generate a corresponding prompt text through the preset AR auxiliary model, and the prompt text is sent to the multimodal large model of the multimodal processing module for processing and outputting response information. If the target interaction information contains a target image, the default function instruction (such as image recognition) is used to generate a corresponding prompt text through the preset AR auxiliary model, and the prompt text and the target image are sent to the multimodal large model for processing and outputting response information. The default function instruction can be customized. If the annotation result contains both "execution object" and "execution action", it can be determined that there are specific instructions in the target audio, and the execution object and the execution action are further combined to construct an operation instruction.

[0123] A plurality of preset instruction words are provided in the preset instruction word library, such as navigation, translation, etc., which can be set according to the application software module on the service terminal so as to match the corresponding application software module and then call it. The operation instruction is matched through the preset instruction word library. In the specific implementation, the keyword matching and other methods can be adopted. If the match is successful, the corresponding application software module is called according to the operation instruction, and the application software module performs the interactive operation according to the operation instruction or based on the operation instruction and the target image through the application software module, so as to output the response information. For example, if the execution object is weather and the execution action is viewing, the weather App or similar weather application software module of the service terminal is directly called to return the information. In the process of calling the application software module for processing, refer to Figure 2 , and can also determine whether to generate and execute self-related control instructions such as firmware update, or external control instructions for operating smart homes connected through service terminal communication according to demand settings. If not, the response information can be output by completing the interaction on the application software module;

[0124] If the match is not successful, it is processed by the multimodal processing module, and the operation instruction is encapsulated and converted into a clear, concise and easy-to-understand prompt text, so that the multimodal large model can clearly know the next operation or expected behavior. The multimodal large model performs data processing based on the prompt text or the prompt text and the target image, and then outputs the response information. For example, if the execution object is an image and the execution action is classification, the generated prompt text is "There is currently an image screen. Your task is to classify this image and give the reason for the classification result". The prompt text and the current target image are input into the multimodal large model to obtain the response generation output;

[0125] Through a series of natural language processing steps, user instructions are accurately parsed, the execution subjects and execution actions in the instructions are identified, and the powerful processing capabilities of the multimodal large model and the extensive calls of application software modules are effectively integrated to automatically execute corresponding tasks, providing a more intelligent and efficient solution. Whether it is text, images or other forms of multimedia content, users can quickly get response information, which greatly improves the speed of information acquisition and utilization efficiency.

[0126] In a more specific embodiment, the multimodal processing module is used to:

[0127] When receiving the target image sent by the intelligent assistant module, the preset multimodal large model is called to perform image recognition on the target image and determine the image keyword;

[0128] When receiving the prompt text sent by the intelligent assistant module, the multimodal large model is used to perform named entity recognition on the prompt text to determine the text keywords;

[0129] Taking image keywords and / or text keywords as target keywords, generating topic texts based on the target keywords based on the multimodal large model, outputting multiple topic texts and sorting them, and determining the response text;

[0130] Generate response images and / or response audios according to target topic texts through a multimodal large model;

[0131] The response text, the response image, or / and the response audio are output as the response information.

[0132] It should be noted that in the multimodal large model of the multimodal processing module, if the target image sent by the intelligent assistant module is received, the target recognition operation of the picture or video is performed to identify the target of the picture or video. The loss function of the training process can be composed of two parts, namely, the classification loss and and positioning loss , classification loss is used to measure the difference between the category predicted by the model and the true category; positioning loss is used to measure the difference between the predicted bounding box parameters and the true bounding box parameters. The whole process can be described as follows:

[0133] ;

[0134] in, ;

[0135] In the formula, is the recognition probability, is the object label, is a hyperparameter used to weigh the relative importance of classification loss and localization loss, For the region, is the boundary label, , , , are the coordinate values ​​of the four rectangle boundaries;

[0136] After minimizing the loss function, the recognition result is the object in the environment, that is, the image keyword of the current environment. ;

[0137] If the prompt text sent by the intelligent assistant module is received, the input sequence of the prompt text Perform named entity recognition (NER) to obtain text keywords , named entity recognition can identify entities in text that represent specific meanings according to interaction requirements. The specific process can be customized. The specific process of named entity recognition can be described as follows:

[0138] ;

[0139] Then the above image keywords and / or text keywords are used as target keywords. , for example, the merging process of image keywords and text keywords can be described as follows:

[0140] ;

[0141] Then the multimodal large model generates text for specific topics based on the target keywords and outputs multiple topic texts. , and then through As the output text, the Ranker selects the best topic text output and obtains the response text by maximizing the result of the Ranker. The specific process is described as follows:

[0142] ;

[0143] in, ;

[0144] Finally, the response text is obtained to prompt the user, and the response text (sequence ) Generate a response image or a response video. It can be understood that the multimodal large model can adaptively determine the generation of a response image or a response video through internal logic. The specific process can be described as follows:

[0145] .

[0146] In an embodiment of the present invention, an interactive system based on smart glasses includes: smart glasses and a service terminal; the service terminal is equipped with an information preprocessing module, an intelligent assistant module, an application software module, a multimodal processing module and a feedback module; the information preprocessing module and the feedback module are both communicatively connected to the smart glasses, the intelligent assistant module is connected to the information preprocessing module, and the application software module and the multimodal processing module are both connected to the intelligent assistant module and the feedback module; the information preprocessing module is used to preprocess the initial interactive information received from the smart glasses and output target interactive information; the intelligent assistant module is used to perform instruction recognition processing according to the target interactive information, generate recognition results, and selectively call the multimodal large model of the application software module and the multimodal processing module to perform interactive operations according to the recognition results, and output response information; the feedback module is used to feed back the response information to the smart glasses for display. Based on the above scheme, the clarity and reliability of data are improved by preprocessing the interactive information, and the multimodal large model technology is introduced to realize more intelligent interactive content generation. The interactive information is used for command recognition and processing to selectively call the application software module and the multimodal large model to complete the interaction, and the response information is fed back to the smart glasses for display in real time, thereby providing users with more convenient and rich operations and improving the user experience.

[0147] See also Figure 3 , Figure 3 A flowchart of a method for interacting with smart glasses provided in accordance with the second embodiment of the present invention.

[0148] The present invention provides an interactive method based on smart glasses, comprising:

[0149] Step 301: when receiving initial interaction information sent by the smart glasses, pre-process the initial interaction information and output target interaction information;

[0150] Step 302: Perform command recognition processing according to the target interaction information to generate a recognition result, and perform interactive operations according to the recognition result and the preset multimodal large model to output response information;

[0151] Step 303: Feedback the response information to the smart glasses for display.

[0152] In an embodiment of the present invention, the interactive information sent by the smart glasses is preprocessed and recognized to generate recognition results, interactive operations are performed according to the recognition results and the preset multimodal large model, and response information is output to the smart glasses for display. Based on the above scheme, the clarity and reliability of the data are improved by preprocessing the interactive information, the multimodal large model technology is introduced to achieve more intelligent interactive content generation, and the response information is fed back to the smart glasses in real time for display, thereby providing users with more convenient and rich operations and improving the user experience.

[0153] See also Figure 4 , Figure 4 A schematic diagram of the structure of smart glasses provided in Embodiment 3 of the present invention.

[0154] The present invention provides a pair of smart glasses, comprising a frame 1;

[0155] A wireless communication module 3 is provided on the mirror frame 1;

[0156] A lens 2 is arranged in the lens frame 1, and a display module 4 is arranged in the lens 2;

[0157] Both sides of the frame 1 are pivotally connected with temples 5, and the temples 3 are provided with an image acquisition module 6, an audio input module 7, a touch button 8, a control storage module 9, an audio output module 10 and a wireless charging module 11;

[0158] The wireless communication module 3, the display module 4, the image acquisition module 6, the audio input module 7, the touch button 8, the control storage module 9, the audio output module 10 and the wireless charging module 11 are connected via a bus;

[0159] The wireless communication module 3 is used for communication connection with the service terminal.

[0160] It should be noted that the wireless communication module can adopt wireless communication methods such as WiFi / Bluetooth 6, so that the smart glasses can communicate and transmit data efficiently with the service terminal, realize functions such as notification synchronization and data transmission, and support real-time messaging, video conferencing and remote collaboration.

[0161] The image acquisition module can use a camera, and in specific implementation can include two miniature 1080p high-definition fisheye cameras, which can capture a wider picture and related image information, and can cooperate with the image recognition technology of a multimodal large model to realize object recognition, text scanning and other functions.

[0162] The audio input module can be in the form of a microphone. In a specific implementation, four micro-electromechanical system (MEMS) microphones form a microphone array. The four microphones are distributed in different positions of the smart glasses, preferably at both ends of the two temples, which can capture sound more accurately and perform spatial positioning.

[0163] The audio output module provides audio output function and supports calls and audio playback. In the specific implementation, it includes an audio amplifier and a bone conduction sensor. Two audio amplifiers and bone conduction sensors are set on each temple. Multiple audio amplifiers can support multi-channel audio output, achieve stereo effect, and improve the user's auditory experience. The bone conduction playback technology transmits sound to the skull, bypassing the external auditory canal and middle ear to directly stimulate the inner ear, thereby realizing the transmission of audio information. It can make smart glasses provide a more comfortable and safer auditory experience, and more suitable for long-term wear.

[0164] The display module provides visual information display, enabling users to see augmented reality (AR) content, notifications, navigation information, etc.

[0165] The control storage module can be integrated with a glasses chip and an eMMC memory. In the specific implementation, the glasses chip can adopt a WTM-8 series chip; the control storage module can store the initial image and the initial audio, and after receiving the response information fed back by the service terminal, control the corresponding module to display the corresponding content according to the information format of the response information.

[0166] The wireless charging module provides energy for the smart glasses, ensuring their normal operation and battery life.

[0167] In one embodiment, see Figure 5 , the display module 4 includes a MicroLED chip 401, a driving circuit 402 and a touch film 403;

[0168] The MicroLED chip 401 and the touch film 403 are both connected to the bus through the driving circuit 402;

[0169] The MicroLED chip 401 is disposed on the inner surface of the lens 2 , the touch film 403 is disposed on the outer surface of the lens, and the driving circuit 402 is disposed on the lens frame 1 .

[0170] It should be noted that in the specific implementation, the display module can be a small refractive MicroLED display touch screen, including a MicroLED chip, a driving circuit and a transparent touch film. MicroLED uses LED as a light source with lower power consumption and longer service life. Each LED light source is an independent pixel, which enables MicroLED to achieve higher contrast and wider color gamut. Users can touch one side of the touch film and the touch button on the temple to perform touch adjustment of information input and function selection, such as waking up the glasses.

[0171] In one implementation, the integration process of the display module and the lens is summarized as follows: the MicroLED chip is integrated with the driving circuit to form a complete MicroLED display, and the parameters such as the size, resolution and brightness of the MicroLED display meet the design requirements of the lens; the touch film is integrated into the MicroLED display using an adhesive, and it is necessary to ensure that the two fit tightly without affecting the display effect, and to ensure electrical connection and mechanical stability; the MicroLED display and the touch film are integrated into the lens, and the driving circuit uses the housing of the semi-frame circuit to adapt to the shape of the frame and accommodate all necessary electronic components without adding too much weight; the electrical connection uses connecting elements such as nanowires and flexible circuit boards; the above components need to be packaged to ensure durability and waterproof performance, and the packaging material should have a certain mechanical strength;

[0172] It is understandable that "semi-frame circuit" generally refers to a circuit design in which part of the circuit board or circuit component is surrounded or supported by a frame or supporting structure, while other parts may be exposed to the outside or supported by a lighter structure; the housing of a semi-frame circuit generally refers to a housing or external structure used to encapsulate and protect the semi-frame circuit. This housing can be used in various electronic devices to protect the circuit, provide structural support, and prevent environmental factors (such as dust, moisture, etc.) from having adverse effects on the circuit. The semi-frame housing may only cover part of the circuit, while other parts are exposed to the outside or supported by other means. This design can reduce weight or save materials.

[0173] In one embodiment, see Figure 4 , the wireless charging module 11 includes a radio frequency transmitter, a radio frequency receiver 1101 and a battery 1102 connected in sequence;

[0174] The radio frequency transmitter is externally arranged on the temple 4;

[0175] The radio frequency receiver 1101 and the battery 1102 are both disposed on the temple 4 .

[0176] It should be noted that in the specific implementation, the wireless charging module can be based on radio frequency (RF) wireless charging technology. By integrating a micro-sized RF wireless charging module for power supply, the battery life of smart glasses can be improved. Compared with the charging distance of traditional smart glasses, the wireless charging distance is longer. For example, Energous' WattUp technology based on wireless RF charging technology can achieve wireless charging within a few meters. By using RF energy transmission, it can provide power for small devices (such as wearable devices and IoT sensors). WattUp's receiver chips and modules are optimized and very small, suitable for integration into micro devices. It can provide a convenient and efficient wireless charging solution, ensure a lightweight appearance, and provide power to the device without physical connection; the power generator The sending end can use an external power supply with an embedded enhanced RF transmitter plug. The RF transmitter is used to generate and send RF signals of a specific frequency, converting electrical energy into RF electromagnetic waves. The transmission frequency is usually between 900MHz and 2.4GHz. These frequencies can effectively transmit electrical energy and have little impact on the human body. An antenna system is usually provided in the RF receiver to receive the RF signal sent from the RF transmitter, and rectify and adjust the received RF signal into direct current for supplying smart glasses. Before energy transmission is realized, spatial positioning is required between the transmitting end and the receiving end. Therefore, it is necessary to embed a "beacon antenna" and a "receiving antenna array" in the RF receiver. After positioning, the RF transmitter transmits the RF signal to the RF receiver in a direction.

[0177] In a specific embodiment, the smart glasses are provided with a battery compartment for installing batteries. The battery compartment can adopt a magnetic opening and closing mechanism. The user only needs to apply light force to open the cover and replace or remove the battery. This design is simple and user-friendly, and can also reduce mechanical wear.

[0178] Understandably, smart glasses have a wide range of uses, for example: in healthcare, smart glasses serve as surgical aids, providing critical real-time information during surgical operations, such as the patient's vital signs, CT / MRI images superimposed on the surgical area, and doctors can communicate with remote experts in real time through smart glasses to obtain instant advice and support; in industry and manufacturing, they provide technicians with real-time guidance on equipment maintenance, display parts that need to be inspected or replaced through AR technology, and provide real-time guidance on parts assembly on the production line to improve assembly efficiency and accuracy; in retail and shopping, users can complete virtual try-ons on smart glasses, and when purchasing clothing or accessories, users can see the wearing effect in real time through smart glasses, reducing the time and trouble of trying on. Provide personalized product recommendations based on user preferences and shopping history; in terms of assistive tools, smart glasses can be used as an aid for the visually impaired, helping them to better navigate and understand their surroundings through voice description and environmental perception technology; smart glasses can be used as an aid for the hearing impaired, helping them to better participate in conversations and understand voice content through real-time subtitles and speech-to-text functions; in terms of fitness and health monitoring: smart glasses can provide real-time movement guidance and correction during fitness training to help users achieve the best training results. They can also monitor users' physiological indicators in real time, such as heart rate, number of steps, calorie consumption, etc., and provide health advice.

[0179] In an embodiment of the present invention, multiple functional modules are integrated into the smart glasses. This highly integrated design enables the smart glasses to be functionally rich while still maintaining a comfortable wearing experience.

[0180] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the interaction method based on smart glasses as described in any of the above embodiments.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0184] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive system based on smart glasses, It is characterized in that include: Smart glasses and service terminals; The service terminal is equipped with an information preprocessing module, an intelligent assistant module, an application software module, a multimodal processing module and a feedback module; The information preprocessing module and the feedback module are both connected to the smart glasses in communication, the smart assistant module is connected to the information preprocessing module, and the application software module and the multimodal processing module are both connected to the smart assistant module and the feedback module; The information preprocessing module is used to preprocess the received initial interaction information sent by the smart glasses and output target interaction information; The intelligent assistant module is used to perform instruction recognition processing according to the target interaction information, generate a recognition result, and selectively call the application software module and the multimodal large model of the multimodal processing module to perform interactive operations according to the recognition result, and output response information; The feedback module is used to feed back the response information to the smart glasses for display.

2. The interactive system based on smart glasses according to claim 1, It is characterized in that The smart glasses are provided with a wireless communication module, an image acquisition module, an audio input module, an audio output module, a control storage module, a wireless charging module and a display module connected via a bus; The wireless communication module is communicatively connected with the information preprocessing module and the feedback module; The image acquisition module is used to acquire the initial image in the initial interaction information; The audio input module is used to collect the initial audio in the initial interaction information; The audio output module is used to display the response audio of the response information; The display module is used to display the response text and / or the response image of the response information; The control storage module is used to store the initial image and the initial audio, and control the audio output module and the display module to display the response information; The wireless charging module is used for supplying power.

3. The interactive system based on smart glasses according to claim 1, It is characterized in that The information preprocessing module is specifically used for: Receiving interaction information sent by the smart glasses; If the initial interaction information is unimodal information, performing data denoising on the initial interaction information and outputting target interaction information; If the initial interaction information is multimodal information, data alignment is performed according to the timestamp of the initial interaction information, and then data denoising is performed to output the target interaction information.

4. The interactive system based on smart glasses according to claim 1, It is characterized in that The target interaction information includes target audio and / or target image; the intelligent assistant module is specifically used for: If the target interaction information contains target audio, converting the target audio into instruction text through a preset speech recognition model; Performing lexical analysis on the instruction text based on a preset SpaCy library to generate multiple instruction phrases; Using a preset NLTK library to perform part-of-speech tagging on each of the instruction phrases, and outputting the phrase part-of-speech of each of the instruction phrases; Determine the dependency relationship between each of the instruction phrases based on the part of speech of each phrase through a preset AllenNLP model analysis; According to the dependency relationship, a preset SRL model is used to perform semantic role labeling on each of the instruction phrases to determine a labeling result; If the annotation result contains both an execution object and an execution action, the execution object and the execution action are combined to construct an operation instruction; If the operation instruction successfully matches the preset instruction word library and the target interactive information contains a target image, calling the corresponding application software module according to the operation instruction to perform an interactive operation on the target image and outputting response information; If the operation instruction fails to match the preset instruction word library and the target interactive information contains a target image, the operation instruction is encapsulated into a prompt text, and the multimodal large model of the multimodal processing module is called to perform an interactive operation based on the prompt text and the target image, and output response information; If the annotation result does not contain both the execution object and the execution action and the target interaction information contains a target image, or the target interaction information contains only a target image, a prompt text is generated according to the default function instruction through a preset auxiliary model, and is sent to the multimodal large model of the multimodal processing module for processing in combination with the target image, and response information is output.

5. The interactive system based on smart glasses according to claim 4, It is characterized in that The multimodal processing module is used for: When the target image sent by the intelligent assistant module is received, a preset multimodal large model is called to perform image recognition on the target image to determine the image keyword; When the prompt text sent by the intelligent assistant module is received, named entity recognition is performed on the prompt text through the multimodal large model to determine the text keywords; The image keyword and / or the text keyword are used as target keywords, and a topic text is generated according to the target keyword based on the multimodal macro model, and multiple topic texts are output and sorted to determine a response text; Generate a response image or / and a response audio according to the target topic text through the multimodal large model; The response text, response image and / or response audio are output as response information.

6. An interactive method based on smart glasses, It is characterized in that include: When receiving the initial interaction information sent by the smart glasses, preprocessing the initial interaction information and outputting target interaction information; Performing instruction recognition processing according to the target interaction information to generate a recognition result, and performing an interaction operation according to the recognition result to output response information; The response information is fed back to the smart glasses for display.

7. A kind of smart glasses, It is characterized in that Including frame; The frame is provided with a wireless communication module; A lens is arranged in the frame, and a display module is arranged on the lens; Both sides of the frame are pivotally connected with temples, and the temples are provided with an image acquisition module, an audio input module, a touch button, a control storage module, an audio output module and a wireless charging module; The wireless communication module, the display module, the image acquisition module, the audio input module, the touch button, the control storage module, the audio output module and the wireless charging module are connected via a bus; The wireless communication module is used for communication connection with the service terminal.

8. The smart glasses according to claim 7, It is characterized in that The display module includes a MicroLED chip, a driving circuit and a touch film; The MicroLED chip and the touch film are both connected to the bus through the driving circuit; The MicroLED chip is arranged on the inner surface of the lens, the touch film is arranged on the outer surface of the lens, and the driving circuit is arranged on the lens frame.

9. The smart glasses according to claim 7, It is characterized in that The wireless charging module includes a radio frequency transmitter, a radio frequency receiver and a battery connected in sequence; The radio frequency transmitter is externally arranged on the temple; The radio frequency receiver and the battery are both arranged on the temple.

10. A computer device, It is characterized in that It comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the interaction method based on smart glasses as claimed in claim 6.

Citation Information

Cited By

  • Interaction method and device, wearable equipment and storage medium

    CN119668412A

  • Interaction control method, system and equipment for intelligent glasses

    CN120335613A

  • An interaction control method, system and device for smart glasses

    CN120335613B

  • Mixed reality interaction method and system, electronic equipment and storage medium

    CN120747432A

  • A mixed reality interaction method and system, electronic device, storage medium

    CN120747432B