Smart glasses streaming voice dialogue interaction system and method based on large language model

Abstract

The present application discloses a smart glasses streaming voice dialogue interaction system and method based on a large language model, the system comprising: a data acquisition module for collecting interactive information of user and smart glasses voice dialogue and smart glasses resource status information; a data conversion module for processing interactive information using real-time streaming processing technology and adaptively selecting the corresponding type of interactive information recognition model, converting the recognized content into text information; a data processing module for receiving text information and calculating the comprehensive score according to the preset scoring rules, matching the large language model with different model configurations loaded according to the comprehensive score to perform intent recognition and generate reply text; a data interaction module for converting the generated reply text into voice output; a data feedback module for receiving feedback on user interaction satisfaction to optimize the interactive information recognition model or the large language model. The present application can achieve an efficient, accurate and smooth voice interaction experience.

Description

Technical Field

[0001] The present application relates to the field of large language model interaction technology, and specifically to a smart glasses streaming voice dialogue interaction system and method based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, voice interaction has become one of the important ways of human-computer interaction. As an emerging wearable device, smart glasses combine AR / VR technology to provide users with rich visual information display capabilities.

[0003] However, existing smart glasses have many problems in voice interaction, such as low recognition rate, slow response speed, and unnatural dialogue. Although some smart glasses have optimized the model in language interaction, it is still difficult to meet the user's needs for efficient, accurate, and smooth interaction. Therefore, it is of great significance to develop a smart glasses streaming voice dialogue interaction system based on a large language model to achieve efficient, accurate, and smooth voice interaction experience. Summary of the invention

[0004] In order to solve the problems of low recognition rate, slow response speed, unnatural dialogue, etc. in the process of language interaction between users and smart glasses, the present application provides a smart glasses streaming voice dialogue interaction system and method based on a large language model.

[0005] In a first aspect, the present application provides a smart glasses streaming voice dialogue interaction system based on a large language model, comprising:

[0006] A data collection module is used to collect interactive information between the user and the smart glasses, and information about the resource status of the smart glasses during the interactive process of the user and the smart glasses; the interactive information includes user voice information, voice information played by the smart glasses, user visual information, and visual information displayed by the smart glasses;

[0007] A data conversion module is used to process the collected interactive information using real-time streaming processing technology, adaptively select a corresponding type of interactive information recognition model according to the type of the processed interactive information, and convert the recognized content into text information;

[0008] A data processing module is used to receive text information and use a deep learning algorithm to obtain the conversation content type of the interactive information, wherein the conversation content types include: common knowledge interaction type, professional knowledge interaction type, and complex reasoning interaction type; a weighted comprehensive score of the matching score corresponding to the current resource status sufficiency and the complexity of the conversation content in the current text information is calculated according to a preset scoring rule; the preset scoring rule includes: matching the corresponding score based on the preset resource status sufficiency range of the resource status information, and matching the corresponding score based on the preset conversation content complexity of the recognized conversation content type; loading a large language model built into the smart glasses with different model configurations according to the preset threshold range of the obtained comprehensive score; the large language models with different model configurations are large language models with different magnitudes divided according to model scale, number of parameters, and computing power; using the matched large language model to identify the intent of the text information, and generating a reply text in combination with the context;

[0009] The data interaction module converts the generated reply text into voice output;

[0010] The data feedback module is used to receive feedback on user interaction satisfaction and determine whether the user interaction satisfaction reaches the preset satisfaction level. If the user satisfaction is not reached, the model optimization of the interaction information recognition model or the large language model is completed through incremental training until the user interaction satisfaction reaches the preset satisfaction level.

[0011] By adopting the above solution, real-time streaming processing technology and adaptive interactive information recognition technology are used to segment and adaptively recognize the interactive information of the user's continuous speech, reduce delays and improve recognition efficiency; based on the actual interactive resource status and user needs, the large language model with the corresponding model configuration is adaptively matched and loaded to accurately parse the intent and achieve more timely response; multimodal interactive information is collected, and the memory capacity of the large language model is used to combine the context to achieve cross-round context understanding to generate natural reply text; the model is continuously optimized using user feedback to improve the naturalness of the conversation and provide users with a richer and more intuitive interactive experience;

[0012] Preferably, the data processing module is also used to receive text information and calculate whether the data volume of the text information is greater than the preset data volume. If it is greater than the preset data volume, part of the text content is transmitted to the edge device, and the reply text is generated using the large language model of the edge device and transmitted back to the smart glasses.

[0013] By adopting the above solution, considering that the data volume of some voice segments is large, the response time of the large model built into the simple application will be too long, so some interactive information is transmitted to the edge device, and edge computing is used to assist in rapid response.

[0014] Preferably, the data processing module is also used to receive text information and after calculating that the data volume of the text information is greater than a preset data volume, perform natural language processing on the received text information to extract keywords, determine whether the extracted keywords are sensitive information, and transmit part of the text information that does not involve sensitive information to the edge device based on the judgment result.

[0015] By adopting the above solution, taking into account the security risks of data transmitted to edge computing, the sensitive information of the text information is judged, and some text information that does not involve sensitive information is selected to be transmitted to the edge device to ensure the security of user data.

[0016] Preferably, the data processing module is also used to count the time from receiving the text message to generating the reply text. When the counted time exceeds the preset time, a large language model with a higher level configuration is switched, and the reply text is generated using the switched large language model.

[0017] By adopting the above solution, the timeout of reply text generation can be monitored in real time, and a better large language model can be switched in time to better generate reply text, thereby avoiding delays in reply text generation and improving user interaction experience.

[0018] Preferably, the data interaction module is also used to obtain the interaction tone and speed of the user's habits or needs and use them as the tone and speed of the target interaction; and use speech synthesis technology to convert the generated reply text into natural speech output according to the tone and speed of the target interaction.

[0019] By adopting the above solution, user portraits or user needs are taken into consideration, and reply content that meets the user's personalized needs is generated to improve the user interaction experience.

[0020] Preferably, the data interaction module is also used to convert the generated reply text into voice output while displaying the visual information pre-associated with the reply text through smart glasses.

[0021] By adopting the above solution, in addition to voice responses, relevant text or

[0022] Visual content further enhances the user's multi-sensory interactive experience.

[0023] Preferably, it also includes:

[0024] The data prediction and preloading block is also used to use a deep learning algorithm to predict the interaction information of the next time period based on the interaction information of the historical streaming voice conversation between the user and the smart glasses; determine the resource status information of the smart glasses for the next time period; pre-match the corresponding large language model in combination with the predicted interaction information of the next time period and the resource status information of the smart glasses for the next time period, and when the interaction information of the next time period is obtained and the similarity between the received interaction information and the predicted interaction information of the next time period is greater than a preset similarity, the pre-matched large language model is loaded in advance.

[0025] By adopting the above solution, historical data and machine learning algorithms are used to predict future resource conditions and user needs, and based on the prediction results, appropriate model configurations are loaded in advance or switched to, so as to reduce waiting time and improve response speed.

[0026] In a second aspect, the present application provides a smart glasses streaming voice dialogue interaction method based on a large language model, comprising:

[0027] Collecting interactive information of voice dialogue between the user and the smart glasses, and information on the resource status of the smart glasses during the voice dialogue interaction between the user and the smart glasses; the interactive information includes user voice information, voice information played by the smart glasses, user visual information, and visual information displayed by the smart glasses;

[0028] The collected interactive information is processed using real-time streaming technology, and the corresponding type of interactive information recognition model is adaptively selected according to the type of the processed interactive information, and the recognized content is converted into text information;

[0029] Receive text information and use deep learning algorithms to obtain the conversation content type of interactive information, the conversation content types include: common knowledge interaction type, professional knowledge interaction type and complex reasoning interaction type, calculate the weighted comprehensive score of the current resource status sufficiency and the corresponding matching score of the conversation content complexity in the current text information according to the preset scoring rules; the preset scoring rules include: matching the corresponding score based on the preset resource status sufficiency range of the resource status information, and matching the corresponding score based on the preset conversation content complexity of the recognized conversation content type; according to the preset threshold range of the obtained comprehensive score, the large language model built into the smart glasses with different model configurations is loaded; the large language model with different model configurations is a large language model with different magnitudes divided according to the model scale, the number of parameters and the computing power; use the matched large language model to identify the intent of the text information, and generate a reply text in combination with the context;

[0030] Convert the generated response text into speech output;

[0031] Receive feedback on user interaction satisfaction and determine whether the user interaction satisfaction reaches the preset satisfaction level. If the user satisfaction is not reached, optimize the interaction information recognition model or the large language model through incremental training until the user interaction satisfaction reaches the preset satisfaction level.

[0032] By adopting the above solution, efficient speech recognition, natural language processing, feedback mechanism and model dynamic mechanism are integrated to achieve efficient, accurate and smooth voice interaction between users and smart glasses.

[0033] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0034] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0035] In summary, this application has the following beneficial effects:

[0036] 1. Use real-time streaming processing technology and adaptive interactive information recognition technology to recognize interactive information of users' continuous speech in segments, reduce delays and improve recognition efficiency;

[0037] 2. Combine the actual interactive resource status and user needs, adaptively match and load the large language model with the corresponding model configuration, accurately analyze the intent, and achieve more timely response;

[0038] 3. Collect multimodal interaction information to improve the accuracy of user intent recognition, use the memory capacity of the large language model, and combine context understanding to generate more natural response text;

[0039] 4. Collect user feedback and use it to optimize the performance of speech recognition, natural language processing, and speech generation modules, improve the naturalness of conversations, and provide users with a richer and more intuitive interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural diagram of a smart glasses streaming voice dialogue interaction system based on a large language model described in a specific embodiment;

[0041] Figure 2 It is a flowchart of the smart glasses streaming voice dialogue interaction method based on a large language model described in a specific embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] Example 1

[0044] like Figure 1 As shown, an embodiment of the present application discloses a smart glasses streaming voice dialogue interaction system based on a large language model, which specifically includes: a data acquisition module 1, a data conversion module 2, a data processing module 3, a data interaction module 4 and a data feedback module 5, and communication connections are established between each module.

[0045] The data collection module 1 is used to collect the interactive information of the voice dialogue between the user and the smart glasses, and the resource status information of the smart glasses during the voice dialogue interaction between the user and the smart glasses. The interactive information includes user voice information, voice information played by the smart glasses, user visual information, and visual information displayed by the smart glasses.

[0046] Specifically, the data acquisition module uses several hardware components for data collection; among them, the key components include microphone arrays, speakers, and displays installed on smart glasses, such as: multi-channel microphone arrays, which are used to collect user voice input from different directions, and filter background noise through beamforming technology, and convert analog voice signals into digital signals through high-speed A / D converters to ensure the high fidelity of voice data. High-fidelity speakers are used to output voice responses generated by the system to ensure that users can clearly hear voice feedback in different environments. OLED or MicroLED display technology is used to present text or image information related to user conversations, and supports high-definition, low-latency display effects. A high-definition camera is used to capture image information of the user's eye movements and facial expressions during a conversation with the user.

[0047] The data conversion module 2 is used to process the collected interactive information using real-time streaming processing technology, adaptively select a corresponding type of interactive information recognition model according to the type of the processed interactive information, and convert the recognized content into text information.

[0048] Specifically, the data conversion module 2 obtains the interactive information collected by the data collection module 1, and uses real-time streaming processing technology to process the continuous interactive information in the interactive information in segments, such as: the language information of the user in the current 3s time period. The type of interactive information is determined by checking the file format of the processed interactive information or performing metadata inspection and content analysis on the processed interactive information, which is voice information or image information.

[0049] According to the type of the processed interactive information, the corresponding type of interactive information recognition model is adaptively selected, and the corresponding type of interactive information recognition model is used to recognize specific dialogue information, smart glasses display information corresponding to the user's eyeball visual focus (such as: a certain application indicator), user expression information (such as: impatience, doubt) and other contents, and the recognized content is converted into text information. Among them, the corresponding type of interactive information recognition model includes a speech recognition model and an image recognition model. The speech recognition model can adopt the Sherpa Zipformer model, and the image recognition model can adopt a neural network model generated by training with historical image information labeled with smart glasses display information corresponding to the user's eyeball visual focus, user expression information and other contents.

[0050] In addition, in order to better recognize voice or image, the interactive information collected by the data acquisition module 1 is preprocessed, that is, the data conversion module 2 is also used to preprocess the processed interactive information, and the preprocessing includes: data denoising and enhancement, etc. Among them, the excessive noise filtering and gain control of the voice signal can be completed according to the pre-recorded user's voice dialogue, that is, if there are both signals with user voice features and non-user voice signals, the non-user voice signal is noise filtered.

[0051] The data processing module 3 is used to receive text information and use a deep learning algorithm to obtain the conversation content type of the interactive information, calculate the weighted comprehensive score of the matching score corresponding to the abundance of current resource conditions and the complexity of the conversation content in the current text information according to a preset scoring rule, and match and load the large language model built into the smart glasses with different model configurations according to the preset threshold range of the obtained comprehensive score, use the matched large language model to recognize the intent of the text information, and generate a reply text in combination with the context.

[0052] Specifically, in this embodiment, the conversation content types include: common knowledge interaction type, professional knowledge interaction type and complex reasoning interaction type; the conversation content type of the acquired text information is acquired by using the constructed first neural network model, the input of the model is text information, and the output of the model is the conversation content type of the text information, and the first neural network model is trained and generated through historical text information of historically annotated conversation content types.

[0053] The preset scoring rules include: matching the corresponding score based on the preset resource status sufficiency range of the resource status information, and matching the corresponding score based on the preset conversation content complexity of the identified conversation content type;

[0054] The resource status information includes: the power of the smart glasses, CPU usage, memory occupancy, storage space, etc. Each type of resource status information has a corresponding preset resource status sufficiency range and a score corresponding to the preset resource status sufficiency range. The scores of all types of resource status information matching are weighted and integrated to obtain the corresponding score of the preset resource status sufficiency range matching of the resource status information;

[0055] For example, the preset resource status sufficiency ranges corresponding to the power of the smart glasses include: [0, 20%], (20%, 60%], (60%, 80%], (80%, 100%], and the corresponding preset matching scores are 20, 60, 80, 100; finally, the corresponding score A0=x1 (power weight of the smart glasses)*A1 (the power of the smart glasses corresponds to the preset resource status sufficiency range and the corresponding score) + x2 (CPU usage weight)*A2 (the CPU usage corresponds to the preset resource status sufficiency range and the corresponding score)+...+xn (the weight of the nth type of resource status information)*An (the nth type of resource status information corresponds to the preset resource status sufficiency range and the corresponding score).

[0056] Among them, the complexity of the preset dialogue content of the content type is: if it is a common knowledge interaction type, the complexity of the preset dialogue content is low complexity, and the matching score is B1; if it is a professional knowledge interaction type, the complexity of the preset dialogue content is medium complexity, and the matching score is B2; if it is a complex reasoning interaction type, the complexity of the preset dialogue content is high complexity, and the matching score is B3;

[0057] The calculation formula for calculating the weighted comprehensive score of the matching score corresponding to the abundance of the current resource status and the complexity of the conversation content in the current text information according to the preset scoring rules is:

[0058] Comprehensive score C = a*A0+ b*BN

[0059] Where a is the score weight corresponding to the current resource abundance, b is the score weight corresponding to the complexity of the conversation content in the current text information, and BN is one of B1, B2, and B3.

[0060] According to the preset threshold range of the obtained comprehensive score, a large language model built into the smart glasses with different model configurations is matched and loaded; wherein the preset threshold ranges of the comprehensive score C in this embodiment are [C1, C2], (C2, C3], (C3, C4], respectively, corresponding to different model configurations F1\F2\F3; the large language models with different model configurations are large language models with different levels of configuration (F1 lightweight\F2 between lightweight and heavyweight\F3 heavyweight) divided according to model scale, number of parameters and computing power.

[0061] The matching large language model is used to identify the intent of text information and generate reply text based on the context. The matching large language model can analyze the text information through the NLU module to identify the user's intent. Combined with pre-trained models such as BERT, it can more accurately understand the user's semantics and handle complex conversation scenarios. The matching large language model provides context information in multiple rounds of conversations.

[0062] Coherent and natural responses. Use the context manager to record the user's conversation history to ensure the continuation of the conversation content.

[0063] The combined system generates personalized responses by combining user portraits and conversation context. Reinforcement learning algorithm

[0064] The dialogue generation strategy will be adjusted based on user preferences and feedback to make the responses more in line with user needs.

[0065] The data interaction module 4 is used to convert the generated reply text into voice output.

[0066] The data feedback module 5 is used to receive feedback on user interaction satisfaction and determine whether the user interaction satisfaction reaches a preset satisfaction level. When the user satisfaction is not reached, the interactive information recognition model or the large language model is optimized through incremental training until the user interaction satisfaction reaches the preset satisfaction level.

[0067] Specifically, the user interaction satisfaction is the user's satisfaction score for the reply text. The user interaction satisfaction survey can be displayed regularly on the smart glasses screen, and the user's satisfaction score for the reply text can be collected to complete the result judgment of whether the subsequent user interaction satisfaction reaches the preset satisfaction level.

[0068] Example 2

[0069] The difference from Example 1 is that, considering that it may be difficult to achieve smooth voice interaction during the peak period of voice interaction between the user and the smart glasses, in order to further improve the voice dialogue experience between the user and the smart glasses; the system also includes:

[0070] The data processing module 3 is also used to receive text information and calculate whether the data volume of the text information is greater than the preset data volume. If it is greater than the preset data volume, part of the text content is transmitted to the edge device, and the reply text is generated by using the large language model of the edge device and transmitted back to the smart glasses.

[0071] Specifically, since the voice conversations in some time periods of the interactive information of the streaming voice conversations between the user and the smart glasses are relatively dense, the corresponding interactive information data volume is large, so a preset data volume of the text information is set, and the interactive information of each time period is compared with the preset data volume to determine whether to transmit part of the text content to the edge device, among which part of the text content can choose to send part of the continuous text content in time that exceeds the preset data volume to the edge device.

[0072] In addition, considering that data transmission to edge devices may involve data security issues, in order to further protect data security, the data processing module 3 is also used to receive text information and calculate that when the data volume of the text information is greater than the preset data volume, natural language processing is performed on the received text information to extract keywords, and it is determined whether the extracted keywords are sensitive information. The sensitive information is pre-set information about user privacy, such as identity information, etc. According to the judgment result, part of the text information that does not involve sensitive information is transmitted to the edge device.

[0073] Example 3

[0074] The difference from the first embodiment is that, during the voice interaction process, considering the situation that the reply text may not be generated in time, in order to remind or process in time, the system further includes:

[0075] The data processing module 3 is also used to count the time from receiving the text message to generating the reply text. When the counted time exceeds the preset time, it indicates that the current large language model has a fault or the current large language model is unable to process the current text message in time. Therefore, a large language model with a higher level configuration is switched, and the reply text is generated using the switched large language model.

[0076] In addition, if the currently matched large language model is already the largest configured large language model, then a large language model with the same configuration as the standby one is selected for switching, and the switched large language model is used to generate a reply text or generate a prompt message indicating a failure in generating the current reply text.

[0077] Example 4

[0078] The difference from Example 1 is that, in the process of voice interaction, in order to achieve a natural voice closer to the user's habits and improve the naturalness of the conversation, the system also includes:

[0079] The data interaction module 4 is also used to obtain the interaction tone and speed of the user's habits or needs and use them as the tone and speed of the target interaction, and use speech synthesis technology to convert the generated reply text into natural speech output according to the tone and speed of the target interaction.

[0080] The interactive intonation and speech speed of the user's habit or user's needs may be pre-selected before the user interacts, or may be the most commonly used interactive intonation and speech speed when the satisfaction of the user's historical feedback is greater than the second preset satisfaction according to the satisfaction statistics of the user's historical feedback. This embodiment selects to use Vits speech synthesis technology to convert the generated text content into natural speech.

[0081] In addition, in order to enhance the user's multi-sensory interactive experience, the data interaction module 4 is also used to convert the generated reply text into voice output while displaying the visual information pre-associated with the reply text through the smart glasses. Among them, the reply text will be pre-associated with some text special effects or image visual information, such as: the popular science reply text will be associated with a popular science animation, and the popular science animation associated with the reply text will be played synchronously when the smart glasses play the reply text.

[0082] Example 5

[0083] The difference from the embodiment 1 is that, in order to further reduce the waiting time and improve the response speed, the system further includes: a data prediction and preloading block 6;

[0084] The data prediction and preloading block 6 is also used to use a deep learning algorithm to predict the interaction information of the next time period based on the interaction information of the historical streaming voice conversation between the user and the smart glasses, that is, to build a deep learning model, and use the interaction information of the historical streaming voice conversation between the user and the smart glasses to predict the generated interaction information of the next time period.

[0085] The resource status information of the smart glasses in the next period is determined, that is, the resource status information of the smart glasses in the next period can be determined by integrating the total resources and the resource status information of the smart glasses in the current period.

[0086] The corresponding large language model is pre-matched in combination with the predicted interaction information of the next time period and the resource status information of the smart glasses of the next time period. When the interaction information of the next time period is obtained and the similarity between the received interaction information and the predicted interaction information of the next time period is greater than the preset similarity, the pre-matched large language model is loaded in advance.

[0087] like Figure 2 As shown, this embodiment also provides a smart glasses streaming voice dialogue interaction method based on a large language model using the system described in any of the above embodiments, including:

[0088] S1. Collecting interactive information of voice dialogue between the user and the smart glasses, and information on resource status of the smart glasses during the voice dialogue interaction between the user and the smart glasses.

[0089] S2. Use real-time streaming processing technology to process the collected interactive information, adaptively select a corresponding type of interactive information recognition model according to the type of the processed interactive information, and convert the recognized content into text information.

[0090] S3. Receive text information and use a deep learning algorithm to obtain the conversation content type of the interactive information, and calculate a weighted comprehensive score of the matching score corresponding to the abundance of current resource conditions and the complexity of the conversation content in the current text information according to a preset scoring rule; match the large language model built into the smart glasses with different model configurations according to the preset threshold range of the obtained comprehensive score, and use the matching large language model to recognize the intent of the text information, and generate a reply text based on the context.

[0091] S4. Convert the generated reply text into speech output.

[0092] S5. Receive feedback on user interaction satisfaction, and determine whether the user interaction satisfaction reaches a preset satisfaction level. If the user satisfaction is not reached, optimize the interaction information recognition model or the large language model through incremental training until the user interaction satisfaction reaches the preset satisfaction level.

[0093] In addition, in a specific embodiment, step S3 in the method further includes:

[0094] Receive text information and calculate whether the data volume of the text information is greater than the preset data volume. If it is greater than the preset data volume, perform natural language processing on the received text information to extract keywords, and determine whether the extracted keywords are sensitive information. Based on the judgment result, transmit part of the text information that does not involve sensitive information to the edge device, then transmit part of the text content to the edge device, use the large language model of the edge device to generate a reply text and transmit it back to the smart glasses.

[0095] In a specific embodiment, step S3 in the method further includes:

[0096] The time from receiving the text message to generating the reply text is counted. When the counted time exceeds the preset time, a large language model with a higher level configuration is switched, and the reply text is generated using the switched large language model.

[0097] In a specific embodiment, step S4 in the method further includes:

[0098] The interaction tone and speed of the user's habits or needs are obtained and used as the tone and speed of the target interaction; the generated reply text is converted into natural speech output according to the tone and speed of the target interaction using speech synthesis technology; while converting the generated reply text into speech output, the visual information pre-associated with the reply text is displayed through smart glasses.

[0099] In a specific embodiment, the method further comprises:

[0100] A deep learning algorithm is used to predict the interaction information of the next time period based on the interaction information of the historical streaming voice conversations between users and smart glasses; the resource status information of the smart glasses for the next time period is determined; the corresponding large language model is pre-matched in combination with the predicted interaction information of the next time period and the resource status information of the smart glasses for the next time period; when the interaction information of the next time period is obtained and the similarity between the received interaction information and the predicted interaction information of the next time period is greater than a preset similarity, the pre-matched large language model is loaded in advance.

[0101] The embodiment of the present application also discloses a computer-readable storage medium.

[0102] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executes the above-mentioned smart glasses streaming voice dialogue interaction method based on a large language model. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0103] The embodiment of the present application also discloses a computer device.

[0104] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned smart glasses streaming voice dialogue interaction method based on a large language model.

[0105] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A smart glasses streaming voice dialogue interaction system based on a large language model, characterized in that: include: A data collection module is used to collect interactive information between the user and the smart glasses, and information about the resource status of the smart glasses during the interactive process of the user and the smart glasses; the interactive information includes user voice information, voice information played by the smart glasses, user visual information, and visual information displayed by the smart glasses; A data conversion module is used to process the collected interactive information using real-time streaming processing technology, adaptively select a corresponding type of interactive information recognition model according to the type of the processed interactive information, and convert the recognized content into text information; A data processing module is used to receive text information and use a deep learning algorithm to obtain the conversation content type of the interactive information, wherein the conversation content types include: common knowledge interaction type, professional knowledge interaction type, and complex reasoning interaction type; a weighted comprehensive score of the matching score corresponding to the current resource status sufficiency and the complexity of the conversation content in the current text information is calculated according to a preset scoring rule; the preset scoring rule includes: matching the corresponding score based on the preset resource status sufficiency range of the resource status information, and matching the corresponding score based on the preset conversation content complexity of the recognized conversation content type; loading a large language model built into the smart glasses with different model configurations according to the preset threshold range of the obtained comprehensive score; the large language models with different model configurations are large language models with different magnitudes divided according to model scale, number of parameters, and computing power; using the matched large language model to identify the intent of the text information, and generating a reply text in combination with the context; The data interaction module converts the generated reply text into voice output; The data feedback module is used to receive feedback on user interaction satisfaction and determine whether the user interaction satisfaction reaches the preset satisfaction level. If the user satisfaction is not reached, the model optimization of the interaction information recognition model or the large language model is completed through incremental training until the user interaction satisfaction reaches the preset satisfaction level.

2. The smart glasses streaming voice dialogue interaction system based on a large language model according to claim 1 is characterized in that: The data processing module is also used to receive text information and calculate whether the data volume of the text information is greater than the preset data volume. If it is greater than the preset data volume, part of the text content is transmitted to the edge device, and the reply text is generated by using the large language model of the edge device and transmitted back to the smart glasses.

3. The smart glasses streaming voice dialogue interaction system based on a large language model according to claim 2 is characterized in that: The data processing module is also used to receive text information and after calculating that the data volume of the text information is greater than a preset data volume, perform natural language processing on the received text information to extract keywords, determine whether the extracted keywords are sensitive information, and transmit part of the text information that does not involve sensitive information to the edge device based on the judgment result.

4. The smart glasses streaming voice dialogue interaction system based on a large language model according to claim 1, characterized in that: The data processing module is also used to count the time from receiving the text message to generating the reply text. When the counted time exceeds the preset time, a large language model with a higher level configuration is switched, and the reply text is generated using the switched large language model.

5. The smart glasses streaming voice dialogue interaction system based on a large language model according to claim 1 is characterized in that: The data interaction module is also used to obtain the interaction tone and speed of the user's habits or needs and use them as the tone and speed of the target interaction; and use speech synthesis technology to convert the generated reply text into natural speech output according to the tone and speed of the target interaction.

6. The smart glasses streaming voice dialogue interaction system based on a large language model according to claim 1, characterized in that: The data interaction module is also used to convert the generated reply text into voice output while displaying the visual information pre-associated with the reply text through smart glasses.

7. The smart glasses streaming voice dialogue interaction system based on a large language model according to claim 1, characterized in that: Also includes: The data prediction and preloading block is also used to use deep learning algorithms to predict the interaction information of the next period based on the interaction information of the historical user and the streaming voice dialogue of the smart glasses; Determine the resource status information of the smart glasses for the next time period; pre-match the corresponding large language model with the predicted interaction information of the next time period and the resource status information of the smart glasses for the next time period, and load the pre-matched large language model in advance when the interaction information of the next time period is obtained and the similarity between the received interaction information and the predicted interaction information of the next time period is greater than a preset similarity.

8. A method for smart glasses streaming voice dialogue interaction based on a large language model using the system described in claims 1 to 7, characterized in that: include: Collecting interactive information between the user and the smart glasses, and information about the resource status of the smart glasses during the interactive process of the user and the smart glasses; The interactive information includes user voice information, voice information played by smart glasses, user visual information and visual information displayed by smart glasses; The collected interactive information is processed using real-time streaming technology, and the corresponding type of interactive information recognition model is adaptively selected according to the type of the processed interactive information, and the recognized content is converted into text information; Receive text information and use deep learning algorithms to obtain the conversation content type of interactive information, the conversation content types include: common knowledge interaction type, professional knowledge interaction type and complex reasoning interaction type, calculate the weighted comprehensive score of the current resource status sufficiency and the corresponding matching score of the conversation content complexity in the current text information according to the preset scoring rules; the preset scoring rules include: matching the corresponding score based on the preset resource status sufficiency range of the resource status information, and matching the corresponding score based on the preset conversation content complexity of the recognized conversation content type; according to the preset threshold range of the obtained comprehensive score, the large language model built into the smart glasses with different model configurations is loaded; the large language model with different model configurations is a large language model with different magnitudes divided according to the model scale, the number of parameters and the computing power; use the matched large language model to identify the intent of the text information, and generate a reply text in combination with the context; Convert the generated response text into speech output; Receive feedback on user interaction satisfaction and determine whether the user interaction satisfaction reaches the preset satisfaction level. If the user satisfaction is not reached, optimize the interaction information recognition model or the large language model through incremental training until the user interaction satisfaction reaches the preset satisfaction level.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to claim 8.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to claim 8 when executed by the processor.

Images