Intelligent glasses voice interaction method, system and device, and storage medium

Abstract

The application provides a smart glasses voice interaction method and system, equipment and a storage medium, and belongs to the technical field of voice interaction. The method comprises the following steps: determining an interrupt intention based on user voice instruction data and an interrupt trigger signal; generating a standardized task based on the interrupt intention, the user voice instruction data and hardware state data, inputting the standardized task into a double-buffer thread pool, and inserting the standardized task into a task buffer queue in a priority order by a scheduling thread group; acquiring the standardized task from the task buffer queue in a queue order by an execution thread group, performing a target operation based on a task type of the standardized task, generating a standardized execution result, and storing the standardized execution result in a result buffer queue; reading the standardized execution result from the result buffer queue by the scheduling thread group, and converting the standardized execution result into target output data based on the hardware state data. The application can realize accurate and efficient multi-modal interrupt detection and interaction feedback.

Description

Technical Field

[0001] This application belongs to the field of voice interaction technology, and more specifically, relates to voice interaction methods, systems, devices, and storage media for smart glasses. Background Technology

[0002] As a core category of wearable smart devices, smart glasses enable human-computer interaction in a lightweight and portable form. Their voice interaction function, with its hands-free advantage, has become the mainstream interaction method. However, as users' demands for smoother interaction, lower power consumption, and better interference resistance increase, existing smart glasses voice interaction technologies still have many problems. Current technologies mostly use a single sensor to collect trigger signals, which is easily affected by environmental noise and device vibration, leading to a high rate of misinterpretation of interrupted trigger signals and thus invalid interactions. Furthermore, the feedback methods of existing technologies are relatively simple and cannot meet user needs. Summary of the Invention

[0003] The purpose of this application is to provide a voice interaction method, system, device, and storage medium for smart glasses to achieve accurate and efficient multimodal interruption detection and interactive feedback.

[0004] A first aspect of this application provides a voice interaction method for smart glasses, including: In response to the detection of an interrupt trigger signal and user voice command data, the interrupt intention is determined based on the user voice command data and the interrupt trigger signal; The system acquires hardware status data from the smart glasses, generates standardized tasks based on interrupt intent, user voice command data, and hardware status data, and then passes these standardized tasks into a double-buffered thread pool. The double-buffered thread pool includes a scheduling thread group, an execution thread group, a task buffer queue, and a result buffer queue. The scheduling thread group inserts standardized tasks into the task buffer queue according to priority. The execution thread group retrieves standardized tasks from the task buffer queue according to the queue order, performs the target operation based on the task type of the standardized task, and generates standardized execution results. The standardized execution results are stored in the result buffer queue; the scheduling thread group is also used to read the standardized execution results from the result buffer queue, determine the feedback method based on the hardware status data, and convert the standardized execution results into target output data based on the feedback method.

[0005] A second aspect of this application provides a smart glasses voice interaction system, including: The interrupt detection module is used to determine the interrupt intention based on the user voice command data and the interrupt trigger signal in response to the detection of the interrupt trigger signal and user voice command data. The task execution module is used to acquire hardware status data of the smart glasses, generate standardized tasks based on interrupt intent, user voice command data, and hardware status data, and pass the standardized tasks into a double-buffered thread pool. The double-buffered thread pool includes a scheduling thread group, an execution thread group, a task buffer queue, and a result buffer queue. The scheduling thread group is used to insert standardized tasks into the task buffer queue according to priority. The execution thread group is used to retrieve standardized tasks from the task buffer queue according to the queue order, execute the target operation based on the task type of the standardized task, and generate standardized execution results. The data output module is used to store the standardized execution results into the result buffer queue; the scheduling thread group is also used to read the standardized execution results from the result buffer queue, determine the feedback method based on the hardware status data, and convert the standardized execution results into target output data based on the feedback method.

[0006] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described smart glasses voice interaction method.

[0007] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described smart glasses voice interaction method.

[0008] The beneficial effects of the smart glasses voice interaction method, system, device, and storage medium provided in this application embodiment are as follows: To address the high misjudgment rate of interrupt triggering in existing technologies, this application embodiment determines the interruption intent by coordinating the interrupt trigger signal with user voice command data. This can accurately filter invalid triggers caused by environmental noise and device shaking, allowing smart glasses to respond only to user-initiated interaction needs and improving interaction accuracy.

[0009] When generating standardized tasks, the embodiments of this application integrate hardware status data and schedule tasks according to priority through scheduling thread groups, so that task execution is dynamically matched with device computing power and battery life, avoiding the situation of high-priority tasks lag and low-priority tasks wasting resources, and taking into account both smooth interaction and power consumption control.

[0010] By leveraging the queue separation and thread group division design of the dual-buffered thread pool, this embodiment of the application can achieve orderly connection between task execution, result storage, and feedback conversion, solving the scheduling inefficiency problem caused by coupling of various links in the prior art and improving the interaction response speed. Simultaneously, this embodiment of the application dynamically adjusts the feedback method based on hardware status, adapting the optimal output form to different device states, further reducing power consumption, extending device battery life, and meeting the needs of daily use scenarios for smart glasses. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a voice interaction method for smart glasses provided in an embodiment of this application; Figure 2 This is a structural block diagram of a smart glasses voice interaction system provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0015] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0016] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a voice interaction method for smart glasses provided in an embodiment of this application. The method can be executed by an electronic device, and specifically, the method may include S101 to S103.

[0017] S101: In response to the detection of an interrupt trigger signal and user voice command data, determine the interrupt intention based on the user voice command data and the interrupt trigger signal.

[0018] In this embodiment, determining the interruption intent based on user voice command data and interruption trigger signal specifically includes: parsing the interruption trigger signal to determine the signal type and signal confidence parameter of the interruption trigger signal; the signal type includes wake-up or termination; converting the user voice command data into text data, and determining the interruption intent based on the signal type, signal confidence parameter, and text data.

[0019] In this embodiment, the interrupt trigger signal is parsed to determine the signal type and signal confidence parameter. Specifically, this includes: reading the data packet of the interrupt trigger signal, extracting the checksum and data length identifier from the header of the data packet, and verifying the checksum and data length identifier; if the verification passes, extracting the signal type identifier field and the original confidence field from the data packet; converting the signal type identifier field into a signal type using a preset identifier mapping table; and converting the original confidence field into a confidence parameter.

[0020] In this embodiment, the signal confidence parameter refers to a parameter derived from the original confidence field, used to characterize the reliability of the interrupt trigger signal. Text data refers to the text form of the user's voice command data after conversion, used for intent analysis. Data packet refers to a data carrier encapsulating the signal type identifier field and the original confidence field, constituting the interrupt trigger signal. Checksum refers to the encoding in the data packet header used to verify data integrity. Data length identifier refers to an identifier in the data packet header representing the data capacity, assisting in data verification. Signal type identifier field refers to a specific field in the data packet representing the signal category, used for conversion to signal type. Original confidence field refers to the unprocessed confidence-related data in the data packet. Preset identifier mapping table refers to a table storing the correspondence between the signal type identifier field and the wake-up or termination type, used for type conversion.

[0021] Considering that accurate determination of interruption intent is a core prerequisite for voice interaction in smart glasses, reliability must be ensured through multi-layered verification and multi-dimensional data fusion. Therefore, this embodiment first verifies the data packet of the interruption trigger signal to filter out corrupt or invalid data and prevent erroneous data from affecting intent determination. This embodiment converts the signal type identifier field through a mapping table to ensure the uniformity and accuracy of signal type recognition. The original confidence score field is converted into a standard parameter to quantify signal reliability. This embodiment combines signal type, confidence score parameters, and text data for comprehensive determination, avoiding misjudgments caused by a single data dimension, accurately distinguishing between wake-up and termination intents, ensuring that smart glasses only respond to the user's genuine interaction needs, and reducing invalid operations.

[0022] For example, when the smart glasses are in interactive monitoring mode, if both an interruption trigger signal and user voice command data are detected simultaneously, the interruption intent determination process is initiated. This embodiment first reads the data packet corresponding to the interruption trigger signal, extracts the preset checksum and data length identifier from the packet header, and uses a built-in verification algorithm to verify their consistency and validity, determining whether the data packet is complete and has not been tampered with. If the verification fails, the interruption trigger signal is deemed invalid, the current signal and corresponding user voice command data are discarded, and the interactive monitoring state is returned; if the verification passes, the signal type identifier field and the original confidence level field are accurately extracted from the data packet to ensure that no critical data is omitted.

[0023] This embodiment can invoke a preset identifier mapping table, matching the extracted signal type identifier field one by one with the entries in the mapping table, and converting it into a wake-up or termination signal type based on the matching result, thus completing signal category identification. Simultaneously, this embodiment can process the original confidence field according to preset standardized conversion rules, converting it into a signal confidence parameter of a uniform magnitude for subsequent reliability assessment. This embodiment can initiate a speech-to-text processing flow, converting user voice command data through the speech recognition module built into the smart glasses, removing redundant audio information, and generating clear and accurate text data to ensure that the user's voice content can be effectively parsed.

[0024] If the signal type is wake-up, and the confidence parameter is higher than the preset reliability threshold, and the text data contains key information related to wake-up, then the interruption intention is determined to be wake-up interaction; if the signal type is termination, and the confidence parameter meets the standard and the text data conforms to the termination interaction description, then the interruption intention is determined to be termination interaction; if the confidence parameter is lower than the threshold or the data information is contradictory, then it is determined to be an invalid intention, and the subsequent interaction process will not be started.

[0025] This embodiment effectively filters invalid or corrupted data by performing multi-layered verification on interrupt trigger signals, ensuring the basic reliability of intent determination. Standardized conversion of signal type and confidence parameters ensures the uniformity and accuracy of data processing. Based on a multi-dimensional comprehensive judgment of signal type, confidence parameters, and text data, the probability of misjudgment caused by a single data dimension is significantly reduced, accurately distinguishing between wake-up and termination intents. This design effectively reduces invalid interactions caused by environmental interference or erroneous signals, improving the accuracy and reliability of voice interaction in smart glasses, providing users with a smoother and more accurate interactive experience, and fully meeting the usage scenarios of lightweight and highly interactive smart glasses.

[0026] S102: Obtain hardware status data of smart glasses, generate standardized tasks based on interrupt intent, user voice command data and hardware status data, and pass the standardized tasks into a double-buffered thread pool. The double-buffered thread pool includes a scheduling thread group, an execution thread group, a task buffer queue and a result buffer queue. The scheduling thread group is used to insert standardized tasks into the task buffer queue according to priority; the execution thread group is used to retrieve standardized tasks from the task buffer queue according to the queue order, and execute the target operation based on the task type of the standardized task to generate standardized execution results. In this embodiment, a standardized task is generated based on interruption intent, user voice command data, and hardware status data. This includes: mapping the interruption intent to a task type field based on a standardized task structure template, encapsulating the user voice command data into a task core data content field, and encapsulating the hardware status data into a task auxiliary status field, thereby obtaining a standardized task containing a task type field, a task core data content field, and a task auxiliary status field, and assigning a task identifier to the standardized task.

[0027] In this embodiment, after the scheduling thread group inserts the standardized tasks into the task buffer queue according to priority order, the method further includes: the scheduling thread group adjusts the number of available threads in the execution thread group based on hardware status data, and passes the number of available threads to the execution thread group; the execution thread group initializes worker threads based on the number of available threads, and uses the number of available threads as the maximum concurrent execution limit to allocate worker threads and perform concurrent processing on the standardized tasks obtained from the task buffer queue.

[0028] In this embodiment, hardware status data refers to quantifiable parameters generated during the operation of the smart glasses hardware. These parameters reflect the device's hardware resource usage, energy consumption, and operating status. Examples include computational load parameters such as CPU utilization, memory usage, processor frequency, number of background processes, and task execution queue length; peripheral operating status parameters such as display screen operating status, display screen brightness level, speaker operating status, speaker output power, microphone pickup gain, bone conduction sensor operating current, and posture sensor sampling frequency; and other hardware status parameters such as remaining storage space, storage read / write speed, device operating temperature, Bluetooth module operating status, number of connected Bluetooth devices, Wi-Fi signal strength, and Wi-Fi transmission rate. The standardized task structure template is a preset, unified task format template used to standardize the composition of task fields. The task type field is a field mapped from the interrupt intent, representing the specific type of task. The task core data content field encapsulates user voice command data, providing the core basis for task execution. The task auxiliary status field encapsulates hardware status data, assisting in task execution adaptation. The task identifier is a unique identification information assigned to the standardized task, used for task tracking. The number of available threads is the number of threads that can be called upon by the execution thread group after the scheduling thread group is adjusted. A worker thread is the specific thread unit within the execution thread group that performs the task. The maximum concurrent execution limit refers to the maximum number of tasks that can be executed simultaneously, based on the number of available threads. Concurrent processing refers to the execution method where multiple worker threads handle different tasks simultaneously.

[0029] Considering the limited hardware resources of smart glasses, a balance must be struck between task execution efficiency and power consumption control. Therefore, this embodiment adopts a standardized task structure template to unify task format, ensuring the standardization and efficiency of dual-buffered thread pool scheduling and avoiding scheduling chaos caused by inconsistent formats. This embodiment encapsulates multi-source data into corresponding fields, clearly defining data responsibilities and facilitating rapid retrieval during task execution. This embodiment dynamically adjusts the number of available threads to adapt to the device resource situation reflected by hardware status data, avoiding task lag when hardware resources are insufficient or power waste when resources are idle. This embodiment uses the number of available threads as the concurrency limit to prevent excessive concurrency from causing computational overload, improving task processing efficiency while ensuring device operational stability.

[0030] For example, after determining the interruption intention, the smart glasses use the built-in hardware status monitoring module to obtain hardware status data such as power parameters and computing load parameters in real time. At the same time, they retrieve existing interruption intention and user voice command data and store the three types of data in the device's local data cache to ensure that the data is available in real time.

[0031] This embodiment can call a preset standardized task structure template and fill in the fields according to the template requirements: the interruption intent is directly mapped to the corresponding task type field, clarifying the task execution direction; the user voice command data is completely encapsulated into the task core data content field, ensuring the core data support for task execution; and the hardware status data is fully encapsulated into the task auxiliary status field, providing a basis for task adaptation. After encapsulation, a unique task identifier is assigned to the task according to preset rules, forming a standardized task containing a task type field, a task core data content field, a task auxiliary status field, and a task identifier.

[0032] In this embodiment, the generated standardized tasks can be passed to a double-buffered thread pool. After the scheduling thread group receives the tasks, it inserts the standardized tasks into the task buffer queue in descending order according to the priority level corresponding to the task type field, ensuring that high-priority tasks are executed first.

[0033] The scheduling thread group pre-reads the quantization thresholds configured by the system, with preset high battery thresholds of 70%, low battery thresholds of 30%, high load thresholds of 60%, and low load thresholds of 30%. The maximum number of available threads in the execution thread group is set to 8, with 2 basic backup threads (used to maintain core interactive functions). The scheduling thread group parses the hardware status data, extracting battery parameters (remaining battery percentage) and computing load parameters (CPU utilization) for quantization matching: if the battery parameter is ≥70% and the CPU utilization is ≤30% (i.e., the quantized high battery, low load state), the number of available threads is increased to 6-8 to fully utilize surplus hardware resources; if 30% < battery parameter < 70% and 30% < CPU utilization < 60% (i.e., the quantized medium battery, medium load state), the number of available threads is maintained at 4-5 to balance resource consumption and execution efficiency; if the battery parameter is ≤30% and the CPU utilization is ≥60% (i.e., the quantized low battery, high load state), the number of available threads is reduced to 2-3, retaining only core threads to ensure basic functions.

[0034] After receiving the number of available threads, the execution thread group starts the corresponding number of worker threads and completes worker thread initialization. Using the number of available threads as the maximum concurrent execution limit, standardized tasks are sequentially retrieved from the task buffer queue and assigned one by one to idle worker threads, enabling concurrent processing of multiple tasks. Each worker thread, based on the task type field of the standardized task, calls the corresponding execution module, combines the user voice command data in the task's core data content field and the hardware status data in the task's auxiliary status field, executes the target operation, and finally generates a standardized execution result.

[0035] This embodiment generates tasks in a unified format using a standardized task structure template, ensuring the efficiency and standardization of dual-buffered thread pool scheduling and avoiding scheduling delays caused by format confusion. Multi-source data is encapsulated in separate fields, and a unique task identifier design facilitates data retrieval and tracking during task execution. The scheduling thread group dynamically adjusts the number of available threads based on hardware status, ensuring precise matching between task execution and device resources, effectively solving the problems of stuttering or power consumption spikes caused by fixed-thread scheduling. The processing method, which uses the number of available threads as the concurrency limit, improves task execution efficiency while preventing hardware overload, balancing smooth interaction and device stability, and fully meeting the needs of lightweight and resource-constrained smart glasses usage scenarios.

[0036] S103: Store the standardized execution results into the result buffer queue; the scheduling thread group is also used to read the standardized execution results from the result buffer queue, determine the feedback method based on the hardware status data, and convert the standardized execution results into target output data based on the feedback method.

[0037] In this embodiment, the hardware status data includes power parameters and computing load parameters; The feedback method is determined based on hardware status data, and the standardized execution results are converted into target output data based on the feedback method, including: If the power parameter is not less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback and screen feedback. If the power parameter is not less than the target power threshold and the computing load parameter is greater than the target load threshold, or if the power parameter is less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback. If the power parameter is less than the target power threshold and the computing load parameter is greater than the target load threshold, then the feedback method is determined to be screen feedback and short notification sound feedback. Based on feedback methods, the standardized execution results are transformed to generate target output data; Specifically, voice feedback refers to converting the text content in the standardized execution result into a PCM audio stream, which is then played through the smart glasses' speaker; screen feedback refers to converting the text content in the standardized execution result into character data, which is then displayed on the smart glasses' screen; and short prompt tone feedback refers to calling a prompt tone of fixed duration and single frequency, converting the prompt tone into short audio data, which is then played through the smart glasses' speaker.

[0038] In this embodiment, the target battery threshold refers to a preset critical value for determining whether the battery is sufficient, used to classify hardware states to match the feedback method. The target load threshold refers to a preset critical value for determining whether the computing power load is high or low, assisting in hardware state classification. Target output data refers to data that, after being converted by the feedback method, can be directly output through the smart glasses peripheral. Short audio data refers to audio data that, after being converted by a short prompt tone, can be played through a speaker. Short prompt tone refers to a preset prompt signal of fixed duration and single frequency, used to quickly convey core status information. Character data refers to text data that, after being converted from the standardized execution result, is adapted for display screen display.

[0039] Considering the limited hardware resources of smart glasses, the feedback method must be precisely adapted to the hardware state, balancing interactive experience and power consumption control. The device's capacity varies under different combinations of power and computing loads. When resources are abundant, multimodal feedback can improve the intuitiveness of interaction, while lightweight feedback can reduce power consumption when resources are scarce. Therefore, this embodiment clearly defines the specific data conversion rules and output paths for each feedback method, ensuring the uniformity and executability of feedback signals and avoiding playback abnormalities or power waste caused by mismatch between the feedback method and the hardware state. This embodiment ensures that there is an appropriate feedback method for all scenarios through comprehensive hardware state determination rules, guaranteeing the continuity and stability of interaction.

[0040] For example, after the execution thread group completes the target operation of the standardized task, it generates a standardized execution result containing text content and execution status. Following the preset rules of the double-buffered thread pool, the standardized execution result is completely stored in the result buffer queue, and the queue status flag is updated to inform the scheduling thread group that there are pending results. The scheduling thread group monitors the status flag of the result buffer queue in real time. When a pending result is detected, it initiates a data reading process, retrieving the standardized execution results from the result buffer queue sequentially. Simultaneously, through the hardware monitoring module of the smart glasses, it acquires two types of hardware status data in real time: power parameters and computing load parameters, ensuring that the result data and hardware data are retrieved synchronously.

[0041] The scheduling thread group calls the preset target power threshold and target load threshold to determine the acquired hardware status data: first, it compares the power parameter with the target power threshold, then it compares the computing load parameter with the target load threshold, and matches the corresponding feedback method judgment condition according to the numerical relationship between the two.

[0042] If the determination result is that the power parameter is not less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback and screen feedback; if the determination result is that the power parameter is not less than the target power threshold and the computing load parameter is greater than the target load threshold, or the power parameter is less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback; if the determination result is that the power parameter is less than the target power threshold and the computing load parameter is greater than the target load threshold, then the feedback method is determined to be screen feedback and short prompt tone feedback.

[0043] This embodiment achieves dynamic adaptation between feedback methods and hardware status through comprehensive status determination based on power and computing load parameters, overcoming the shortcomings of existing technologies that rely on a single feedback method. When resources are sufficient, multimodal feedback combining voice and screen is used, allowing users to intuitively obtain interactive information and improving the user experience. When resources are scarce, lightweight feedback is used to effectively reduce device power consumption and extend battery life. Each feedback method has clearly defined data conversion rules and output paths, ensuring stable adaptation of target output data to peripherals and avoiding playback or display anomalies. The overall design balances the flexibility of interaction with the rational use of device resources, meeting user needs in different scenarios while ensuring device stability, fully aligning with the lightweight and highly adaptable characteristics of smart glasses.

[0044] As can be seen from the above, in response to the problem of high misjudgment rate of interrupt triggering in existing technologies, this embodiment determines the interruption intention by coordinating the interrupt trigger signal with the user's voice command data. This can accurately filter out invalid triggers caused by environmental noise and device shaking, allowing smart glasses to respond only to the user's active interaction needs and improving the accuracy of interaction.

[0045] In this embodiment, hardware status data is integrated when generating standardized tasks, and tasks are scheduled according to priority through scheduling thread groups. This allows task execution to be dynamically matched with the device's computing power and battery life, avoiding situations where high-priority tasks lag and low-priority tasks waste resources, thus balancing smooth interaction and power consumption control.

[0046] By leveraging the queue separation and thread group division design of the dual-buffered thread pool, this embodiment achieves orderly connection between task execution, result storage, and feedback conversion, solving the scheduling inefficiency caused by coupling of various links in existing technologies and improving interaction response speed. Simultaneously, this embodiment dynamically adjusts the feedback method based on hardware status, adapting to the optimal output form under different device states, further reducing power consumption, extending device battery life, and meeting the needs of daily use scenarios for smart glasses.

[0047] In one embodiment of this disclosure, the interrupt trigger signal is generated in the following manner: The system acquires raw posture data, raw user vocal cord vibration data, and raw environmental mixed speech data. The raw posture data includes acceleration and angular velocity data obtained through the posture sensor built into the smart glasses frame. The raw user vocal cord vibration data is obtained through the bone conduction sensor of the smart glasses. The raw environmental mixed speech data is obtained through the microphone of the smart glasses. The original posture data is low-pass filtered, and the filtered posture data is matched with the target interaction posture threshold to obtain the posture matching degree; the target interaction posture threshold is the range of historical action parameters triggered by the user's voice interaction. If the posture matching degree is not less than the posture matching degree threshold, the user's original vocal cord vibration data and the environmental mixed speech data are synchronized and aligned on the time axis. The short-time correlation coefficient of the aligned user's original vocal cord vibration data and the environmental mixed speech data is calculated. If the short-time correlation coefficient is not less than the correlation coefficient threshold, the aligned user's original vocal cord vibration data and the environmental mixed speech data are weighted and fused to generate a clean speech signal. Mel frequency cepstral coefficients (MFCC) feature vectors are extracted from the pure speech signal. The similarity between the MFCC feature vectors and the keyword feature vectors in the keyword feature template library is calculated and the maximum similarity is selected. If the maximum similarity is not less than the similarity threshold, the signal type corresponding to the keyword feature vector with the maximum similarity is determined. The confidence parameter is obtained by weighting the pose matching degree, short-time correlation coefficient, and maximum similarity. The signal type and confidence parameters are encapsulated into a data packet and used as an interrupt trigger signal.

[0048] In this embodiment, low-pass filtering refers to the noise filtering operation on the original posture data to remove high-frequency interference signals. The target interaction posture threshold refers to the range of historical action parameters triggered by the user's voice interaction, used to determine the validity of the posture. The posture matching degree threshold refers to the critical value for determining whether the posture matching is qualified. The short-time correlation coefficient refers to a parameter characterizing the homology of two types of speech data after alignment. The correlation coefficient threshold refers to the critical value for determining whether the speech data are homologous. Weighted fusion refers to the operation of assigning weights to homologous speech data and synthesizing them. The clean speech signal refers to the user's speech signal after denoising processing. The MFCC feature vector refers to the Mel-frequency cepstral coefficient feature vector extracted from the clean speech signal, used for speech feature matching. The keyword feature template library refers to a database storing preset keyword standard feature vectors. The similarity threshold refers to the critical value for determining whether the feature matching is qualified. The confidence parameter is a quantitative parameter calculated based on three core matching indicators: posture matching degree, short-time correlation coefficient, and maximum similarity, weighted according to a preset weight ratio. The value ranges from 0 to 1 and is used to accurately characterize the reliability of the interruption trigger signal. The higher the value, the closer the signal is to the user's real interaction intention, providing a key basis for the accurate determination of the subsequent interruption intention.

[0049] Considering that the accuracy of the interruption trigger signal directly affects the reliability of voice interaction, interference needs to be eliminated through multi-source data fusion and multi-layer verification. Therefore, this embodiment collects multi-dimensional raw data, which can verify the user's interaction intent from both posture and voice aspects, avoiding misjudgment based on a single data dimension. Low-pass filtering improves the accuracy of posture data, and posture matching can filter out non-active interaction scenarios. Time axis synchronization and correlation coefficient verification ensure that the voice data comes from the user themselves, eliminating environmental noise interference. Feature matching and confidence quantification further verify the effectiveness of keywords and signal reliability. Through multi-layer verification and parameter encapsulation design, this embodiment ultimately generates an accurate and reliable interruption trigger signal, laying the foundation for subsequent intent determination.

[0050] For example, when the smart glasses are in standby monitoring mode, they initiate multi-source sensor data acquisition. The attitude sensor collects acceleration and angular velocity data in real time to form raw attitude data; the bone conduction sensor captures the user's vocal cord vibration to generate raw vocal cord vibration data; the microphone collects ambient audio to obtain raw ambient mixed speech data. All three types of data are simultaneously stored in the device's temporary cache.

[0051] This embodiment can use a low-pass filtering algorithm to process the original posture data, filtering out high-frequency noise caused by slight device shaking, and obtaining smoothed posture data. Then, the target interaction posture threshold is retrieved. This threshold is based on statistics of a large number of users' historical interaction action parameters. The filtered posture data is then compared with the target interaction posture threshold to calculate the posture matching degree.

[0052] If the posture matching degree is less than the posture matching degree threshold, it is determined that there is no intention of user active interaction, the data collected this time is discarded, and the system returns to standby state; if the posture matching degree is not less than the threshold, the user's original vocal cord vibration data and the original environmental mixed speech data are synchronized and aligned on the time axis based on the current timestamp to ensure that the two types of data correspond accurately in the time dimension.

[0053] This embodiment calculates the short-term correlation coefficient between the two types of data after alignment. If the coefficient is less than the correlation coefficient threshold, it is determined to be environmental noise or other people's speech, the process is terminated, and the data is cleared. If the coefficient is not less than the threshold, the two types of data are weighted and fused according to a preset weight ratio to synthesize a clean speech signal with environmental noise removed. This embodiment can extract MFCC feature vectors from the clean speech signal, which can accurately represent key speech features. A keyword feature template library is retrieved, and the extracted MFCC feature vector is compared with the feature vectors of each keyword in the template library one by one. The maximum similarity value is recorded and selected.

[0054] If the maximum similarity is less than the similarity threshold, it is determined that there is no valid keyword match, and the process terminates. If the maximum similarity is not less than the threshold, the signal type to which the keyword feature vector corresponding to the maximum similarity belongs is determined. Simultaneously, the pose matching degree, short-time correlation coefficient, and maximum similarity are weighted according to preset weights to obtain a confidence parameter characterizing the signal reliability. In this embodiment, the determined signal type and confidence parameter can be encapsulated into a standardized data packet, which serves as the interruption trigger signal. This packet is stored in a dedicated signal buffer, awaiting collaboration with user voice command data to participate in the interruption intent determination.

[0055] This embodiment collects posture and voice data from multiple sensors and generates an interruption trigger signal using a multi-layered verification mechanism. This effectively filters out interference factors such as environmental noise and device vibration, significantly reducing the probability of invalid triggers. Low-pass filtering and posture matching screen for non-active interaction scenarios, time synchronization and correlation coefficient verification ensure the homogeneity of voice data, feature matching accurately identifies keywords, and confidence parameters quantify signal reliability. The multi-layered verification design ensures the accuracy and reliability of the interruption trigger signal, providing a solid foundation for subsequent interruption intent determination, reducing invalid interactions, improving the accuracy of voice interaction and user experience of smart glasses, and meeting the usage scenarios of lightweight and highly interactive smart glasses.

[0056] Corresponding to the smart glasses voice interaction method in the above embodiment, Figure 2 This is a structural block diagram of a smart glasses voice interaction system provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The smart glasses voice interaction system 20 includes: an interruption detection module 21, a task execution module 22, and a data output module 23.

[0057] The interruption detection module 21 is used to determine the interruption intention based on the user voice command data and the interruption trigger signal in response to the detection of the interruption trigger signal and user voice command data. The task execution module 22 is used to acquire hardware status data of the smart glasses, generate standardized tasks based on interrupt intent, user voice command data, and hardware status data, and pass the standardized tasks into a double-buffered thread pool. The double-buffered thread pool includes a scheduling thread group, an execution thread group, a task buffer queue, and a result buffer queue. The scheduling thread group is used to insert standardized tasks into the task buffer queue according to priority. The execution thread group is used to retrieve standardized tasks from the task buffer queue according to the queue order, execute the target operation based on the task type of the standardized task, and generate standardized execution results. The data output module 23 is used to store the standardized execution results into the result buffer queue; the scheduling thread group is also used to read the standardized execution results from the result buffer queue, determine the feedback method based on the hardware status data, and convert the standardized execution results into target output data based on the feedback method.

[0058] In one embodiment of this application, the smart glasses voice interaction system 20 further includes: an interrupt signal generation module, used for: The system acquires raw posture data, raw user vocal cord vibration data, and raw environmental mixed speech data. The raw posture data includes acceleration and angular velocity data obtained through the posture sensor built into the smart glasses frame. The raw user vocal cord vibration data is obtained through the bone conduction sensor of the smart glasses. The raw environmental mixed speech data is obtained through the microphone of the smart glasses. The original posture data is low-pass filtered, and the filtered posture data is matched with the target interaction posture threshold to obtain the posture matching degree; the target interaction posture threshold is the range of historical action parameters triggered by the user's voice interaction. If the posture matching degree is not less than the posture matching degree threshold, the user's original vocal cord vibration data and the environmental mixed speech data are synchronized and aligned on the time axis. The short-time correlation coefficient of the aligned user's original vocal cord vibration data and the environmental mixed speech data is calculated. If the short-time correlation coefficient is not less than the correlation coefficient threshold, the aligned user's original vocal cord vibration data and the environmental mixed speech data are weighted and fused to generate a clean speech signal. MFCC feature vectors are extracted from the clean speech signal. The similarity between the MFCC feature vectors and the keyword feature vectors in the keyword feature template library is calculated and the maximum similarity is selected. If the maximum similarity is not less than the similarity threshold, the signal type corresponding to the keyword feature vector with the maximum similarity is determined. The confidence parameter is obtained by weighting the pose matching degree, short-time correlation coefficient, and maximum similarity. The signal type and confidence parameters are encapsulated into a data packet and used as an interrupt trigger signal.

[0059] In one embodiment of this application, when determining the interruption intent based on user voice command data and an interruption trigger signal, the interruption detection module 21 is specifically used for: The interrupt trigger signal is parsed to determine the signal type and signal confidence parameters; the signal type includes wake-up or termination. The user's voice command data is converted into text data, and the interruption intention is determined based on the signal type, signal confidence parameter, and text data.

[0060] In one embodiment of this application, when the interrupt detection module 21 parses the interrupt trigger signal and determines the signal type and signal confidence parameter of the interrupt trigger signal, it is specifically used for: Retrieve the data packet triggered by the interrupt signal, extract the checksum and data length identifier from the header of the data packet, and verify the checksum and data length identifier. If the verification passes, extract the signal type identifier field and the original confidence field from the data packet; The signal type identifier field is converted into a signal type using a preset identifier mapping table; Convert the original confidence field into a confidence parameter.

[0061] In one embodiment of this application, when the task execution module 22 generates a standardized task based on interruption intent, user voice command data, and hardware status data, it is specifically used to: map the interruption intent to a task type field based on the standardized task structure template, encapsulate the user voice command data into a task core data content field, encapsulate the hardware status data into a task auxiliary status field, and obtain a standardized task containing a task type field, a task core data content field, and a task auxiliary status field, and assign a task identifier to the standardized task.

[0062] In one embodiment of this application, after the scheduling thread group inserts standardized tasks into the task buffer queue according to priority order, the task execution module 22 is further configured to adjust the number of available threads in the execution thread group based on hardware status data and pass the number of available threads to the execution thread group; the execution thread group initializes worker threads based on the number of available threads, and uses the number of available threads as the maximum concurrent execution limit to allocate worker threads and perform concurrent processing on the standardized tasks obtained from the task buffer queue.

[0063] In one embodiment of this application, the hardware status data includes power parameters and computing load parameters; the data output module 23, when determining the feedback method based on the hardware status data and converting the standardized execution result into target output data based on the feedback method, is used for: If the power parameter is not less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback and screen feedback. If the power parameter is not less than the target power threshold and the computing load parameter is greater than the target load threshold, or if the power parameter is less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback. If the power parameter is less than the target power threshold and the computing load parameter is greater than the target load threshold, then the feedback method is determined to be screen feedback and short notification sound feedback. Based on feedback methods, the standardized execution results are transformed to generate target output data; Specifically, voice feedback refers to converting the text content in the standardized execution result into a PCM audio stream, which is then played through the smart glasses' speaker; screen feedback refers to converting the text content in the standardized execution result into character data, which is then displayed on the smart glasses' screen; and short prompt tone feedback refers to calling a prompt tone of fixed duration and single frequency, converting the prompt tone into short audio data, which is then played through the smart glasses' speaker.

[0064] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned system embodiments, for example... Figure 2 The functions of the interrupt detection module 21, task execution module 22, and data output module 23 are shown.

[0065] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0066] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0067] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0068] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in the embodiments of the smart glasses voice interaction method provided in the embodiments of this application, or they can execute the implementation methods of the electronic device 300 described in the embodiments of this application, which will not be repeated here.

[0069] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0070] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0071] Those skilled in the art will recognize that the modules / units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules / units, or it may be an electrical, mechanical, or other form of connection.

[0074] The modules / units described as separate components may or may not be physically separate. Similarly, the components shown as modules / units may or may not be physical modules / units; they may be located in one place or distributed across multiple network modules / units. Some or all of the modules / units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0075] Furthermore, the functional modules / units in the various embodiments of this application can be integrated into one processing module / unit, or each module / unit can exist physically separately, or two or more modules / units can be integrated into one module / unit. The integrated modules / units described above can be implemented in hardware or in the form of software functional modules / units.

[0076] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice interaction method for smart glasses, characterized in that, include: In response to the detection of an interruption trigger signal and user voice command data, an interruption intent is determined based on the user voice command data and the interruption trigger signal; The system acquires hardware status data of the smart glasses, generates standardized tasks based on the interruption intent, user voice command data, and hardware status data, and passes the standardized tasks to a double-buffered thread pool. The double-buffered thread pool includes a scheduling thread group, an execution thread group, a task buffer queue, and a result buffer queue. The scheduling thread group inserts the standardized tasks into the task buffer queue according to priority. The execution thread group retrieves the standardized tasks from the task buffer queue according to the queue order, performs target operations based on the task type of the standardized tasks, and generates standardized execution results. Store the standardized execution results into a result buffer queue; The scheduling thread group is also used to read the standardized execution result from the result buffer queue, determine the feedback method based on the hardware status data, and convert the standardized execution result into target output data based on the feedback method; The interrupt trigger signal is generated in the following manner: The system acquires raw posture data, raw user vocal cord vibration data, and raw environmental mixed speech data. The raw posture data includes acceleration and angular velocity data obtained through the posture sensor built into the smart glasses frame. The raw user vocal cord vibration data is obtained through the bone conduction sensor of the smart glasses. The raw environmental mixed speech data is obtained through the microphone of the smart glasses. The original posture data is low-pass filtered, and the filtered posture data is matched with the target interaction posture threshold to obtain the posture matching degree; the target interaction posture threshold is the range of historical action parameters triggered by the user's voice interaction. If the posture matching degree is not less than the posture matching degree threshold, then the user's original vocal cord vibration data and the environmental mixed speech data are synchronized and aligned on the time axis. Calculate the short-time correlation coefficient between the aligned user vocal cord vibration raw data and the environmental mixed speech raw data. If the short-time correlation coefficient is not less than the correlation coefficient threshold, then perform weighted fusion on the aligned user vocal cord vibration raw data and the environmental mixed speech raw data to generate a clean speech signal. MFCC feature vectors are extracted from the clean speech signal. The similarity between the MFCC feature vectors and the keyword feature vectors in the keyword feature template library is calculated and the maximum similarity is selected. If the maximum similarity is not less than the similarity threshold, the signal type corresponding to the keyword feature vector corresponding to the maximum similarity is determined. The confidence parameter is obtained by weighting the pose matching degree, the short-time correlation coefficient, and the maximum similarity. The signal type and the confidence parameter are encapsulated into a data packet and used as an interrupt trigger signal.

2. The smart glasses voice interaction method as described in claim 1, characterized in that, The step of determining the interruption intent based on the user voice command data and the interruption trigger signal includes: The interrupt trigger signal is parsed to determine the signal type and signal confidence parameter of the interrupt trigger signal; the signal type includes wake-up or termination; The user's voice command data is converted into text data, and the interruption intention is determined based on the signal type, the signal confidence parameter, and the text data.

3. The smart glasses voice interaction method as described in claim 2, characterized in that, The step of parsing the interrupt trigger signal to determine the signal type and signal confidence parameters of the interrupt trigger signal includes: Read the data packet of the interrupt trigger signal, extract the checksum and data length identifier in the header of the data packet, and verify the checksum and data length identifier; If the verification passes, extract the signal type identifier field and the original confidence field from the data packet; The signal type identifier field is converted into a signal type using a preset identifier mapping table; Convert the original confidence field into a confidence parameter.

4. The smart glasses voice interaction method as described in claim 1, characterized in that, The process of generating a standardized task based on the interruption intent, the user voice command data, and the hardware status data includes: Based on a standardized task structure template, the interruption intent is mapped to a task type field, the user voice command data is encapsulated into a task core data content field, and the hardware status data is encapsulated into a task auxiliary status field, resulting in a standardized task containing a task type field, a task core data content field, and a task auxiliary status field. A task identifier is then assigned to the standardized task.

5. The smart glasses voice interaction method as described in claim 1, characterized in that, After the scheduling thread group inserts the standardized tasks into the task buffer queue according to priority, the process further includes: The scheduling thread group adjusts the number of available threads in the execution thread group based on the hardware status data, and passes the number of available threads to the execution thread group. The execution thread group initializes worker threads based on the number of available threads, and uses the number of available threads as the maximum concurrent execution limit to allocate worker threads and perform concurrent processing on standardized tasks obtained from the task buffer queue.

6. The smart glasses voice interaction method as described in claim 1, characterized in that, The hardware status data includes power parameters and computing load parameters; The step of determining the feedback method based on the hardware status data, and converting the standardized execution result into target output data based on the feedback method, includes: If the power parameter is not less than the target power threshold and the computing load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback and screen feedback. If the power parameter is not less than the target power threshold and the computing power load parameter is greater than the target load threshold, or if the power parameter is less than the target power threshold and the computing power load parameter is not greater than the target load threshold, then the feedback method is determined to be voice feedback. If the power parameter is less than the target power threshold and the computing load parameter is greater than the target load threshold, then the feedback method is determined to be screen feedback and short notification sound feedback. Based on the feedback method, the standardized execution results are transformed to generate target output data; Specifically, voice feedback refers to converting the text content in the standardized execution result into a pulse code modulation audio stream, which is then played through the smart glasses' speaker; screen feedback refers to converting the text content in the standardized execution result into character data, which is then displayed on the smart glasses' screen; and short prompt tone feedback refers to calling a prompt tone of fixed duration and single frequency, converting the prompt tone into short audio data, which is then played through the smart glasses' speaker.

7. A smart glasses voice interaction system using the smart glasses voice interaction method according to any one of claims 1-6, characterized in that, include: An interruption detection module is used to determine the interruption intention based on the user voice command data and the interruption trigger signal in response to the detection of an interruption trigger signal and user voice command data. The task execution module is used to acquire hardware status data of the smart glasses, generate standardized tasks based on the interruption intent, the user voice command data, and the hardware status data, and pass the standardized tasks into a double-buffered thread pool. The double-buffered thread pool includes a scheduling thread group, an execution thread group, a task buffer queue, and a result buffer queue. The scheduling thread group is used to insert the standardized tasks into the task buffer queue according to priority. The execution thread group is used to retrieve the standardized tasks from the task buffer queue according to the queue order, execute the target operation based on the task type of the standardized task, and generate standardized execution results. The data output module is used to store the standardized execution results into a result buffer queue; The scheduling thread group is also used to read the standardized execution result from the result buffer queue, determine the feedback method based on the hardware status data, and convert the standardized execution result into target output data based on the feedback method.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images