Voice command-based processing method, device, and system

Abstract

The embodiments of the present disclosure disclose a processing method, device and system based on voice instructions. A specific implementation of the method includes: recognizing voice data, and synchronously sending the above-mentioned voice data to the terminal and the cloud; in response to receiving the user intention information sent by the above-mentioned terminal or the above-mentioned cloud or identifying the user intention information, executing the preset operation corresponding to the above-mentioned user intention information, wherein the user intention information sent by the above-mentioned terminal is obtained by the above-mentioned terminal recognizing the received voice data, and the user intention information sent by the above-mentioned cloud is obtained by the above-mentioned cloud recognizing the received voice data; in response to recognizing the user intention information, sending a cancellation signal to the above-mentioned terminal and the above-mentioned cloud, wherein the above-mentioned cancellation signal is used to interrupt the task corresponding to the above-mentioned voice data in the above-mentioned terminal and the above-mentioned cloud. This implementation method achieves higher-quality voice instruction processing through multi-terminal recognition and collaboration, thereby improving user experience and system practicality.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method, device, and system for processing voice commands. Background Art

[0002] With the increasing popularity of smart interactive devices (such as AR glasses and smart headphones), users' demand for real-time voice interaction is growing. Currently, voice command processing on smart devices mainly adopts single-end and dual-end architectures. The aforementioned single-end architecture generally refers to a solution that relies entirely on the cloud for intent recognition, where voice commands must be uploaded to the cloud server for processing before the results are returned. The aforementioned dual-end architecture generally adds a lightweight intent matching module on the device, where unmatched commands still need to be uploaded to the cloud for processing.

[0003] However, when using the above method to process voice commands, the following technical problems often arise: when processing voice commands through the above single-end architecture, there may be problems such as complete dependence on the network, high latency (usually ≥2 seconds), and functional failure in weak or no network environments; when processing voice commands through the above dual-end architecture, there may be problems such as limited device-side command coverage, complex intentions still requiring cloud intervention, resulting in insignificant latency reduction, and frequent cloud calls bringing high computing and communication costs.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the Invention

[0005] The content of this disclosure is used to briefly introduce concepts that will be described in detail in the detailed description section below. The content of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] Some embodiments of the present disclosure propose a processing method based on voice commands, a head-mounted display device, and a command processing system based on a multi-terminal architecture to explain one or more of the technical problems mentioned in the above background technology section.

[0007] In the first aspect, some embodiments of the present disclosure propose a processing method based on voice instructions, which includes: recognizing voice data, and synchronously sending the above-mentioned voice data to the terminal and the cloud; in response to receiving user intention information sent by the above-mentioned terminal or the above-mentioned cloud or identifying user intention information, executing a preset operation corresponding to the above-mentioned user intention information, wherein the user intention information sent by the above-mentioned terminal is obtained by the above-mentioned terminal recognizing the received voice data, and the user intention information sent by the above-mentioned cloud is obtained by the above-mentioned cloud recognizing the received voice data; in response to recognizing the user intention information, sending a cancellation signal to the above-mentioned terminal and the above-mentioned cloud, wherein the above-mentioned cancellation signal is used to interrupt the task corresponding to the above-mentioned voice data in the above-mentioned terminal and the above-mentioned cloud.

[0008] Optionally, the above-mentioned recognition of voice data includes: performing preliminary recognition on the above-mentioned voice data to obtain a preliminary recognition result; in response to the above-mentioned preliminary recognition result satisfying a preset preliminary recognition condition, generating user intention information and a cancellation signal; in response to the above-mentioned preliminary recognition result not satisfying the preset preliminary recognition condition, performing secondary recognition on the above-mentioned voice data to obtain a secondary recognition result; in response to the above-mentioned secondary recognition result satisfying the preset secondary recognition condition, generating user intention information and a cancellation signal.

[0009] Optionally, the user intention information sent by the above-mentioned terminal is generated through the following steps: classifying and identifying the above-mentioned received voice data according to the terminal intention recognition model to obtain a classification recognition result; in response to the above-mentioned classification recognition result satisfying a preset classification recognition condition, generating an intention type corresponding to the above-mentioned received voice data; performing parameter extraction on the voice data received by the above-mentioned terminal according to the above-mentioned intention type to obtain parameter data; generating user intention information and a cancellation signal according to the above-mentioned intention type and the above-mentioned parameter data, wherein the above-mentioned cancellation signal is used to interrupt the task corresponding to the above-mentioned voice data in the above-mentioned cloud.

[0010] Optionally, the user intention information sent by the cloud is generated by the following steps: generating at least one initial user intention information based on the received voice data; generating user intention information based on the at least one initial user intention information.

[0011] Optionally, the above-mentioned generating at least one initial user intent information based on the above-mentioned received voice data includes: performing intent extraction on the above-mentioned voice data to obtain an intent keyword list; performing complexity extraction on the above-mentioned intent keyword list according to pre-set complexity information to obtain a complex keyword set; generating the complexity level corresponding to the above-mentioned voice data according to the above-mentioned complex keyword set and the above-mentioned pre-set complexity information; generating a model identifier based on the matching information of the pre-set large language model and the complexity level corresponding to the above-mentioned voice data; inputting the above-mentioned voice data into the target large language model corresponding to the above-mentioned model identifier to obtain at least one initial user intent information.

[0012] Optionally, the above-mentioned generating user intent information based on the at least one initial user intent information includes: performing semantic conversion on the at least one initial user intent information to obtain at least one converted initial user intent information; generating a candidate atomic operation list based on the at least one converted initial user intent information and the glasses-side exclusive capability map; adding spatiotemporal constraint marks to the above-mentioned candidate atomic operation list to obtain an operation list with spatiotemporal dependency marks; testing the above-mentioned operation list with spatiotemporal dependency marks to obtain a test result; in response to the above-mentioned test result indicating that the test has failed, generating feedback information, and visualizing the above-mentioned feedback information; in response to the above-mentioned test result indicating that the test has passed, generating user intent information based on the above-mentioned operation list with spatiotemporal dependency marks.

[0013] Optionally, the above-mentioned operation list with time-space dependency marks is checked to obtain a test result, including: performing a correctness test on the above-mentioned operation list with time-space dependency marks; in response to the above-mentioned operation list with time-space dependency marks passing the above-mentioned correctness test, obtaining hardware performance status information; performing a feasibility test on the above-mentioned operation list with time-space dependency marks based on the above-mentioned hardware performance status information; in response to the above-mentioned operation list with time-space dependency marks passing the above-mentioned feasibility test, performing a conflict test on the above-mentioned operation list with time-space dependency marks based on the above-mentioned hardware performance status information; in response to the above-mentioned operation list with time-space dependency marks passing the above-mentioned conflict test, performing a parameter check on the above-mentioned operation list with time-space dependency marks; in response to the above-mentioned operation list with time-space dependency marks passing the above-mentioned parameter check, determining the preset information representing the passing of the test as the test result.

[0014] Optionally, the above method also includes: sending the acquired multimodal data information corresponding to the above voice data to the above cloud, wherein the above multimodal data information is used by the above cloud to generate user intention information based on the above at least one initial user intention information.

[0015] In a second aspect, some embodiments of the present disclosure propose a head-mounted display device, comprising: at least one display screen for imaging in front of the user's eyes; one or more processors; and a storage device on which one or more programs are stored. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0016] On the third aspect, some embodiments of the present disclosure propose an instruction processing system based on a multi-terminal architecture, which includes: an eyeglass end, configured to recognize voice data, and synchronously send the above-mentioned voice data to the terminal and the cloud; in response to receiving or recognizing user intention information, executing a preset operation corresponding to the above-mentioned user intention information; in response to recognizing the user intention information, sending a cancellation signal to the above-mentioned terminal and the above-mentioned cloud; the terminal, configured to recognize the received voice data; in response to recognizing the user intention information, sending the recognized user intention information to the above-mentioned eyeglass end; the cloud, configured to generate at least one initial user intention information based on the received voice data; generate user intention information based on the above-mentioned at least one initial user intention information, and send the generated user intention information to the above-mentioned eyeglass end.

[0017] The various embodiments of the present disclosure described above have the following beneficial effects: The voice command processing methods of some embodiments of the present disclosure enable highly efficient, low-latency, and robust voice interaction processing, significantly improving user experience and system practicality. Specifically, traditional voice interaction architectures, such as single-end and dual-end architectures, can lead to functional failure in weak or no network environments, high latency in processing complex intents (typically ≥ 2 seconds), and high computational and communication costs associated with frequent cloud calls. To address this, the voice command processing methods of some embodiments of the present disclosure first recognize voice data on the eyewear side and simultaneously send it to the terminal and the cloud, ensuring multi-level data processing and rapid response. This provides a foundation for multi-end collaboration for efficient processing of complex intents, ensuring data diversity and timely processing. Then, in response to receiving or recognizing user intent information sent by the terminal or the cloud, the eyewear side executes the preset action corresponding to the user intent information. This combination of localized processing and recognition significantly reduces latency and improves system responsiveness. At the same time, after receiving voice data from the glasses, the terminal can also identify the voice data and generate user intent information, further optimizing task allocation and resource utilization. This lightweight processing on the terminal ensures efficient recognition and rapid feedback for even complex intents. Furthermore, the cloud can also analyze the voice data and generate user intent information after receiving it from the glasses. This deep cloud processing significantly improves the processing accuracy and system robustness of even more complex intents. Furthermore, in response to identifying user intent, the glasses send a cancel signal to the terminal and the cloud, interrupting the task corresponding to the voice data in both terminals and the cloud. This avoids unnecessary waste of computing and communication resources, improves system efficiency, and reduces energy consumption. Consequently, through multi-terminal collaboration and dynamic task allocation, efficient processing and low-latency response for voice interactions are achieved, generating highly accurate and reliable interaction results that can effectively handle complex voice interaction scenarios. This provides a more reliable and practical solution for fields such as augmented reality and smart devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0019] Figure 1 is a flow chart of some embodiments of a voice instruction-based processing method according to the present disclosure;

[0020] Figure 2is a schematic structural diagram of a head-mounted display device suitable for implementing some embodiments of the present disclosure;

[0021] Figure 3 It is a structural diagram of some embodiments of the instruction processing system based on the multi-terminal architecture according to the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Figure 1 The process 100 of some embodiments of the voice instruction-based processing method according to the present disclosure is shown. The voice instruction-based processing method includes:

[0029] Step 101: Recognize the voice data and send the voice data to the terminal and the cloud synchronously.

[0030] In some embodiments, the execution entity (e.g., glasses) of the voice command-based processing method can recognize voice data and simultaneously transmit the voice data to a terminal and the cloud. The execution entity can be a head-mounted display device. The voice data can be text information generated by the glasses processing a user's voice signal. The voice signal is typically a time-domain waveform signal captured by a microphone. The text information is typically text-based (e.g., "Turn on Bluetooth" or "Recognize the image in front of you," etc.). The glasses are head-mounted display devices that integrate components such as a microdisplay, sensors, and a processor. They can provide voice interaction, visual perception, and information display in various environments (e.g., AR, MR, and VR). The glasses can be binocular diffractive waveguide AR glasses. The terminal can be a device directly used by the user, typically a smartphone, tablet, or personal computer (PC). The terminal can quickly handle simple tasks, ensuring multi-level data processing and rapid response. The cloud can be a combination of remote servers and data centers, responsible for storing and processing large amounts of data. The cloud can handle complex tasks and various files, providing high-precision intent recognition and global consistency. The glasses and the terminal support both offline and online operation. The cloud operates only when online. The glasses are suitable for high-frequency, clear CMD (Command)-type voice commands. The terminal can process simple, high-frequency user voice commands and quickly complete intent classification and command processing without relying on a network. The cloud can utilize a large cloud-based language model for processing, providing more efficient and comprehensive voice command intent classification and task execution capabilities. In practice, the glasses can recognize the voice data generated after processing the user's voice signal. When the glasses receive any voice data, they can recognize it. Furthermore, when offline, the voice data can be transmitted to the terminal via Bluetooth or Wi-Fi Direct. When online, the voice data can be transmitted to the cloud via a cloud server or message queue telemetry. When the voice data is successfully recognized, user intent information can be obtained. Furthermore, the terminal and the cloud can also transmit data to the glasses using the same methods. The terminal can also receive and transmit data with the cloud via a cloud server or message queue telemetry when a network connection is available. The user intent information can be one or more command messages (such as "turn on Bluetooth" or "activate camera, take a photo, send the photo to the cloud, display the recognized information from the cloud") derived from the recognition or processing of the voice data by the glasses, the terminal, or the cloud. The command information can be text messages corresponding to the functions of the glasses.For example, the above-mentioned glasses terminal can use the decision tree classification method to identify the voice data. First, a hierarchical judgment rule is constructed based on relatively common predefined instruction words, and feature branches are set according to priorities starting from the root node. For example, in the first layer, it is judged whether there are brightness-related words such as "bright" or "dark". If it matches, it is classified as a "brightness adjustment" instruction; if it does not match, it enters the second layer to judge whether there are device operation words such as "on" or "off". If it matches, it is classified as "device control"; then the third layer can be set to detect path-related words such as "navigation" or "go". For special emergency instructions such as "shutdown" or "SOS", independent branches need to be set for priority recognition. When the match is successful, the instruction that matches in the decision tree is output as the user intention information. When the voice data traverses the entire decision tree and still does not match, it is determined as an unknown instruction and stored in the analysis queue, that is, the user intention information is not recognized. The above-mentioned glasses terminal is suitable for processing composite instructions with clear structures such as "increase brightness, turn off the light, navigate home".

[0031] In some optional implementation manners of some embodiments, the above-mentioned execution subject can identify the voice data through the following steps: <*

[0032] Step 1, perform a preliminary identification on the above-mentioned voice data to obtain a preliminary identification result. Among them, the above-mentioned preliminary identification can be to parse and filter the above-mentioned voice data based on predefined rules. In practice, the above-mentioned glasses terminal can use a KV matching engine to match the above-mentioned predefined rules (such as various regular expressions, for example, "Leqi" can be set as a user intention information corresponding to performing a certain task) with the above-mentioned voice data (generally in the CMD format). When the above-mentioned voice data can match any regular expression in the above-mentioned predefined rules successfully, the preliminary identification result indicates success. When the above-mentioned voice data fails to match any rule in the above-mentioned predefined rules, the preliminary identification result indicates failure.

[0033] Step 2: In response to the preliminary recognition result satisfying a preset preliminary recognition condition, user intent information and a cancellation signal are generated. The preset preliminary recognition condition is that the preliminary recognition result indicates success. The cancellation signal can be a control signal used to notify other terminals to interrupt related tasks currently processing the same voice data. In practice, when the preliminary recognition result satisfies the preset preliminary recognition condition, the glasses can use the regular expression corresponding to the predefined rules in the voice data as user intent information and generate a cancellation signal. For example, the cancellation signal can be encapsulated using a structured protocol and can include a "0x55AA" frame header identifier, a CRC32 checksum of the user intent information (e.g., 0x8A3F5C2D), the glasses device number (glasses identifier, e.g., 0x01), a high-precision timestamp (e.g., 1718001234567890), and a checksum tail code "0xCC." The cancellation signal can be sent by the glasses to the terminal (terminal identifier, e.g., 0x02) and the cloud (cloud identifier, e.g., 0x03) via Bluetooth Low Energy or a cloud server.

[0034] Step three: In response to the above-mentioned preliminary recognition result not meeting the preset preliminary recognition conditions, the above-mentioned voice data is subjected to secondary recognition to obtain a secondary recognition result. The above-mentioned secondary recognition can be performed by matching the preset NLU rules with the above-mentioned voice data. The above-mentioned preset NLU rules are generally various association relationships that associate voice data with user intent information (such as adjusting brightness and associating voice data such as "brighter brightness", "brighten the screen", and "make the screen brighter"; switching devices and associating voice data such as "turn on the lights", "turn off the lights", "turn on the air conditioner", and "turn off the TV"). In practice, when the above-mentioned preliminary recognition result does not meet the preset preliminary recognition conditions, the above-mentioned execution entity can directly match the above-mentioned voice data with the rules set in the above-mentioned preset NLU rules. When the above-mentioned voice data meets any association relationship between user intent information and voice data in the above-mentioned preset NLU rules, the secondary recognition result is represented as successful; when the above-mentioned voice data fails to meet any association relationship between user intent information and voice data in the above-mentioned preset NLU rules, the secondary recognition result is represented as failed.

[0035] Step 4: In response to the secondary recognition result satisfying a preset secondary recognition condition, user intent information and a cancellation signal are generated. The preset secondary recognition condition is that the secondary recognition result indicates success. In practice, the glasses may generate user intent information and a cancellation signal corresponding to the association between the voice data and the preset NLU rule when the secondary recognition result satisfies the preset secondary recognition condition.

[0036] Optionally, the above execution entity may further perform the following steps:

[0037] The first step is to activate a high-precision microphone array in response to identifying a speech segment that matches the user's voiceprint characteristics, and obtain a speech signal. The user's voiceprint characteristics may be pre-set acoustic features that can identify the user (such as speech frequency, amplitude, timbre, intonation, rhythm, and other parameters) stored in an embedded security chip. The high-precision microphone array may be a dual-microphone array. The speech signal may be a time-domain waveform signal that includes the user's voice input and noise. In practice, the glasses can use a low-energy microphone to acquire surrounding audio information in real time. When a speech segment matching the user's voiceprint characteristics is identified, the dual-microphone array can be used to obtain the time-domain waveform signal as the speech signal.

[0038] The second step is to input the speech signal into the encoder of a pre-trained noise separation model to obtain feature data. The pre-trained noise separation model includes the encoder, a mask generation network, and a decoder. The pre-trained noise separation model can be a simplified, pre-trained lightweight Conv-TasNet model structure. The pre-trained noise separation model is trained to remove noise from the time-domain signal while retaining the speech signal corresponding to the user's speech input. In practice, the glasses can input the speech signal (dimensions Batch×1×160) into the encoder's one-dimensional convolutional layer (64 filters, kernel length 20, stride 10, no padding) for local extraction, resulting in the output of the one-dimensional convolutional layer (dimensions Batch×64×15). The output of the one-dimensional convolutional layer is then compressed by splitting it into two parts along the channel dimension using a GLU and multiplying them (64 to 32) to obtain the compressed core features, i.e., feature data (dimensions Batch×32×15).

[0039] The third step is to input the above-mentioned feature data into the convolutional blocks of the mask generation network to obtain channel fusion information. The mask generation network includes convolutional blocks (which can be depthwise separable convolutional blocks) and a mask output layer. Each convolutional block includes a depthwise convolutional layer and a pointwise convolutional layer. In practice, the glasses can input the above-mentioned feature data into four pre-set depthwise separable convolutional blocks. Each depthwise separable convolutional block includes a depthwise convolutional layer (grouping 32 (1 channel per group), kernel length 3, and maintaining dimensionality) and a pointwise convolutional layer (1×1 convolution, 32 output channels). Layer normalization (LayerNorm) and parameterized ReLU (PReLU) are also introduced. Ultimately, channel fusion information (dimensions Batch×32×15) is obtained.

[0040] Fourth, the channel fusion information is input into the mask output layer to obtain a multi-channel tensor. In practice, the glasses can input the channel fusion information into the mask output layer. The mask output layer (1×1 convolution, 64 output channels (32 channels each for speech and noise)) can be normalized using a sigmoid function. The first 32 output channels of the mask output layer represent the speech mask (representing the user's speech input), and the last 32 represent the noise mask. That is, the first channels are trained to extract the speech mask, and the last channels are trained to extract the noise mask. Ultimately, a multi-channel tensor (dimensions Batch×64×15 (32×15 for speech mask, 32×15 for noise mask)) is obtained.

[0041] In the fifth step, the multi-channel tensor is segmented based on the pre-set distribution of convolution kernels in the mask output layer to obtain the speech signal mask. In practice, the glasses can perform channel segmentation on the multi-channel tensor using a single-slice operation. This yields a mask for the first 32 channels, i.e., the speech signal mask (dimensions Batch × 32 × 15).

[0042] In the sixth step, the masked speech signal is input into the decoder to obtain the denoised speech signal. In practice, the execution entity can input the masked speech signal into the transposed convolution layer (1 filter, kernel length 20, stride 10) of the decoder for transposed convolution (reconstruction into a time domain signal), obtaining the denoised speech signal (dimensions Batch × 1 × 160).

[0043] In the seventh step, the denoised speech signal is input into the causal convolution layer of a pre-trained feature extractor to obtain local feature data. The pre-trained feature extractor includes the causal convolution layer, an encoder layer, and a dynamic pooling layer. The pre-trained feature extractor can be a pre-trained lightweight CNN+Transformer hybrid architecture. The pre-trained feature extractor is trained to extract a speech feature matrix from the denoised speech signal. In practice, the glasses can input the denoised speech signal into the causal convolution layer (with 16 channels, a stride of 1, left padding of 79, and right padding of 0) to extract local features (i.e., short-term acoustic features such as pitch and formants) and obtain local feature data (dimensions Batch×16×160).

[0044] In the eighth step, the local feature data is input into the encoder layer to obtain global feature data. The encoder layer can be a single-layer Transformer encoder, consisting of four attention heads (each head corresponding to four dimensions, for a total of 16 dimensions) and 32 hidden layers of a feedforward network. In practice, the glasses can input the local feature data into the encoder layer to model global temporal dependencies (i.e., establishing cross-frame associations, such as connected speech changes and intonation trends). Residual connections and layer normalization are used to stabilize training and accelerate convergence. Finally, the ARM CMSIS-DSP library is used to accelerate matrix operations to obtain global feature data (with dimensions Batch × 16 × 160).

[0045] In the ninth step, the global feature data is fed into the dynamic pooling layer to obtain a speech feature matrix. In practice, the glasses can feed the global feature data into the dynamic pooling layer (pooling window 2, step size 2) for average pooling, compressing only the time dimension to obtain a speech feature matrix (Batch × 16 × 80 dimensions).

[0046] In the tenth step, a preliminary prediction is performed on the speech feature matrix to obtain a preliminary phoneme sequence and the phoneme probability of each preliminary phoneme in the preliminary phoneme sequence in the corresponding frame. The preliminary phoneme is the predicted phoneme with the highest probability for each frame. The preliminary phoneme sequence includes each preliminary phoneme. The phoneme probability of each preliminary phoneme in the corresponding frame is the predicted probability of each preliminary phoneme in the corresponding frame. In practice, the glasses can use a lightweight CRNN model (1D convolution, GRU) for prediction. First, a 1D convolution is performed on the speech feature matrix, using 32 convolution kernels (kernel length 3, causal padding) to extract local temporal features while maintaining temporal resolution. This produces a 1D convolution output (dimensions Batch × 32 × 80). Next, a 64-dimensional hidden state is applied to the 1D convolution output to model long-term dependencies. The state is updated frame by frame to obtain context-aware features (dimensions Batch × 80 × 64). Next, the context-aware features are linearly transformed to 40 dimensions (corresponding to the 40 pre-set phonemes) and the probabilities are normalized using Softmax, resulting in a phoneme probability matrix (dimensions Batch × 80 × 40). Finally, the maximum value of the 40-dimensional probability distribution is taken for each frame to obtain a preliminary phoneme sequence. For example, if the user says "turn on the light," the following results may be obtained: Frame 1: The probability of the phoneme / k / is 0.9, and the probability of / t / is 0.1; Frame 2: The probability of the phoneme / ai / is 0.85, and the probability of / ei / is 0.15. Similarly, a preliminary phoneme sequence (e.g., / k / / ai / / d / / eng / ) is obtained.

[0047] In step 11, based on the preliminary phoneme sequence, a set of candidate instructions and the probability of the preliminary phoneme sequence corresponding to each candidate instruction in the candidate instruction set are generated. The candidate instructions are executable operating instructions selected from a predefined instruction library and matching the phoneme sequence input by the user's voice. The predefined instruction library includes predefined executable instructions. The candidate instruction set includes each candidate instruction. The probability of the preliminary phoneme sequence corresponding to each candidate instruction in the candidate instruction set can be the confidence level of the match between the preliminary phoneme sequence and each candidate instruction, typically a normalized conditional probability value. In practice, the glasses can construct an instruction phoneme prefix tree. This involves converting the text of each instruction in the predefined instruction library into a phoneme sequence. A multi-level index tree is then constructed using phonemes as nodes, resulting in a prefix tree structure (supporting branch search by phoneme sequence and storing instruction names and frequencies of occurrence). Next, the preliminary phoneme sequence is input into the prefix tree, and the Viterbi algorithm is used to obtain each candidate instruction and its score. Finally, the scores of each candidate command are normalized using Softmax, yielding the probability of each candidate command in the candidate command set corresponding to the preliminary phoneme sequence. For example, if a user says "turn on the light," the candidate commands are: "turn on the light" (probability 0.85), "Keden" (probability 0.1), and "turn on" (probability 0.05).

[0048] Step 12: Screen the candidate instructions according to their probability to obtain voice data. In practice, the glasses may use the candidate instruction with the highest probability in the candidate instruction set as the voice data.

[0049] The above-mentioned first to twelfth steps serve as an inventive point of the embodiment of the present disclosure, which solves the problem that "the existing speech recognition technology has insufficient voice signal noise resistance, inaccurate voice feature extraction, and inability to efficiently generate accurate instructions in complex noise environments". The reasons why the existing technology is difficult to meet the needs of high-precision speech recognition are as follows: the existing methods have deficiencies in noise separation, voice feature extraction, phoneme sequence prediction, and instruction recognition, and are difficult to effectively adapt to different noise environments, different user voice features, and different instruction libraries. At the same time, there are defects in model robustness and computational efficiency, resulting in low recognition accuracy and the inability to fully meet the needs of high-precision speech recognition. If the above factors are solved, it can be achieved to improve the accuracy of speech recognition, optimize the robustness of the model, and achieve high-precision voice instruction recognition. In order to achieve this effect, the present invention adopts a speech recognition and instruction parsing method based on deep learning. In response to the recognition of a speech segment that meets the user's voiceprint characteristics, a high-precision microphone array is activated to obtain a speech signal; the above-mentioned speech signal is input into the encoder of a pre-trained noise separation model to obtain feature data, wherein the pre-trained noise separation model includes the above-mentioned encoder, a mask generation network and a decoder; the above-mentioned feature data is input into each convolution block of the above-mentioned mask generation network to obtain channel fusion information, wherein the above-mentioned mask generation network includes each convolution block and a mask output layer, and each convolution block in the above-mentioned each convolution block includes a depth convolution layer and a point convolution layer; the above-mentioned channel fusion information is input into the above-mentioned mask output layer to obtain a multi-channel tensor; according to the distribution of the convolution kernel in the pre-set mask output layer, the above-mentioned multi-channel tensor is channel-segmented to obtain a speech signal mask; the above-mentioned speech The signal mask is input into the above decoder to obtain a denoised speech signal; the denoised speech signal is input into the causal convolution layer in the pre-trained feature extractor to obtain local feature data, wherein the pre-trained feature extractor includes the above causal convolution layer, the encoder layer and the dynamic pooling layer; the local feature data is input into the above encoder layer to obtain global feature data; the global feature data is input into the above dynamic pooling layer to obtain a speech feature matrix; a preliminary prediction is made on the above speech feature matrix to obtain a preliminary phoneme sequence and the phoneme probability of each preliminary phoneme in the preliminary phoneme sequence in the corresponding frame; based on the preliminary phoneme sequence, a candidate instruction set and the instruction probability of each candidate instruction in the candidate instruction set corresponding to the preliminary phoneme sequence are generated; based on the instruction probability of each candidate instruction in the candidate instruction set, each candidate instruction is screened to obtain speech data. Specifically, the embodiment of the present disclosure achieves optimization through the following steps: Voiceprint activation and high-quality voice signal acquisition: activating a high-precision microphone array by identifying the user's voiceprint features to obtain a voice signal containing user voice input and noise. Noise separation: Use a pre-trained noise separation model to remove noise from the speech signal and retain the pure user voice input.Speech feature extraction: The speech feature matrix is ​​extracted from the denoised speech signal through a feature extractor of a lightweight CNN + Transformer hybrid architecture. Phoneme sequence prediction: The speech feature matrix is ​​processed using a lightweight CRNN model to generate a preliminary phoneme sequence and phoneme probability. Candidate instruction generation and screening: Combining the instruction phoneme prefix tree and the Viterbi algorithm, a candidate instruction set and instruction probability are generated, and the final voice instruction is screened out. Through the above steps, the embodiment of the present disclosure can achieve end-to-end optimization from the original speech signal to accurate voice instructions. In this way, it is possible to effectively extract speech features, flexibly adapt to different noise environments and user voices, optimize each link of speech recognition and instruction parsing, improve recognition accuracy and model robustness, and ultimately achieve high-precision voice command recognition.

[0050] Step 102 : In response to receiving user intent information sent by the terminal or the cloud or identifying user intent information, executing a preset operation corresponding to the user intent information.

[0051] In some embodiments, the execution entity may, in response to receiving or identifying user intent information sent by the terminal or the cloud, execute a preset operation corresponding to the user intent information. The user intent information sent by the terminal is derived from the terminal's recognition of received voice data. The user intent information sent by the cloud is derived from the cloud's recognition of received voice data. The received voice data is the voice data sent by the glasses to the terminal and the cloud, i.e., the voice data obtained by the glasses after processing the user's voice signal. The glasses' MCU (such as the Nordic nRF5340) can coordinate various sensors and peripherals to respond to user intent information in real time. In practice, when the glasses recognize user intent information from the voice data or receive user intent information sent by the terminal or the cloud, the glasses may execute the functions corresponding to one or more instructions in the user intent information in the order in which the instructions are contained. For example, if the glasses recognize or obtain the user intent information as "take a picture of the front," the glasses may control the camera to take the picture. Alternatively, the user intention information includes: "take a picture in front", "upload the picture to the cloud", "display the recognition results on the cloud", and the above-mentioned glasses can control the relevant processor to complete the related operations of taking pictures, uploading data, receiving data and visualization.

[0052] In some optional implementations of some embodiments, the user intention information sent by the terminal is generated by the following steps:

[0053] Step one, classify and identify the above-mentioned received voice data according to the terminal intention recognition model to obtain a classification recognition result. The above-mentioned classification recognition result is the keyword extracted by the above-mentioned terminal intention recognition model from the above-mentioned received voice data. The above-mentioned terminal intention recognition model can be a lightweight large language model based on Transformer Encoder. The number of attention heads of the above-mentioned terminal intention recognition model is 4 heads, the hidden layer dimension is 128 dimensions and the feedforward layer dimension is 256 dimensions. In practice, the above-mentioned terminal can locate keywords (such as verbs or nouns in voice data) by introducing the self-attention mechanism of Transformer in the above-mentioned terminal intention recognition model. In addition, pruning or quantization methods can be introduced to lightweight the terminal intention recognition model, and finally at least one keyword is obtained, and the obtained keyword is used as the classification recognition result.

[0054] Step 2: In response to the above-mentioned classification recognition result satisfying the preset classification recognition condition, the intent type corresponding to the above-mentioned received voice data is generated. The above-mentioned preset classification recognition condition may be that at least one keyword in the above-mentioned classification recognition result successfully matches the reference keyword in the preset intent type reference information. The above-mentioned preset intent type reference information may include a preset reference keyword (such as a verb or noun) that represents the intent type. The above-mentioned preset intent type reference information may support the extension of reference keywords by updating the yaml configuration file. The intent type corresponding to the above-mentioned received voice data includes at least one keyword in the above-mentioned classification recognition result that satisfies the preset classification recognition condition. In practice, the above-mentioned terminal may match the keywords in the above-mentioned classification recognition result with each reference keyword in the above-mentioned preset intent type reference information. When at least one keyword successfully matches at least one reference keyword in the above-mentioned preset intent type reference information, the above-mentioned at least one keyword is determined as the intent type corresponding to the above-mentioned received voice data.

[0055] Step three: Parameter extraction is performed on the voice data received by the terminal based on the intent type to obtain parameter data. The parameter data can be structured operational parameters (such as location or time) extracted from the voice data and directly related to the intent type. In practice, the terminal first uses a lightweight word segmentation tool (such as rule-based subword segmentation or a pre-trained word segmenter) to convert the voice data received by the terminal into a subword sequence. The subword sequence is then converted into a 64-dimensional word vector. A 6×32-dimensional context feature matrix is ​​captured using a single-layer bidirectional LSTM (with a hidden layer dimension of 32), resulting in a score for each subword on different labels. Next, using the dynamic programming Viterbi algorithm and combined with the transfer rules between labels (i.e., the rationality of the transfer between labels, such as the "time" label followed by "time" gets 1.2 points, and "location" is deducted 0.5 points), the sequence with the highest comprehensive score (the score is composed of the weighted combination of word label matching and label transfer rationality) is selected from all possible label paths as the label sequence (such as ["B-TIME", "I-TIME", "O", "B-CONTENT"]). Then, by merging consecutive labels (such as merging "B-TIME" and "I-TIME" into time parameters) and numerical normalization (such as converting "3 pm" to 3:00 pm), structured operation parameters directly related to the intent type, i.e., parameter data, are obtained.

[0056] Step 4: Generate user intent information and a cancellation signal based on the intent type and parameter data. In practice, the execution entity can combine the intent type and parameter data using JSON Schema (converting them into a standard data structure) for structured encapsulation to obtain user intent information. Simultaneously, a cancellation signal can be generated.

[0057] In some optional implementations of some embodiments, the user intent information sent by the cloud is generated by the following steps:

[0058] Step 1: Generate at least one initial user intent information based on the received voice data. This initial user intent information can be the output of a large language model generated by the cloud-based analysis of the received voice data, representing raw, unstructured semantic output. It may contain multiple intents (e.g., two independent commands: "play music" and "lower the volume"), may contain natural language redundancy (e.g., "I want the system to turn on the lights in the living room"), and may contain ambiguous expressions (e.g., the word "appropriate" in "adjust to the appropriate brightness" is unquantified). In practice, after receiving the voice data sent by the glasses, the cloud-based system can construct a dynamic semantic graph instead of keyword extraction. First, the voice data is subjected to dependency parsing to generate a grammar tree (using Stanford Parser or UDpipe), extracting the subject-verb-object core framework as graph nodes. Next, a GNN is used to model the semantic relationships between the grammar tree nodes, resulting in a weighted semantic relationship matrix. Finally, temporal features of user behavior (feature vectors of the last 30 minutes of operation records extracted using an LSTM) are integrated with the semantic graph for cross-modal fusion, and a contextual relevance score is calculated using an attention mechanism. Ultimately, a differentiable rule engine (such as Neural Logic Machines) maps the fused features into a multi-dimensional intent vector. Based on a reinforcement learning strategy (using reinforcement learning methods to select the most appropriate large language model combination for intent recognition based on the characteristics and processing requirements of the multi-dimensional intent vector), the optimal large model combination (such as BERT + GPT-3 hybrid reasoning) is dynamically selected to obtain initial user intent information. A confidence distribution heat map can also be generated for subsequent verification.

[0059] Step 2: Generate user intent information based on the at least one initial user intent information. In practice, the cloud can first build a multimodal intent distillation layer to fuse real-time sensor data from the glasses (such as spatial orientation and ambient light intensity) with the at least one initial user intent information. Using a lightweight network, it extracts core operation atoms (e.g., compressing "shoot and recognize" into device instruction code), while dynamically adjusting execution strategies based on device power and temperature. An adaptive DAG orchestration engine then runs on the cloud to analyze hardware characteristics (e.g., camera model support for HDR) and establish cross-modal dependencies (e.g., AR rendering requires spatial positioning). It also pre-defines a primary and backup execution scheme for each operation (with local NPU priority and cloud-based downgrade). Finally, a compact instruction set is generated using a quantized coding protocol, converting natural language into binary control flow (e.g., 0xA1 to start 4K video), and incorporating parallel execution optimization (dual-core task splitting) and security protection instructions (memory boundary locks). The multimodal intent distillation layer can be a cloud-based structure or functional module that fuses multimodal data (e.g., real-time sensor data from the glasses and initial user intent information). The lightweight network can be a network structure that extracts core operation atoms. The adaptive DAG orchestration engine can be an engine that analyzes hardware characteristics and establishes cross-modal dependencies (dependencies between different modalities, such as AR rendering and spatial positioning, to determine the execution order and conditions of operations), as well as pre-setting a primary and backup execution plan for each operation. The quantized encoding protocol can be an encoding protocol for converting natural language into a compact binary control flow. For example, when processing a "real-time translation menu" command, the cloud automatically removes translation engine calls that cannot be completed by the glasses side and reconstructs it into a collaborative workflow of camera capture, cloud-based OCR, and AR overlay. It generates a quantized instruction packet (generated via the quantized encoding protocol, containing information such as sensor synchronization requirements and service session binding) as user intent information, including sensor synchronization requirements (IMU 100Hz sampling) and service session binding (associating the generated instruction with a specific service session for subsequent tracking and management).

[0060] In some optional implementations of some embodiments, the cloud may generate at least one initial user intent information based on the received voice data through the following steps:

[0061] Step 1: Intent extraction is performed on the received voice data to obtain a list of intent keywords. These intent keywords refer to the verbs and nouns in the voice data. In practice, the cloud can use NLP tools (such as spaCy, NLTK, or jieba) to segment and annotate the received voice data. Non-critical components (such as non-verbs or non-nouns) are then filtered to obtain individual intent keywords, which are then combined into a list of intent keywords. For example, if the voice data is "Play XXX (any singer's name)'s 'YY (any song name) and turn down the volume," the verbs that can be extracted include "play" and "turn down," and the nouns include "XXX," "YY," and "volume." The resulting list of intent keywords is ("play," "XXX," "YY," "turn down," and "volume").

[0062] Step 2: Based on the pre-set complexity information, the complexity of the above-mentioned intent keyword list is extracted to obtain a complex keyword set. The pre-set complexity information can be the complexity corresponding to the intent keyword defined according to the domain requirements (such as more complex keywords can be professional terms, compound logical words, technical phrases). The pre-set complexity information is set in a predefined regular expression of complex keywords or a predefined keyword library (such as Elasticsearch Term Query). The above-mentioned complex keyword set includes various complex keywords. The above-mentioned complex keywords can be intent keywords with higher complexity in the above-mentioned intent keyword list. In practice, the above-mentioned cloud can use a predefined regular expression of complex keywords or a predefined keyword library to match the complexity of each intent keyword in the above-mentioned intent keyword list. After matching, each intent keyword with higher complexity is used as a complex keyword, and the above-mentioned complex keywords are integrated into a complex keyword set.

[0063] Step three: Generate the complexity level corresponding to the received voice data based on the complex keyword set and the pre-set complexity information. The complexity level corresponding to the received voice data is based on pre-set voice data scoring rules. The pre-set voice data scoring rules may calculate a complexity score based on the number and type of complex keywords (e.g., +2 points for each technical term, +1 point for each logical term; a total score > 5 indicates "high complexity"). In practice, any classification model (e.g., SVM, BERT) can be used to classify complex keywords based on their type to obtain the type of each complex keyword. The scores of each complex keyword are then statistically analyzed to obtain the complexity level corresponding to the received voice data. For example, if the complex keyword set is ("if," "and," "index," "air purifier"), the type results obtained by the classification model for each complex keyword can be scored and calculated. For the complex keyword set ("if", "and", "index", "air purifier"), technical words (index, air purifier) ​​scored 4 points, logical words (if, and) scored 2 points, and the final total score was 6. According to the pre-set voice data scoring rules, it can be obtained that the complexity level corresponding to the above-mentioned received voice data is high complexity.

[0064] Step 4: Generate a model identifier based on the matching information of the pre-set large language model and the complexity level of the received speech data. The pre-set large language model matching information can be a predefined relationship between complexity and model (e.g., low complexity corresponds to lightweight models such as Tongyi Qianwen or Kimi; high complexity corresponds to heavy models such as GPT-4 or DeepSeek). The model identifier can be a structured label (e.g., a string, ID, or encoding). Each model identifier corresponds to a different large language model (e.g., Tongyi Qianwen or DeepSeek). In practice, the cloud can adjust the model selection strategy based on the complexity level of the received speech data and real-time resources (e.g., server load). This generates a model identifier corresponding to the speech data. For example, if the complexity level of the received speech data is high, since high complexity corresponds to heavy models, the real-time load of each heavy model can be obtained to generate a model identifier corresponding to the heavy model with the lower load (e.g., if GPT-4 has a lower load, the generated model identifier is model_id="GPT-4").

[0065] Step 5: Input the received voice data into the target large language model corresponding to the model identifier to obtain at least one initial user intent information. The initial user intent information may be the output of the large language model. The target large language model may be the large language model corresponding to the model identifier. In practice, the cloud may input the received voice data into the target large language model to obtain the output of the target large model as the at least one initial user intent information.

[0066] In some optional implementations of some embodiments, the cloud may generate user intent information based on the at least one initial user intent information by performing the following steps:

[0067] Step 1: Perform semantic conversion on the at least one initial user intent information to obtain at least one converted initial user intent information. The at least one converted initial user intent information can be the corresponding information obtained after converting the at least one initial user intent information. The at least one converted initial user intent information can be structured executable instructions. This semantic conversion converts information from "what the user says" to "what the system does." By combining it with a large language model, it converts open-domain natural language intent into closed-domain executable instructions. In practice, the cloud can merge continuous and similar initial user intent information within the at least one initial user intent information and split different types of initial user intent information. Then, tools such as Drools and EasyRules can be used to define the conversion logic, mapping the colloquial description into output structured executable instructions (including multi-terminal executable instructions), thereby obtaining the at least one converted initial user intent information. For example. If the initial user intent is "Play the latest science fiction film in my home theater in the living room, but don't make too much noise," the cloud can merge consecutive and similar initial user intents within the initial user intent information and separate different categories of initial user intents, such as into the two dimensions of "device control" and "media playback." Through spatial semantic analysis, "living room" is automatically bound to the pre-stored audio equipment, "science fiction film" is matched to "XX" (any movie title) through the content library search, and "don't make too much noise" is converted to a 55dB threshold using a noise model. The converted initial user intent can be: first, the TV's HDMI-CEC interface (physical address 1.0.0.0) is activated, a playback task is pushed via the DLNA protocol, the audio system's DBAP (Dolby Atmos Profile) preset cinema mode is simultaneously invoked, and finally, a dynamic volume control is implemented (which adjusts the gain in real time based on ambient noise). When a Dolby Vision-quality video is detected, HDR metadata calibration instructions are automatically added to ensure color space synchronization.

[0068] Step 2: Generate a list of candidate atomic operations based on the converted at least one initial user intent information and the glasses-side specific capability graph. The glasses-side specific capability graph may be a knowledge graph describing the relationships between AR glasses hardware resources and software capabilities. The entity nodes of the glasses-side specific capability graph represent nodes corresponding to hardware entities and nodes corresponding to functional modules. The edges of the glasses-side specific capability graph represent hardware support relationships and functional dependencies. Each entity node corresponds to an edge relationship. The hardware support relationship may be the direct support capability of a hardware component for other hardware or software functions. The functional dependency relationship may be the dependency of a software module or function on hardware resources. The list of candidate atomic operations may include various hardware-level basic operation units (operation items). These hardware-level basic operation units may be the minimum executable instructions that directly control pre-configured and operational glasses-side hardware resources, typically involving calls to underlying hardware drivers or firmware interfaces. These operation instructions are indivisible and represent the atomic steps for building complex functions. In practice, the cloud can extract entity nodes from the converted at least one initial user intent information (i.e., extract hardware entities and functional modules). Next, a bidirectional breadth-first search algorithm is used to traverse the edge relationships in the glasses-specific capability graph, integrating the edge relationships corresponding to the entity nodes into candidate atomic operations, thereby obtaining a list of candidate atomic operations. For example, the cloud extracts entities from the at least one initial user intent information after conversion to obtain nodes corresponding to hardware entities such as "binocular camera" and "OV9281 sensor," as well as nodes corresponding to functional modules such as "NPU computing unit" and "offline OCR engine." A bidirectional breadth-first search algorithm is then used to traverse the glasses-specific capability graph to obtain hardware support relationships such as "OV9281 - Support video streaming - CSI-2 interface" and functional dependency relationships such as "OCR engine - Dependency on computing power - NPU." Finally, the hardware support relationships and functional dependency relationships are integrated into a list of candidate atomic operations, including operation items such as "OV9281 sensor initialization," "Enable CSI-2 video streaming," "NPU loading," "OCR model allocation," and "DMA buffer (1080p@30)."

[0069] Step three: Add spatiotemporal constraint tags to the candidate atomic operation list to obtain a spatiotemporal dependency-tagged operation list. The spatiotemporal constraint tags can be metadata tags attached to atomic operations, describing the constraints imposed on the operation in the spatial coordinate system and time dimension. The spatiotemporal dependency-tagged operation list can be a set of atomic operations (operation items) that include complete spatiotemporal constraint descriptions, reflecting the spatial correlation and temporal coordination requirements between the operations. In practice, the cloud can use a dependency syntax tree (a grammatical parse tree that reveals the logical relationship between the core predicate and modifiers in a sentence) to parse the spatial adverbials in the converted at least one initial user intent information to obtain spatial relationship information in the converted at least one initial user intent information. Then, by combining predefined sensor and coordinate system mapping rules (a set of structured logic that maps spatial relationships described in natural language and physical data collected by sensors to spatial coordinate systems and temporal constraints computable on the device), spatiotemporal tags are added to each operation item in the candidate atomic operation list. Finally, transformation relationships between multiple coordinate systems are pre-established using Lie group transformation matrices to obtain a spatiotemporal dependency-tagged operation list. For example, the cloud can use the predicate verb "follow" and the spatial adverb "line of sight focus" extracted from at least one initial user intention information after the above conversion, such as "follow the line of sight focus display", to obtain the spatial relationship information in the at least one initial user intention information after the above conversion, such as "follow the line of sight focus". Then, combined with the predefined sensor and coordinate system mapping rules, "focus" is mapped to the eye coordinate system, and "display" is associated with the world coordinate system. Spatiotemporal tags are added to the operation items in the above candidate atomic operation list, such as "bind to the eye coordinate system, require 120Hz tracking frequency", to obtain a candidate atomic operation list with added spatiotemporal tags. Finally, the conversion relationship between multiple coordinate systems is established through the Lie group transformation matrix to obtain spatiotemporal dependency tags such as the required sensor type, coordinate system binding requirements, and the spatiotemporal dependency tags are added to the candidate atomic operation list with added spatiotemporal tags to obtain an operation list with spatiotemporal dependency tags.

[0070] Step 4: Verify the operation list with spatiotemporal dependency markers to obtain a verification result. The verification result indicates whether each operation item in the operation list with spatiotemporal dependency markers has passed the verification. The verification result may be information conclusively determining whether the operation list with spatiotemporal dependency markers meets predetermined standards or requirements. In practice, the cloud can first generate each key entity in the converted at least one user intent information and each key entity for each operation item in the operation list with spatiotemporal dependency markers. Next, the cloud can perform a match verification on each key entity of each operation item with each key entity of the converted at least one initial user intent information. For each key entity of each operation item and each key entity of the converted at least one user intent information, the cloud can calculate the similarity between the two key entities using a string similarity algorithm (such as the Levenshtein distance algorithm) or a semantic similarity algorithm (such as cosine similarity after calculating word embeddings using a pre-trained language model such as Word2Vec or BERT), and set a similarity threshold (e.g., a string similarity threshold of 0.8 and a semantic similarity threshold of 0.7) to achieve match verification. If the similarity is greater than or equal to the threshold, it indicates that the two key entities are matched successfully; otherwise, the match fails. If the key entity of each operation item can be successfully matched with any key entity of at least one of the converted initial user intent information, the test result indicates that the test has passed. If the key entity of any operation item fails to match, the test result indicates that the test has not passed. The key entities can be words, phrases or structured information extracted from text data that can represent the core semantic information of the text. For example, for the converted initial user intent information "make an appointment for a video conference at 10 am tomorrow", the cloud first uses the entity extraction algorithm to extract the key entities "appointment", "tomorrow morning 10 am", and "video conference" from the intent. Then, each operation item in the operation list with the spatiotemporal dependency label is tested. For example, for the operation item "open the scheduling application and create a new event", the extracted key entity is "scheduling application" "create a new event", which matches the key entity "appointment" in the converted initial user intent information, indicating the start of the appointment behavior, and passes the test. Next, the key entities for the action item "Set event time to tomorrow morning at 10:00 AM" are "Set event time" and "Tomorrow morning at 10:00 AM," which match the key entity "Tomorrow morning at 10:00 AM" in the converted initial user intent information, thus passing the test. Another example is the key entities for the action item "Add event type to video conference," which are "Add event type" and "Video conference," which match the key entity "Video conference" in the converted initial user intent information, thus passing the test. That is, each action item is tested. If each action item passes the test, a test result is obtained, indicating that the test has passed. If any action item fails the test, a test result is obtained, indicating that the test has failed.

[0071] Step 5: In response to the above-mentioned inspection result indicating that the inspection has failed, feedback information is generated, and the above-mentioned feedback information is visualized. The above-mentioned feedback information can be a data information that has not passed the inspection in the above-mentioned operation list with spatiotemporal dependency tags (such as the operation item or the corresponding spatiotemporal constraint tag that has not passed the inspection). In practice, the above-mentioned cloud can use the data information that has not passed the inspection in the above-mentioned operation list with spatiotemporal dependency tags as feedback information, and send the feedback information to the above-mentioned glasses end, and the above-mentioned glasses end can visualize the above-mentioned feedback information. For example, because Bluetooth is not turned on, the operation item "Connect to mobile phone application via Bluetooth" in the above-mentioned operation list with spatiotemporal dependency tags is not matched to the corresponding entity node. The feedback information "Failed to connect to mobile phone application via Bluetooth" is generated. A blue "i" icon is displayed in the field of view of the glasses end, indicating that there is information that needs to be viewed. After recognizing that the icon is clicked, a transparent message box pops up to display the feedback information.

[0072] Step 6: In response to the above-mentioned test result characterization passing the test, user intent information is generated based on the above-mentioned operation list with spatiotemporal dependency tags. In practice, when the above-mentioned test result characterization passes the test, the above-mentioned cloud uses a rule matching algorithm to strip the cloud dependencies of each operation item in the above-mentioned operation list with spatiotemporal dependency tags according to the external resource whitelist (which can be a predefined set of operation items allowed to be executed on the local device), obtains a sequence of executable operation items on the glasses side (which can be operation items that can only be executed on the local device), and uses the above-mentioned sequence of executable operation items on the glasses side as user intent information. Among them, the above-mentioned external resource whitelist can include operation items related to the local computing module, operation items related to hardware direct connection operations, and operation items related to lightweight AI models. For example, the cloud uses a rule matching algorithm to extract a sequence of executable operation items on the glasses side (such as "calling the OV9281 camera for shooting", "using the NPU model analysis", and "calling the GPT-4 translation engine") from each operation item in the above-mentioned operation list with spatiotemporal dependency tags (such as "OCR detection" or "professional translation") according to the external resource whitelist (which may include local computing module-related operation items (such as controlling the offline OCR engine, basic SLAM positioning, etc.); hardware direct connection operation-related operation items (such as sensor data preprocessing, low-latency rendering, etc.); lightweight AI model-related operation items (such as gesture recognition, voice wake-up and other embedded models)). Because "calling the GPT-4 translation engine" does not involve operation items in the external resource whitelist, the cloud dependency (such as "professional translation") is stripped off to obtain a sequence of executable operation items on the glasses side. Finally, the above-mentioned sequence of executable operation items on the glasses side is used as user intent information.

[0073] In some optional implementations of some embodiments, the cloud may verify the operation list marked with spatiotemporal dependency through the following steps to obtain a verification result:

[0074] Step 1: Perform a correctness check on the aforementioned operation list with spatiotemporal dependency markers. This correctness check can include checking whether each candidate atomic operation corresponds to the correct hardware entity or functional module, and whether the corresponding parameter settings for the operation item are reasonable (e.g., whether the device address, protocol type, etc. are correct). In practice, the cloud can call upon the glasses-specific capability map to perform a correctness check on the aforementioned operation list with spatiotemporal dependency markers. If the hardware entity or functional module corresponding to each operation item in the spatiotemporal dependency markers has a corresponding entity node in the glasses-specific capability map, and the corresponding parameter settings for the operation item are reasonable, the operation list with spatiotemporal dependency markers passes the correctness check. Conversely, if any operation item fails the correctness check, the operation list with spatiotemporal dependency markers fails the correctness check, and the verification result indicates failure, discontinuing the verification of the operation list with spatiotemporal dependency markers. For example, the voice data can be "Start AR navigation while maintaining the display of navigation information relative to the current line of sight, and synchronize the navigation information to my mobile app in real time." The spatiotemporal dependency-tagged operation list obtained through the above steps may include the following operation items and their associated spatiotemporal constraint tags, hardware resources, and parameters: initializing the AR navigation module, activating the binocular camera, invoking the NPU for environmental perception calculations, updating navigation information based on SLAM results, encoding navigation information into a transmission format, connecting to a mobile app via Bluetooth, transmitting navigation information to the phone in real time, monitoring the navigation process and handling exceptions, and other operation items. The glasses' dedicated capability map can be used to verify whether operation items in the operation list, such as "Initialize the AR Navigation Module," correctly correspond to the glasses' navigation function module. The operation item parameter settings can then be checked, such as verifying that the mobile device address corresponding to the "Real-time Navigation Information to the Phone" operation item is correct and that the synchronization protocol complies with requirements. If each operation item in the spatiotemporal dependency-tagged operation list, such as "Initialize the AR Navigation Module," corresponds to the glasses' navigation function module, and the corresponding parameters (mobile device address) for each operation item, such as "Real-time Navigation Information to the Phone," are also set appropriately, the spatiotemporal dependency-tagged operation list passes the aforementioned correctness check.

[0075] Step 2: In response to the above-mentioned operation list with time-space dependency marks passing the above-mentioned correctness test, the hardware performance status information is obtained. The above-mentioned hardware performance status information can be detailed data on the operating status and performance of the device hardware resources. In practice, when each operation item in the above-mentioned operation list with time-space dependency marks passes the above-mentioned correctness test, the above-mentioned execution subject obtains the current hardware performance status information (such as CPU status information, memory information, battery information, etc.) through the pre-set hardware performance monitoring module on the glasses. The above-mentioned pre-set hardware performance monitoring module on the glasses can be a software and hardware component integrated on the glasses, which collects data in real time through sensors and chips, and can obtain hardware performance status information through the operating system API or background service program.

[0076] Step three, based on the above hardware performance status information, perform a feasibility test on the above operation list with spatiotemporal dependency marks. The feasibility test can be to compare the various operation items in the above operation list with spatiotemporal dependency marks with the current hardware performance status information, and check whether the current hardware (entity nodes) meet the execution requirements of each operation item. The above execution requirement can be that the hardware corresponding to the operation item is in normal operating state. In practice, the above execution subject can send the hardware performance status information to the above cloud, and the above cloud performs a feasibility test on the above operation list with spatiotemporal dependency marks. When the hardware corresponding to each operation item meets the execution requirements of each operation item, the above operation list with spatiotemporal dependency marks passes the feasibility test. On the contrary, if the hardware corresponding to any operation item fails to be tested successfully (does not meet the execution requirements), the above operation list with spatiotemporal dependency marks fails the feasibility test, and the above test result indicates that the test has failed, and the test of the above operation list with spatiotemporal dependency marks is interrupted. For example, the cloud can use hardware performance status information and the key entities corresponding to each operation item in the above-mentioned operation list with spatiotemporal dependency tags to determine whether the physical nodes corresponding to each hardware item can support the startup and operation of the AR navigation function and whether they can meet the requirements for synchronizing data with mobile phone applications. When the physical nodes corresponding to each hardware item can support the startup and operation of the AR navigation function and meet the requirements for synchronizing data with mobile phone applications (such as the normal connection between the wireless transmission port related to information transmission on the glasses and the mobile phone), it can be obtained that the above-mentioned operation list with spatiotemporal dependency tags passes the above-mentioned feasibility test.

[0077] In step 4, in response to the operation list with spatiotemporal dependency markings passing the feasibility check, a conflict check is performed on the operation list with spatiotemporal dependency markings based on the hardware performance status information. The conflict check can be performed by checking whether there is resource contention or operation order conflict between the operation items using a conflict detection algorithm. The conflict detection algorithm can be a predefined algorithm for identifying resource contention or operation order conflict. In practice, the cloud can use the conflict detection algorithm to perform a conflict check on each piece of hardware (physical node) corresponding to each key entity in the operation list with spatiotemporal dependency markings. If the hardware corresponding to each key entity does not encounter resource contention or operation order conflict, the operation list with spatiotemporal dependency markings passes the conflict check. Conversely, if any piece of hardware (physical node) fails the test, the operation list with spatiotemporal dependency markings fails the conflict check, and the test result indicates that the test failed, and the test of the operation list with spatiotemporal dependency markings is interrupted. For example, the cloud can use a conflict detection algorithm to detect that there will be no camera resource contention or operation order conflicts when executing AR navigation and data synchronization operations based on the key entities corresponding to each operation item in the operation list with spatiotemporal dependency tags and the hardware corresponding to each key entity. Therefore, the operation list with spatiotemporal dependency tags can pass the conflict check.

[0078] In step 5, in response to the above-mentioned operation list with spatiotemporal dependency passing the above-mentioned conflict check, the above-mentioned operation list with spatiotemporal dependency is parameter-checked. The above-mentioned parameter check can be performed by checking the parameters in each operation item one by one to ensure that the parameters involved comply with the predefined range and format. In practice, the above-mentioned cloud can check the parameters corresponding to each operation item in the above-mentioned operation list with spatiotemporal dependency one by one. When the parameters corresponding to each operation item in the above-mentioned operation list with spatiotemporal dependency all comply with the predefined range and format, the above-mentioned operation list with spatiotemporal dependency passes the above-mentioned parameter check. Conversely, if the parameters corresponding to any operation item fail to be successfully checked, the above-mentioned operation list with spatiotemporal dependency fails the above-mentioned parameter check, and the above-mentioned test result indicates that the test has failed, and the test of the above-mentioned operation list with spatiotemporal dependency is interrupted. For example, check whether the location parameters displayed by the navigation information comply with the predefined coordinate system requirements, whether the navigation update frequency is within a reasonable range, etc. When the format and range of the parameters corresponding to each operation item meet the predefined range and format, it can be concluded that the above-mentioned operation list with spatiotemporal dependency passes the above-mentioned parameter check.

[0079] In step six, in response to the above-mentioned operation list with time-space dependency marks passing the above-mentioned parameter verification, the preset information indicating that the test has been passed is determined as the test result. The above-mentioned preset information indicating that the test has been passed can be a preset data information (such as the words "passed the test"). In practice, when the above-mentioned operation list with time-space dependency marks passes the above-mentioned parameter verification, the above-mentioned cloud can determine the preset information indicating that the test has been passed as the test result. Then the current test result indicates that the above-mentioned operation list with time-space dependency marks has passed the test.

[0080] Optionally, the glasses can transmit the acquired multimodal data corresponding to the voice data to the cloud. The multimodal data is used by the cloud to generate user intent information based on the converted at least one initial user intent information. The multimodal data can be an image or video. In practice, the voice data may be "identify the image or video in front of you," and the user intent information sent from the cloud to the glasses may be "take a photo and upload it to the cloud" or "take a video and upload it to the cloud." After capturing the image or video, the glasses can transmit it to the cloud. Since the converted at least one initial user intent information includes instructions executable by multiple devices, the voice data may be "identify the image or video in front of you," and the cloud-executable instructions in the converted at least one initial user intent information may include "identify the multimodal data uploaded by the glasses," "analyze the multimodal data," and "send the recognition results to the glasses." When the cloud obtains a recognition result for the image or video through the above steps, the obtained user intent information may be "visualize the recognition results on the cloud."

[0081] Step 103: In response to identifying the user intention information, a cancellation signal is sent to the terminal and the cloud.

[0082] In some embodiments, the execution entity may send a cancellation signal to the terminal and the cloud in response to recognizing user intent information. The cancellation signal is used to interrupt the tasks corresponding to the voice data in the terminal and the cloud. In practice, the glasses can process offline instructions and determine simple intents through regular relationships. Since the glasses have the fastest recognition speed among the three terminals, the glasses can simultaneously send a cancellation signal to the terminal and the cloud when recognizing user intent information. The terminal uses the terminal intent recognition model to perform terminal-side judgment on more scattered but more frequent clear instructions. When the terminal recognizes the cancellation signal sent by the glasses, it can interrupt all tasks related to the voice data in the terminal. The cloud can call on the existing large language model for in-depth analysis and recognition of more complex information. When the cloud recognizes the cancellation signal sent by the glasses or the terminal, it can interrupt all tasks related to the voice data in the cloud.

[0083] The various embodiments of the present disclosure described above have the following beneficial effects: The voice command processing methods of some embodiments of the present disclosure enable highly efficient, low-latency, and robust voice interaction processing, significantly improving user experience and system practicality. Specifically, traditional voice interaction architectures, such as single-end and dual-end architectures, can lead to functional failure in weak or no network environments, high latency in processing complex intents (typically ≥ 2 seconds), and high computational and communication costs associated with frequent cloud calls. To address this, the voice command processing methods of some embodiments of the present disclosure first recognize voice data on the eyewear side and simultaneously send it to the terminal and the cloud, ensuring multi-level data processing and rapid response. This provides a foundation for multi-end collaboration for efficient processing of complex intents, ensuring data diversity and timely processing. Then, in response to receiving or recognizing user intent information sent by the terminal or the cloud, the eyewear side executes the preset action corresponding to the user intent information. This combination of localized processing and recognition significantly reduces latency and improves system responsiveness. At the same time, after receiving voice data from the glasses, the terminal can also identify the voice data and generate user intent information, further optimizing task allocation and resource utilization. This lightweight processing on the terminal ensures efficient recognition and rapid feedback for even complex intents. Furthermore, the cloud can also analyze the voice data and generate user intent information after receiving it from the glasses. This deep cloud processing significantly improves the processing accuracy and system robustness of even more complex intents. Furthermore, in response to identifying user intent, the glasses send a cancel signal to the terminal and the cloud, interrupting the task corresponding to the voice data in both terminals and the cloud. This avoids unnecessary waste of computing and communication resources, improves system efficiency, and reduces energy consumption. Consequently, through multi-terminal collaboration and dynamic task allocation, efficient processing and low-latency response for voice interactions are achieved, generating highly accurate and reliable interaction results that can effectively handle complex voice interaction scenarios. This provides a more reliable and practical solution for fields such as augmented reality and smart devices.

[0084] Reference below Figure 2 , which shows a structural schematic diagram of a head-mounted display device 200 suitable for implementing some embodiments of the present disclosure. Figure 2 The head-mounted display device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0085] like Figure 2As shown, the head-mounted display device 200 may include a processing device 201 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 202 or programs loaded from a storage device 208 into a random access memory (RAM) 203. Various programs and data required for the operation of the head-mounted display device 200 are also stored in the RAM 203. The processing device 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0086] Typically, the following devices may be connected to the I / O interface 205: an input device 206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 207 including, for example, a display, a speaker, a vibrator, etc.; and a communication device 209. The communication device 209 may allow the head-mounted display device 200 to communicate with other devices wirelessly or by wire to exchange data. The head-mounted display device 200 may include at least one display screen (such as a MicroOLED or MicroLED) for forming an image in front of the user's eyes. Although Figure 2 The head mounted display device 200 is shown with various devices, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 2 Each block shown in the figure may represent one device, or may represent multiple devices as needed.

[0087] In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are performed.

[0088] It should be noted that the computer-readable medium described in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. Furthermore, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0089] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0090] The computer-readable medium may be included in the head-mounted display device, or may exist independently and not be incorporated into the head-mounted display device. The computer-readable medium carries one or more programs. When executed by the head-mounted display device, the one or more programs cause the head-mounted display device to: recognize voice data, synchronously transmit the voice data to a terminal and the cloud, and, in response to receiving or recognizing user intent information, execute a preset operation corresponding to the user intent information.

[0091] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0092] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0094] Figure 3 It is a structural diagram of some embodiments of the instruction processing system 300 based on the multi-terminal architecture according to the present disclosure. Figure 3 Including glasses, terminals and cloud.

[0095] In some embodiments, the multi-terminal architecture-based instruction processing system 300 may include:

[0096] The glasses end 301 is configured to recognize voice data and send the above voice data to the terminal and the cloud synchronously; in response to receiving or recognizing user intention information, execute the preset operation corresponding to the above user intention information; in response to recognizing user intention information, send a cancellation signal to the above terminal and the above cloud.

[0097] The terminal 302 is configured to recognize the received voice data; in response to recognizing the user intention information, the recognized user intention information is sent to the above-mentioned glasses end.

[0098] The cloud 303 is configured to generate at least one initial user intention information based on the received voice data; generate user intention information based on the at least one initial user intention information, and send the generated user intention information to the glasses end.

[0099] Figure 1 The operations, features and beneficial effects described for the glasses, terminals and cloud in the corresponding embodiments are also applicable to the instruction processing system 300 based on the multi-terminal architecture and the glasses, terminals and cloud contained therein, and will not be repeated here.

[0100] The above descriptions are merely some preferred embodiments of the present disclosure and illustrate the underlying technical principles. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A method for processing voice commands, applied to glasses, comprising: Recognize voice data and send the voice data to the terminal and the cloud synchronously; In response to receiving user intent information sent by the terminal or the cloud or identifying user intent information, executing a preset operation corresponding to the user intent information, wherein the user intent information sent by the terminal is obtained by the terminal recognizing the received voice data, and the user intent information sent by the cloud is obtained by the cloud recognizing the received voice data, wherein the user intent information sent by the cloud is generated by the following steps: generating at least one initial user intention information based on the received voice data; Generating user intent information according to the at least one initial user intent information includes: Performing semantic conversion on the at least one initial user intention information to obtain converted at least one initial user intention information; generating a candidate atomic operation list based on the converted at least one initial user intention information and the glasses-side exclusive capability map; Adding time and space constraint tags to the candidate atomic operation list to obtain an operation list with time and space dependency tags; Testing the operation list with the time-space dependency mark to obtain a test result; generating feedback information in response to the inspection result indicating that the inspection has failed, and visualizing the feedback information; In response to the inspection result indicating that the inspection has been passed, generating user intention information according to the operation list with the spatiotemporal dependency markers; In response to identifying the user intention information, a cancellation signal is sent to the terminal and the cloud, wherein the cancellation signal is used to interrupt the task corresponding to the voice data in the terminal and the cloud.

2. The method according to claim 1, wherein The recognition of the voice data includes: Performing preliminary recognition on the voice data to obtain a preliminary recognition result; In response to the preliminary recognition result satisfying a preset preliminary recognition condition, generating user intention information and a cancellation signal; In response to the preliminary recognition result not meeting a preset preliminary recognition condition, performing secondary recognition on the voice data to obtain a secondary recognition result; In response to the secondary recognition result satisfying a preset secondary recognition condition, user intention information and a cancellation signal are generated.

3. The method according to claim 1, wherein The user intention information sent by the terminal is generated by the following steps: Classify and identify the received voice data according to the terminal intention recognition model to obtain a classification recognition result; In response to the classification recognition result satisfying a preset classification recognition condition, generating an intent type corresponding to the received voice data; Extracting parameters from the voice data received by the terminal according to the intent type to obtain parameter data; User intention information and a cancellation signal are generated according to the intention type and the parameter data, wherein the cancellation signal is used to interrupt the task corresponding to the voice data in the cloud.

4. The method according to claim 1, wherein Generating at least one piece of initial user intention information according to the received voice data includes: Performing intent extraction on the received voice data to obtain an intent keyword list; Performing complexity extraction on the intent keyword list according to pre-set complexity information to obtain a complex keyword set; Generating a complexity level corresponding to the received voice data according to the complex keyword set and the preset complexity information; generating a model identifier based on matching information of a preset large language model and a complexity level corresponding to the received speech data; The received speech data is input into a target large language model corresponding to the model identifier to obtain at least one initial user intention information.

5. The method according to claim 1, wherein The method further comprises: The acquired multimodal data information corresponding to the voice data is sent to the cloud, wherein the multimodal data information is used by the cloud to generate user intent information based on the at least one initial user intent information.

6. The method according to claim 1, wherein The checking of the operation list with the time-space dependency mark to obtain a checking result includes: Performing a correctness check on the operation list marked with time and space dependencies; In response to the operation list with the time and space dependency markings passing the correctness check, acquiring hardware performance status information; Performing a feasibility check on the operation list marked with time and space dependencies according to the hardware performance status information; In response to the operation list with the time-space dependency mark passing the feasibility check, performing a conflict check on the operation list with the time-space dependency mark according to the hardware performance status information; In response to the operation list with the time-space dependency mark passing the conflict check, performing parameter verification on the operation list with the time-space dependency mark; In response to the operation list with the time-space dependency mark passing the parameter check, preset information indicating that the check has been passed is determined as a check result.

7. A head-mounted display device comprising: at least one display screen for forming an image in front of the user's eyes; one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

8. An instruction processing system based on a multi-terminal architecture, comprising: The glasses are configured to recognize the voice data and synchronously send the voice data to the terminal and the cloud; In response to receiving or identifying user intent information, executing a preset operation corresponding to the user intent information; in response to identifying the user intent information, sending a cancellation signal to the terminal and the cloud, wherein the user intent information sent by the cloud is generated by the following steps: generating at least one initial user intention information based on the received voice data; Generating user intent information according to the at least one initial user intent information includes: Performing semantic conversion on the at least one initial user intention information to obtain converted at least one initial user intention information; generating a candidate atomic operation list based on the converted at least one initial user intention information and the glasses-side exclusive capability map; Adding time and space constraint tags to the candidate atomic operation list to obtain an operation list with time and space dependency tags; Testing the operation list with the time-space dependency mark to obtain a test result; generating feedback information in response to the inspection result indicating that the inspection has failed, and visualizing the feedback information; In response to the inspection result indicating that the inspection has been passed, generating user intention information according to the operation list with the spatiotemporal dependency markers; The terminal is configured to recognize the received voice data; in response to recognizing the user intention information, send the recognized user intention information to the glasses end; The cloud is configured to generate at least one initial user intention information based on the received voice data; generate user intention information based on the at least one initial user intention information, and send the generated user intention information to the glasses end.

Images