A voice interaction method for AI glasses

Abstract

This invention discloses a voice interaction method for AI glasses, belonging to the field of voice interaction for wearable devices. To address the problems of unclear boundaries of voice commands and semantic comprehension deviations caused by background noise, multiple speakers, and omission of short sentences in open environments, this invention achieves the technical effect of reducing false wake-ups, erroneous execution, and continuous dialogue interruption by using multi-microphone target speech separation, overlapping speaker state recognition, streaming endpoint detection, and identity-overlapping-endpoint-semantic consistency gating. It outputs interactive decisions such as execution, confirmation, ignoring, or temporary merging based on anomaly marking, interference state, semantic uniqueness, temporary storage conditions, action risk level, and consistency score.

Description

Technical Field

[0001] This invention relates to the field of voice interaction in wearable devices, and more particularly to a voice interaction method for AI glasses. Background Technology

[0002] As wearable interactive devices, AI glasses typically need to continuously receive voice commands from the wearer in open environments such as commuting, working, and walking outdoors. Because the device is worn close to the human head, the multi-microphone array can obtain some near-field sound source cues. However, in real-world use environments, there will still be traffic noise, keyboard sounds, wind noise, and the voices of people around. The target speaker and interfering speakers in the voice stream often alternate or overlap.

[0003] Existing voice interaction solutions mostly rely on wake words, endpoint detection, and speech recognition results for execution, lacking unified constraints on the target speaker's identity, overlapping speech states, endpoint duration, and contextual semantic completeness. When the wearer utters omitted short sentences, or when the voice of a non-wearer overlaps with the wearer's voice, the system is prone to misidentifying interfering speech as commands, or splitting the same continuous dialogue into unrelated segments, thus causing false wake-ups, incorrect execution, or interruption of continuous dialogue.

[0004] Furthermore, the actions performed by wearable devices are typically instantaneous and context-dependent, with varying levels of risk depending on the action. Without gating the recognition results by considering the device's current state and historical dialogue history, low-confidence recognition results may directly trigger actions such as message transmission, navigation switching, or external device control, impacting the reliability of voice control.

[0005] Therefore, there is a need for a voice interaction method for AI glasses that can overcome the shortcomings of the existing technologies mentioned above. Summary of the Invention

[0006] One objective of this invention is to propose a voice interaction method for AI glasses. Addressing the challenge of existing technologies simultaneously handling target speaker identity, overlapping speech, endpoint boundaries, and short sentence contextual semantics in open environments, this invention proposes a technical solution based on multi-microphone observation sequences, wearer target speech separation, overlapping speaker state classification, streaming endpoint detection, context graph parsing, and identity-overlap-endpoint-semantic consistency gating. This invention effectively reduces false wake-ups, erroneous executions, and interruptions in continuous dialogue.

[0007] This invention provides a voice interaction method for AI glasses, comprising: S1, acquiring multi-microphone speech streams from the AI ​​glasses, wearer registered voiceprints, microphone array spatial features, environmental noise features, historical dialogue states, and device action states, and generating a synchronous interaction observation sequence; S2, inputting the synchronous interaction observation sequence into a self-supervised speech encoder to obtain frame-level speech representations, and generating a target speech mask based on wearer registered voiceprints, voiceprint missing markers, and effective near-field direction markers, separating wearer target speech features, and generating non-target residual signals based on the target speech mask; S3, inputting wearer target speech features, non-target residual signals, and microphone array spatial features into an overlapping perception speaker. The state classifier outputs a speaker state sequence; S4, the wearer's target speech features and speaker state sequence are input into the streaming speech recognition network and endpoint detection branch to generate candidate instruction boundaries, candidate transcribed text, endpoint confidence, and boundary stability; S5, the output of S4, speaker state sequence, historical dialogue states, and device action states are input into the context graph, action risk levels are generated based on device action states, and consistency scores are calculated through the identity-overlap-endpoint-semantic consistency gating module. The interaction decision of execution, confirmation, ignoring, or temporary merging is output according to the decision priority of confirmation anomaly marker, interference state, semantic uniqueness, temporary storage conditions, action risk level, and consistency score.

[0008] Optionally, S1 includes:

[0009] The multi-microphone speech stream is framed according to the same clock index, and the channel energy, inter-channel phase difference, direction of arrival vector and noise spectrum envelope of each observation frame are recorded.

[0010] The wearer's registered voiceprint is written into the voiceprint vector, and a voiceprint missing marker is generated when the voiceprint template is missing or the quality score is less than the voiceprint quality threshold determined by the quantile value of the registered sample.

[0011] The historical dialogue state is written into a state record containing the previous round of intent, filled slots, and actions to be confirmed. The device action state is written into a state record containing the action type, currently executable action, action occupancy flag, action cooldown time, user confirmation history, and action undeletable flag.

[0012] The synchronous interactive observation sequence is synthesized from the channel energy, the inter-channel phase difference, the direction of arrival vector, the noise spectrum envelope, the voiceprint vector, the voiceprint missing marker, the historical dialogue state, and the device action state according to the frame number.

[0013] Optionally, S2 includes:

[0014] The self-supervised speech encoder encodes the speech frames in the synchronous interactive observation sequence and outputs the frame-level speech representation.

[0015] The frame-level speech representation and the voiceprint vector are used to calculate the voiceprint similarity, and the voiceprint similarity is set as the voiceprint missing reference value when the voiceprint missing marker is established.

[0016] The directional coherence is calculated by combining the angle field in the direction of arrival vector with the center angle and half-width of the preset wearer near-field direction sector. When the peak coherence is lower than the effective directional threshold or the directional estimation is invalid, the directional coherence is set to 0 and a spatial weak marker is generated.

[0017] The voiceprint similarity, the directional coherence, and the noise suppression coefficient determined by the noise spectrum envelope are input into the mask mapping layer to obtain the target speech mask with a value range of 0 to 1.

[0018] The target speech mask is multiplied frame by frame with the frame-level speech representation to obtain the wearer's target speech features;

[0019] The non-target residual signal is generated by multiplying the remaining mask component obtained by subtracting the target speech mask from 1, the frame-level speech representation, and the near-field direction component away from the wearer obtained by subtracting the directional coherence from 1. The wearer's target speech features and the non-target residual signal are then synchronously written into the target speech feature record.

[0020] Furthermore, the self-supervised speech encoder is trained through a masked speech frame prediction task and a multi-channel consistency prediction task.

[0021] The training samples include near-field speech of the wearer, far-field speech of the non-wearer, commuting noise, office noise, and outdoor wind noise;

[0022] During training, the voiceprint vector of the same wearer is used as a positive sample condition, and the voiceprint vectors of different speakers are used as negative sample conditions, so that the frame-level speech representation output by the self-supervised speech encoder retains wearer identity-related information and spatial orientation-related information.

[0023] Optionally, S3 includes:

[0024] Receive the non-target residual signal;

[0025] The overlapping perception speaker state classifier performs temporal modeling of the wearer's target speech features, the non-target residual signal, and the microphone array spatial features, and outputs the non-speech probability, wearer's speech probability, non-wearer's speech probability, and overlapping speech probability corresponding to each observation frame;

[0026] When the probability of the state ranked first in probability value is not less than the state confidence threshold determined by the quantile value of the labeled verification sample and the probability difference between different categories meets the state interval threshold, the frame-level state is determined according to the category ranked first in probability value.

[0027] The initial speech segments are first divided based on the speech activity detection results, the silence candidate interval, and the frame-level state;

[0028] When the probability of the state with the highest probability value is less than the state confidence threshold or the difference between probabilities of different categories does not meet the state interval threshold, an insufficient confidence state marker is generated, and a segment-level insufficient confidence marker is generated when the proportion of the insufficient confidence state marker in the initial speech segment reaches the segment-level confidence proportion threshold.

[0029] The wearer's target speaking probability, non-wearer interference probability, overlapping speech probability, and segment-level insufficient confidence marker for each initial speech segment are retained as inputs to the identity-overlap-endpoint-semantic consistency gating module.

[0030] Optionally, S4 includes:

[0031] The streaming speech recognition network uses the wearer's target speech features as acoustic input and the initial speech segment in the speaker's state sequence as the recognition unit and state constraint to generate N-best candidate transcribed text.

[0032] The endpoint detection branch outputs word end markers, sentence end markers, segment-level endpoint confidence and boundary stability for each candidate transcribed text. The boundary stability is calculated by converting the frame offset of the initial candidate boundary start and end frame offset of the same initial speech segment in the streaming update into a time offset after frame shift.

[0033] When the sentence end marker is established, the endpoint duration is not less than the endpoint duration threshold determined by the historical confirmed sample quantile value, and the segment-level endpoint confidence is not less than the endpoint confidence threshold, a preliminary candidate boundary is generated.

[0034] When the boundary stability of the preliminary candidate boundary is not less than the boundary stability threshold, the preliminary candidate boundary is determined as the candidate instruction boundary;

[0035] When the duration of the endpoint is less than the endpoint duration threshold or the confidence level of the segment-level endpoint is less than the endpoint confidence threshold, the corresponding speech segment is marked as a segment to be merged and a temporary storage and non-execution state is set, and no direct execution prohibition flag is generated;

[0036] When the boundary stability is less than the boundary stability threshold and the segment-level endpoint confidence is not less than the endpoint confidence threshold, the corresponding speech segment is marked as a candidate for confirmation, a direct execution prohibition flag is generated, and the speech segment is prohibited from entering the direct execution branch.

[0037] Optionally, S5 includes:

[0038] The context graph includes intent nodes, slot nodes, historical round nodes, device action nodes, and risk nodes;

[0039] The candidate transcribed text is mapped to intent slot candidates, and the omitted slots are filled in according to the historical dialogue state to obtain the context graph matching result;

[0040] When the context graph matching result is merged by equivalent representation, there exists an executable candidate action that meets the slot integrity threshold and is allowed to be executed by the device action state, a semantically unique tag is generated. The equivalent representation merging requires the candidate action to be mapped to the same device action node, the same target object, and the same necessary slot value.

[0041] When there is no executable candidate action that meets the conditions because the semantics do not match the device action, the slot integrity does not reach the slot integrity threshold, or the device action state does not allow execution, a no-executable candidate action flag is generated, and the output is confirmed, ignored, or temporarily stored and merged according to the wearer's target speaking probability, non-wearer interference probability, endpoint state, and slot fillable state.

[0042] The identity-overlap-endpoint-semantic consistency gating module calculates the consistency score based on the wearer's target speaking probability, the non-wearer's interference probability, the overlapping speech probability, the endpoint confidence, the boundary stability, the semantic uniqueness marker, the confirmation anomaly marker set, the scoring de-weighting anomaly marker set, and the action risk level, and outputs the interaction decision based on the direct execution prohibition marker, the decision priority, and the preset score interval in which the consistency score is located.

[0043] Furthermore, the identity-overlap-endpoint-semantic consistency gating module writes the wearer target speaking probability, the non-wearer interference probability, the overlapping speech probability, the endpoint confidence, the boundary stability, the normalized edit distance between N-best candidate transcribed texts, the slot integrity, the semantic uniqueness marker, the confirmation anomaly marker set, the score deweighting anomaly marker set, and the action risk level into the gating feature vector;

[0044] The set of confirmed anomaly markers includes at least the voiceprint missing marker, the segment-level insufficient confidence marker, the spatial weak marker, the model rollback successful marker, and the direct execution prohibited marker; the set of score deweighting anomaly markers includes at least the mask rollback marker, the no executable candidate action marker, and the boundary stability rollback marker.

[0045] When model validation fails and a stable version is switched to, the model rollback success flag is generated. When the mask mapping layer uses the default scene parameters, missing voiceprint reference values, or invalid direction substitution strategies to participate in the mask calculation, the mask rollback flag is generated. When the boundary stability is calculated using the rollback value from the previous round of boundary estimation or when there is no valid previous round of estimation, the boundary stability rollback flag is generated.

[0046] The consistency score is obtained by passing the gated feature vector through a linear weight layer and an interval pruning layer. The weights of the linear weight layer are obtained by training the confirmed sample set, and the interval pruning layer restricts the output to 0 to 1.

[0047] The decision priorities are as follows: when the model verification fails and the rollback is not successful, the output is ignored as a pre-failure protection; the confirmation is triggered by the direct execution prohibition mark; the confirmation is triggered by the confirmation anomaly mark except for the temporary storage and non-execution state; when the non-wearer interference probability reaches the interference judgment threshold and the wearer target speaking probability is less than the wearer judgment threshold, the semantic non-uniqueness is confirmed; the temporary storage and merging is triggered when the fragment or slot to be merged is to be merged; and the scoring interval is determined.

[0048] In the scoring interval determination, the execution interval is only applicable to actions with an action risk level of 1. When the action risk level is 1, the direct execution prohibition flag is not established, and the consistency score falls into the execution interval, the execution is output.

[0049] When the action risk level is 2 or 3, regardless of whether the consistency score reaches the execution interval, the direct execution branch will not be entered, and confirmation will be output when the consistency score reaches the lower limit of the consistency confirmation interval determined by the confirmation sample quantile value.

[0050] When the consistency score falls into the ignore range, output "ignore"; when it falls into the temporary range, output "temporary merge".

[0051] Furthermore, when the interaction decision is temporary merging, the wearer's target speaking probability, voiceprint version, conversation identifier, sentence end marker, segment to be merged marker, intent slot candidate and timestamp of the current voice segment are written into the continuous dialogue cache.

[0052] If the current speech segment and the next speech segment both satisfy the following conditions: the wearer's target speaking probability is not less than the wearer's judgment threshold, the voiceprint version and the conversation identifier are consistent, the time interval between the two speech segments is not greater than the merging time threshold, and the slot sets of the two speech segments are complementary, then the candidate transcribed text and slot sets of the two speech segments are merged into a continuous dialogue instruction, and the context graph is re-entered.

[0053] The complementary slot set means that the missing slots of two speech segments can be mutually supplemented and there are no conflicting values ​​for the same slot key;

[0054] When there are conflicting values ​​in the slot keys or when consecutive dialogues are cached for more than the cache retention time without being merged, output confirmation or ignore and record the reason for not merging.

[0055] Furthermore, the risk level of the action is determined based on the action type, action occupancy marker, user confirmation history, and action revocability marker in the device's action status.

[0056] When the action occupancy flag indicates that it cannot be preempted, the risk level of the action is raised to 3 and a flag indicating that there are no candidate actions to execute is generated, thus prohibiting direct automatic execution;

[0057] The "no executable candidate action" flag includes three categories: semantically unmatched device action, slot integrity not reaching the slot integrity threshold, and device action status not allowing execution.

[0058] For actions that only involve local queries or volume adjustment and whose action occupancy marker does not indicate that they are not preemptible, set the lower limit of the execution interval for risk level 1; for actions that involve information sending, payment, navigation switching, or external device control, set the lower limit of the confirmation interval for risk level 3 and prohibit direct automatic execution.

[0059] When the semantic uniqueness marker is not valid, the output confirmation is given according to the decision priority. When the non-wearer interference probability is not less than the interference judgment threshold and the wearer target speaking probability is less than the wearer judgment threshold, the output ignore is given. When the non-wearer interference probability is not less than the interference judgment threshold and the wearer target speaking probability is not less than the wearer judgment threshold, the output confirmation is given.

[0060] The beneficial effects of this invention are:

[0061] 1. By using the wearer's registered voiceprint, near-field directional features, and environmental noise features together to generate the target speech mask, it is possible to form wearer-oriented target speech features in multi-microphone speech streams, making subsequent recognition and decision-making more reliant on the wearer's speech rather than environmental interference speech.

[0062] 2. By outputting non-speech, wearer speech, non-wearer speech, and wearer speech overlapping with others through the overlapping speaker state classifier, and inputting the wearer target speech probability, non-wearer interference probability, and overlapping speech probability into the gating module, the probability of non-wearer speech triggering commands can be reduced in multi-person speech and overlapping speech scenarios.

[0063] 3. By jointly calculating the consistency score based on endpoint confidence, boundary stability, N-best transcription difference, slot integrity, unique matching result of context graph, and action risk level, and outputting control results according to the decision priority of anomaly marking, interference state, semantic uniqueness, and temporary conditions, it is possible to balance action safety and dialogue continuity in short sentence omission and continuous dialogue scenarios. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a voice interaction method for AI glasses.

[0066] Figure 2 This is a flowchart of step S5 of the present invention, which is the identity-overlap-endpoint-semantic consistency gating module. Detailed Implementation

[0067] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0068] refer to Figures 1-2A voice interaction method for AI glasses includes: S1, acquiring multi-microphone speech streams from the AI ​​glasses, wearer registered voiceprints, microphone array spatial features, environmental noise features, historical dialogue states, and device action states to generate a synchronous interaction observation sequence; S2, inputting the synchronous interaction observation sequence into a self-supervised speech encoder to obtain frame-level speech representations, and generating a target speech mask based on the wearer registered voiceprint, voiceprint missing markers, and near-field direction valid markers, separating the wearer's target speech features, and generating a non-target residual signal based on the target speech mask; S3, inputting the wearer's target speech features, the non-target residual signal, and the microphone array spatial features into an overlap-sensing speaker state. S4. The classifier outputs a speaker state sequence; S5. The wearer's target speech features and the speaker state sequence are input into the streaming speech recognition network and endpoint detection branch to generate candidate instruction boundaries, candidate transcribed text, endpoint confidence, and boundary stability; S6. The output of S4, the speaker state sequence, the historical dialogue state, and the device action state are input into the context graph. An action risk level is generated based on the device action state, and a consistency score is calculated through the identity-overlap-endpoint-semantic consistency gating module. The interaction decision of execution, confirmation, ignoring, or temporary merging is output according to the decision priority of confirmation anomaly marker, interference state, semantic uniqueness, temporary storage conditions, action risk level, and consistency score.

[0069] In this specific embodiment, S1 includes:

[0070] The voice interaction controller of the AI ​​glasses establishes a synchronous acquisition session on a local low-power coprocessor, and the session identifier of the synchronous acquisition session is written into the interaction cache. The field, the multi-microphone array includes three pickup channels: the front of the left temple, the front of the right temple, and near the nose pad. The sampling rate is set to 16kHz, the frame length is set to 20ms, and the frame shift is set to 10ms. The controller reads the pulse code modulation sampling sequence of each channel, the voiceprint template in the wearer's registered voiceprint library, the microphone array installation geometry parameters, the noise spectrum envelope output by the environmental noise estimator, the previous round intent and filled slots saved by the history dialogue manager, and the currently executable actions and action occupancy flags saved by the device action manager, and writes the above objects into the observation frame table according to the same monotonically increasing frame number.

[0071] For the In each observation frame, the controller writes the short-time energy of the three channels. The phase difference between the left and right temple channels and between the nose pad channel and the left and right temple channels is written into the memory. The direction-of-arrival (DOA) record is obtained by inputting the array geometry parameters and phase difference into the generalized cross-correlation phase transformation rule. The rule calculates ,in For channel and channels In candidate directions Normalized cross-correlation value on, The channel-to-channel weights are given by the array calibration record. Including angle field Peak coherence field and direction valid fields ;

[0072] The noise estimator reads frames that are not marked as speech by the speech activity detector every 500ms sliding window and updates the noise spectrum envelope using an exponential smoothing method. ,in This is the median value of the frequency band energy in the current window. The noise spectrum envelope of the previous version is now updated. The unit is the normalized power spectral amplitude; when the speech activity detector does not provide a non-speech frame within a window, the controller continues to use... As And Write it into the observation frame table so that subsequent mask calculations can identify the noise envelope as a preserved value;

[0073] The wearer's registered voiceprint is generated from three near-field speech samples collected during the registration phase. Each record in the voiceprint template table includes... 128-dimensional voiceprint vector The registration timestamp, valid token, and template version number; at runtime, the controller reads the valid token as true based on the currently worn account. And write it to the voiceprint vector field of each observation frame. If the voiceprint template table is missing a record for the current account, then set the voiceprint vector field to empty and... Write the information into the observation frame table. Subsequent steps will reduce the identity confidence weight instead of directly performing high-risk actions when this field is 0.

[0074] When the controller performs voiceprint field writing, it writes the wearer's registered voiceprint into the voiceprint vector. and for Synchronously write the template version, registered sampling channel, registered voice quality score, and validity period; the registered voice quality score is calculated as follows: ,in The average signal-to-noise ratio of the three registered speech segments is normalized to a value between 0 dB and 30 dB. The cluster radius of the three-segment voiceprint vectors is normalized to the 95th percentile radius of the registered samples. Templates with a quality threshold of 0.62 are not included in the automatic gating process, and the controller will... and Simultaneously, it is written into the observation frame table, and the voiceprint similarity is set to the voiceprint missing reference value in the subsequent S2; this low-quality branch shares the voiceprint missing marker with the template missing branch, so that S5 can incorporate it into the confirmation anomaly marker set instead of entering the direct execution path;

[0075] Historical dialogue status records include the intentions of the previous round. Slot set Actions to be confirmed and the end time of the previous round The device action status record includes the set of currently executable actions. Action occupancy marker Action cooldown time and action reversibility marker ,in Measured in milliseconds and updated every 100ms by the action manager. This indicates that the corresponding action is being executed or is in a non-preemptive phase; the controller copies the above history and action status to the status field of the observation frame table by frame number, so that the subsequent context graph can directly read the voice and action conditions at the same moment;

[0076] The controller performs an integrity check on each observation frame, checking whether all three audio channels have sampled data. Does the peak coherence field exist? Does it have a version number? Does the historical dialogue status include the end time of the previous round? Does the device action status include an action set? When a single channel experiences a short-term packet loss, the controller fills the gap for that channel with the average energy of the same channel from two adjacent frames. And Write frame records when two or more channels lose packets or When the direction is less than the effective threshold, Set to 0 but still retain the voiceprint, noise, and status fields;

[0077] After completing the above processing, the controller uses the frame number Generate synchronous interactive observation sequences using primary keys ,in For channel energy vectors, The phase difference matrix between channels. For recording the status of historical dialogues, Record the device's operational status; the synchronous interactive observation sequence is written to the local circular buffer in chronological order. The cache length is set to the most recent 12 seconds of voice frames. After the cache is written, the start and end frame numbers, session identifier and cache version are passed to the voice encoding input manager in step S2.

[0078] In this specific embodiment, S2 includes:

[0079] Voice encoding input manager from Read the synchronous interactive observation sequence generated in step S1, and combine the frame-level spectral features, channel energy, inter-channel phase difference matrix, direction-of-arrival vector, and noise spectral envelope of the three speech channels into the model input tensor. The self-supervised speech encoder employs a 12-layer streaming Transformer coding structure, with each layer containing a multi-head self-attention network and a feedforward network. The streaming buffer length is set to 640ms on the left and 80ms on the right, and the encoder outputs a 256-dimensional frame-level speech representation. , Write along with frame number and session identifier surface;

[0080] Model input tensor Before entering the encoder, the signal undergoes amplitude normalization and channel alignment. Amplitude normalization is performed using... ,in For channel Median energy value in the last 2 seconds To and The zero-prevention amount of the same unit is set to 0.001; the channel alignment process uses the frame number of step S1 as the key to write the frame with missing direction estimation into... When the encoder is set to 0, it closes the directional attention branch while retaining the voiceprint and noise branches;

[0081] The encoder model object is loaded from the model manifest table, whose fields include model version, input tensor dimension, hidden layer dimension, streaming buffer length, quantization bit width, checksum hash, and rollback version; in this specific implementation, the version is loaded on the client side. The input dimension is a three-channel 80-dimensional Mel-spectrum plus spatial and noise features, with a quantization bit width of 8 bits; if the model verification hash does not match, the controller freezes the current model inventory version and writes it to the appropriate database. Then rollback to the previous stable version, and write the changes upon successful rollback. The current round of candidate instructions is limited to confirmed results; if rollback fails or no candidate instructions are found, the write operation will proceed. And restrict the output of this round to ignore the results;

[0082] The self-supervised speech encoder was trained before deployment using a masked speech frame prediction task and a multi-channel consistency prediction task. Training samples included near-field speech from the wearer, far-field speech from the non-wearer, commuting noise, office noise, and outdoor wind noise. The masked speech frame prediction task targeted randomly masked spectral frames for reconstruction, while the multi-channel consistency prediction task constrained the phase consistency of different microphone channels at the same time. The training loss was... ,in To mask frame reconstruction loss, For multi-channel consistency loss, For the voiceprint conditional contrast loss, the weights 0.5 and 0.2 are obtained by calibrating the endpoint stability of the validation set;

[0083] During encoder training, the voiceprint vector of the same wearer is used as the positive sample condition, and the voiceprint vectors of different speakers are used as the negative sample condition; the voiceprint conditional contrast loss uses... ,in For cosine similarity, For negative voiceprint samples from non-wearers, temperature parameters With the training set grid search set to 0.07, this loss makes Retain the wearer's identity information and spatial orientation information;

[0084] The training set is divided according to the wearer's identifier. Samples from any wearer do not appear in the training subset and the validation subset at the same time. The validation subset is used to calibrate the voiceprint similarity threshold, directional sector half-width, and mask weight. When the proportion of non-wearer speech misidentified as wearer speech in the validation set exceeds the target upper limit, the training process adds the weight of non-wearer far-field negative samples and re-exports the model version. The exported model version is only written into the model list table after the endpoint stability and false wake-up verification indicators both meet the release threshold.

[0085] During the inference phase, the speech encoding input manager calculates voiceprint similarity. and will Clipping to 0 to 1 as the wearer's voiceprint matching component; when or At that time, the system will Set it to 0.5 and record the missing voiceprint marker in the gating feature, and simultaneously write... The masking layer uses missing voiceprint reference values ​​in mask calculation, ensuring that identity verification is not directly triggered by missing or low-quality templates; Write frame number and template version The table is available for both the mask mapping layer and step S3 to read.

[0086] Near-field directional features are recorded by the direction of arrival. Near-field sector recording with preset wearer Calculations show that Determined by both the factory microphone geometry calibration and the wearer's initial voice calibration, a sector record includes the center angle. Half width Lower limit of peak coherence Wearing posture category and version number; when and When, the directional coherence is calculated as ,in The minimum angular distance calculated based on a 360-degree surround; when or At that time, Set to 0 and write and This indicates that the mask mapping layer uses a direction invalidation substitution strategy to participate in the mask calculation;

[0087] The noise suppression coefficient is determined by the noise spectrum envelope. It is confirmed that the masking layer first calculates for each frequency band. ,in For the first The current noise spectrum envelope value of each frequency band. The data collected before deployment while wearing silent glasses. Each frequency band noise reference envelope The zero-prevention amount is 0.001 for the same unit; then it is weighted according to frequency band. Aggregate as scalar noise suppression coefficient ,each Determined by noise calibration samples and summed to 1; when At that time, the system will Multiply by 0.8 as a retention penalty;

[0088] The mask weight table records voiceprint similarity weight, directional coherence weight, noise suppression weight, input value range, clipping interval, applicable scenario, and version number. For each record, the scenario key is "office near-field," with weight fields of 0.45, 0.35, and 0.20, and a clipping interval of 0 to 1. When a scenario key is not matched, the mask mapping layer reads the default scenario record and... Write the target speech feature record. After reading the mark in step S5, increase the confirmation tendency without changing the feature separation output in step S2.

[0089] The masking layer will use voiceprint similarity directional coherence and noise suppression coefficient Input a linear gating function to obtain the target speech mask. The three weights are obtained by training the confirmed sample set to minimize the false validation loss and sum to 1. The value range is from 0 to 1. The larger the value, the more likely the current frame is to contain the wearer's target speech;

[0090] The speech separator multiplies the target speech mask with the frame-level speech representation frame by frame to obtain the wearer's target speech features. And generate non-target residual signals based on the residual channel representation after the target speech masking. ,in For the penalty conversion component in the near-field direction away from the wearer; the controller will and Synchronous write The table shows that within 300ms consecutively When both values ​​are less than 0.2, the system does not delete the speech segment, but instead marks it as a low-target speech segment and passes it to step S3, where it is combined with the overlap-aware speaker state classifier. Determine whether there is non-wearer's voice or overlapping voice.

[0091] In this specific embodiment, S3 includes:

[0092] Overlap-aware speaker state classifier from Read the wearer's target speech features Non-target residual signals And the microphone array spatial features written in step S1 and The classifier input vector is constructed as follows: ,in The target speech mask generated in step S2, Write to the speaker state inference cache in chronological order;

[0093] The classifier consists of a finite right-context temporal network and a frame-level fully connected output layer. The right context is set to 80ms, and the edge output latency does not exceed 100ms. It outputs probability vectors for four mutually exclusive states. ,in For non-speech probabilities, For the wearer's voice probability, The probability of voice input from a non-wearer. The probability vector is the probability of overlapping speech between the wearer and others; the probability vector is calibrated by temperature scaling, and the calibration temperature is obtained by minimizing the negative log-likelihood of the validation set containing commuting, office and outdoor wind noise, and the probability values ​​are all between 0 and 1 and the sum of the four terms is 1;

[0094] To ensure that the non-target residual signals and the classification correction rules originate from the same source, the classifier directly reads the data written in step S2. And calculate the residual scalar ,in For the frame-level speech representation in step S2, For the same unit, the zero-prevention quantity is 0.001. Write The field is used as the comparison object for the overlap state correction rule;

[0095] The state determination module determines the frame-level state when the probability of the state with the highest probability value is not less than 0.45 and the difference between the probabilities of the top two states is not less than 0.12. ;when and The difference is less than 0.08 and When the threshold is not less than 0.3, the frame state is adjusted from the wearer's voice to overlapping voice between the wearer and others; the thresholds of 0.08 and 0.3 are determined by the trade-off between the overlapping voice recall rate and false wake-up rate in the confirmed sample set, and are stored in [the relevant data]. table and Field;

[0096] The classifier further smooths the frame-level states with a minimum segment length of 200ms. If the number of consecutive frames in the same state is less than 20 and the adjacent states are consistent, then the short state segment is merged into the adjacent state; when the non-speech probability... When the probability threshold is not less than 0.7 and the duration reaches 300ms, the system marks the interval as a silent candidate interval. The 0.7 probability threshold and the 300ms duration are written into the endpoint detection parameter table and read by step S4. The parameter table records the scene key, probability threshold, duration threshold, sample source, version number and update time. The version is updated when the cumulative number of newly confirmed samples reaches 200.

[0097] When the probability output by the classifier is the first state whose probability is less than 0.45, the difference between the probabilities of the top two states is less than 0.12, or the input vector... When there are two or more consecutive invalid frames in different directions, the state determination module marks the corresponding frame as insufficient confidence and writes it into the system. Frames with insufficient confidence do not participate in the direct state confirmation that ranks first in probability value sorting. Instead, they are included in the adjacent states with a weight of 0.5 during segment-level aggregation. If the proportion of frames with insufficient confidence in an initial speech segment is not less than 0.4, then a confirmation mark is added when the segment is passed to step S5.

[0098] The segment-level aggregation module first forms initial speech segments based on the speech activity detection results, silence candidate intervals, and frame-level state sequences, and simultaneously calculates the proportion of frames with insufficient confidence. ,in Within the initial speech segment Frame count, The initial total number of frames in the audio segment; when When the value is not less than 0.4, the system will use the initial speech segment... Write the speaker state sequence and add the flag to the exception flag set in step S5, so that the segment is given priority to enter the confirmation branch instead of being executed directly;

[0099] For each initial speech segment, the state aggregation module calculates the wearer's target speaking probability. Interference probability of non-wearers and overlapping speech probability ,in This refers to the number of frames within the initial speech segment. , and All are dimensionless probabilities output by the classifier softmax; divided by the number of frames in the same speech segment. The result obtained later , and It remains the average probability within the range of 0 to 1; the larger the value, the higher the proportion of the corresponding state in that range.

[0100] when Not less than 0.6 and When the value is less than 0.3, the state aggregation module marks the initial speech segment as a non-wearer interference segment and writes it into the system. Table; when Not less than 0.35 and If the threshold is not less than 0.25, the segment is marked as an overlapping speech segment instead of being directly discarded, for step S5 to determine whether to confirm or ignore it based on semantic uniqueness and action risk; the above threshold is determined by a multi-person speech sample set and labeled with a version number. storage;

[0101] In the output stage, the classifier will classify each frame... , , , and each initial speech segment , , The start and end frame numbers and status version are written into the speaker status sequence. ; for the initial speech segment frame level Calculation of the proportion ,when Time-based segment-level insufficient confidence markers And write it into the same initial speech segment record to avoid S5 directly using scattered frame-level low-confidence markers to trigger confirmation; this sequence is passed as a state constraint to the streaming speech recognition network in step S4, and at the same time... , , and As a gating feature, it is passed to the identity-overlap-endpoint-semantic consistency gating module in step S5.

[0102] In this specific embodiment, S4 includes:

[0103] The streaming speech recognition manager reads the wearer's target speech features output in step S2. and the speaker state sequence output in step S3 Constructing an acoustic input sequence based on the initial speech segment When the initial segment is marked as a non-wearer interference segment in step S3, the recognition manager still saves the segment identifier but does not send the segment to the execution candidate queue, but only uses it as evidence field for subsequent ignore decisions.

[0104] Acoustic input sequence Preserve the frame number and target speech features of each frame. Speaker status Probability of the wearer speaking and low confidence state flags The system then establishes a candidate segment index based on the segment's starting frame number. If the proportion of frames in a candidate segment that cannot be used for direction estimation exceeds 0.35, the recognition manager does not discard the segment but instead reduces the spatial constraint weight to 0.2 and writes it into the candidate instruction table. This provides step S5 with an increased likelihood of confirmation.

[0105] Before each initial speech segment enters the endpoint detection branch, the recognition manager uses... Initial speech segment identifier and start frame number generation parameter version lock record The record locks the endpoint parameter table, speaker threshold table, Tables: Boundary Stability Threshold Table, Gating Weight Table, Action Risk Table, and Slot Integrity Threshold Table; Step S4: Read Pointing to , The thresholds for state confidence, state interval, overlap correction, and boundary stability are as follows: user confirmation, cancellation, ignore, and temporary logs generated in the current period only enter the training or calibration queue of the next version and cannot reverse the threshold version currently being used by this initial speech segment.

[0106] The streaming speech recognition network employs a joint structure of connection-temporal classification and attention decoding, with the acoustic input being... The state constraints are frame-level states. Probability of speaking to the wearer's target The recognition network outputs an N-best set of candidate transcribed texts at each decoding time. In this specific implementation method The value is set to 5, and each candidate includes text, acoustic score, language model score, start and end frame numbers, and corresponding speaker segment identifier;

[0107] After the N-best candidate set is generated, the transcription difference calculator calculates the normalized edit distance between the first candidate and the remaining candidates. ,in For character-level edit distance, The denominator is the maximum of the sum of the lengths of the two candidate characters and 1, to ensure... Falling into 0 to 1; each Write The field, in subsequent step S5, reads the average difference. Determine if the candidate transcription is unique. The smaller the value, the more consistent the multiple candidates are;

[0108] To suppress mistranscription caused by overlapping speech, a state penalty term is added to the decoding score of each candidate in the recognition network. ,in To normalize the acoustic score, To normalize the language model score, and From step S3, the negative sign indicates that the higher the probability of interference from non-wearers and the probability of overlapping speech, the less suitable the candidate is to be executed directly; candidate pressing Sort from largest to smallest, and when there are ties, prioritize candidates with higher endpoint confidence.

[0109] The endpoint detection branch shares the first six layers of encoded representation with the recognition network, and outputs word end markers for each frame. Sentence end marker and endpoint confidence Endpoint confidence is determined by The calculation yielded, where The non-speech probability in step S3, Represents the endpoint trusted component after overlapping speech penalty. The larger the value, the more suitable the frame is as a candidate instruction boundary;

[0110] Endpoint duration threshold is determined by The endpoint parameter table records and historical confirmed sample quantile values ​​are determined. Each record in the endpoint parameter table includes the scene type, noise level, and endpoint duration threshold. Sentence end confidence threshold and version number; in this specific implementation, commuting scenario 450ms, suitable for office scenarios. 320ms, outdoor wind noise scenario The time is 520ms. The scene type is determined by the noise spectrum envelope in step S1 and the device motion state. The endpoint parameter table is stored in the local configuration area and updated after the user confirms that the cumulative number of samples reaches 200.

[0111] The endpoint parameter table is constructed using samples from user confirmation, user cancellation, and auto-ignore logs. Each sample record includes the noise scenario, endpoint silence duration, endpoint confidence, whether the identified text was confirmed, action risk level, and final interaction result. The parameter generator takes the 75th percentile of the silence duration of the user-confirmed sample as... The 20th percentile of the confidence level of the confirmed sample endpoints is taken as... And scenarios with fewer than 50 samples are mapped to default scenario records to prevent unstable thresholds in sparse scenarios;

[0112] End of sentence marker Establishment, Endpoint Duration Not smaller than the current scene And segment-level endpoint confidence Not less than At that time, the endpoint detection branch first generates preliminary candidate boundaries. ,in and These are the start and end frame numbers of the candidate segment, respectively. Confidence level of endpoints within this segment The mean; when or At this time, the system does not output the execution boundary, but instead writes the corresponding audio segment into the queue of segments to be merged. ,set up and No generation And not As a confirmation trigger field;

[0113] Endpoint detection branch in generation Post-calculation of boundary stability ,in and These represent the standard deviations of the offsets of the start and end frames of the three most recent valid preliminary candidate boundaries during the streaming update process for the same initial speech segment. The frame offsets are first converted to millisecond time offsets using a 10ms frame shift before being included in the calculation. The boundary jitter normalization baseline is set to 150ms, and the boundaries of different speech segments must not be mixed into the same... Calculation: When the historical preliminary candidate boundaries for the same initial speech segment are less than three times but there is a previous round-robin boundary estimate with a validity period of no more than 2500ms, the system calculates the backoff stability. ,in and These are the start and end time offsets estimated between the current boundary and the previous round-robin boundary, respectively, both in milliseconds. Then... Set as And write When there is no valid previous round-equal boundary estimate, Set to 0 and write ;when At that time, the identification manager will Determined as candidate instruction boundary The license field will be executed directly. Write to the candidate instruction table, when Less than 0.55 but achieve At that time, the recognition manager only retains the candidate transcribed text and writes the insufficient boundary stability mark into the candidate instruction table, while simultaneously... As a result of a failed execution branch due to instability, a direct execution prohibition flag is generated. Stop the voice segment from entering the direct execution branch and switch it to the candidate that needs to be confirmed, so that step S5 prioritizes the confirmation rather than the execution of the candidate;

[0114] For the segments to be merged, the recognition manager saves the candidate transcribed text, sentence end markers, word end markers, endpoint durations, endpoint confidence, candidate segment start and end frame numbers, and speaker segment identifiers, and maintains... as well as This state indicates that the speech segment has not yet formed an executable boundary. Unlike direct execution prohibition caused by low boundary stability, step S5 cannot proceed solely because the segment has not been written. The system will output confirmation; if the time interval between adjacent segments in the segment queue to be merged exceeds 1500ms or they come from different wearer target speaking probability states, the system will output the earlier segment as a candidate to be confirmed instead of executing it directly, in order to avoid short sentence omissions being incorrectly merged into irrelevant contexts;

[0115] After step S4 is completed, the recognition manager will select the candidate instruction boundary set that satisfies the stability condition. Candidate flags that do not meet stability requirements need to be verified, and the license field should be executed directly. Directly enforce the prohibition mark N-best candidate transcribed text set Segment-level endpoint confidence Boundary stability Fragments to be merged are marked Temporary waiting field Word end marker Sentence end marker and The actual locked versions of each parameter table are written into the candidate instruction table. The table is passed to the context graph parser in step S5 with the segment identifier as the primary key, while retaining the speaker state sequence reference, so that step S5 can continuously read the identity, overlapping state and the same lock version before and after semantic parsing.

[0116] In this specific embodiment, S5 includes:

[0117] The AI ​​glasses' on-device voice interaction controller performs the operation. The controller stores the context graph, gating weight table, action risk table, and continuous dialogue cache in a local secure storage area, and only outputs the gating control results to the action manager. This on-device execution structure enables the voice interaction method to complete the execution, confirmation, ignoring, or temporary merging judgment based on the wearer's identity, overlap state, endpoints, and semantic consistency even when the network is unavailable.

[0118] Before each candidate instruction record enters the context graph, the controller reads the candidate instruction table written in step S4. Parameter version lock record And verify the record. The initial speech segment identifier and starting frame number are consistent with the current candidate; step S5 only reads... Pointing to endpoint parameter table, speaker threshold table, gating weight table, action risk table, The table, slot integrity threshold table, and action cooldown status version, user confirmation, cancellation, ignore, and temporary logs generated in the current period are only written to the next version's training or calibration queue, and do not reversely change the endpoint threshold, gating weight, action risk level, or scoring range of this candidate instruction record, thereby ensuring that within the same candidate instruction record... , , , and Reproducible;

[0119] In this specific embodiment, the context graph parser reads the candidate instruction table written in step S4. And read the wearer's target speaking probability written in step S3. Interference probability of non-wearers Overlapping speech probability and the historical dialogue state written in step S1 and equipment operating status Context diagram Maintained in local memory, the nodes include intent nodes, slot nodes, historical round nodes, device action nodes, and risk nodes, and the edge fields include candidate text identifier, slot source, action constraint, risk level, latest update time, and confidence level.

[0120] The intent slot parser will transcribe the N-best candidate text set. Input a lightweight semantic encoder, output intent probability, slot label sequence, and slot position confidence; for candidate... The parser constructs the intended slot candidates. ,in The semantic parsing confidence score and slot completeness score are calculated as follows: , This represents the number of filled slots. The total number of slots required for this intent node; when At that time, the system will Set to 1 and record that this intent does not require a slot;

[0121] To handle short sentence omissions, the context graph parser reads the previous intent and the set of filled slots from the historical dialogue state, and fills in the omitted slots according to the rules of prioritizing the same intent, the most recent round, and the unexpired slots. The slot expiration time is determined by the slot type: 120 seconds for navigation destination slots, 60 seconds for contact slots, and 30 seconds for local query slots. If the candidate text only contains action words but lacks object slots, the parser only performs completion if the previous intent is the same as the current intent or has a preset inheritance edge; otherwise, the missing slot state is retained.

[0122] The context graph matching module will match each Mapping to a device action node only applies if the action node belongs to the currently executable action set. Action occupancy marker Not for unseizure, action cooldown time 0 and slot integrity Not less than the slot integrity threshold The candidate action is only retained when... When an action is indicated as non-preemptible, the system removes the candidate action from the executable set and writes it to... and This indicates that the device's action state does not allow execution, resulting in no executable candidate actions. The confirmed sample set is labeled according to intent category. In this specific embodiment, information sending is 1.0, navigation query is 0.8, and local volume adjustment is 0.6. The threshold record includes intent category, threshold, number of samples, and version number; when semantics do not match device action, slot completeness... Less than Or when the device's operating state does not allow execution, Set to 0 and write This allows the gating module to process branches by confirming, ignoring, or temporarily saving them.

[0123] The semantic uniqueness marker is determined by the number of retained candidate actions after merging equivalent representations and their relative semantic differences. The context graph parser first merges the retained candidate actions according to the device action node, target object key, necessary slot value, and execution parameters. Only candidates with the same action type, target object, and necessary slot value, and no difference in execution parameters, are grouped into the same action group. When only one action group that meets the slot integrity threshold exists after merging, it will be... Set to 1; when there are still more than two action groups after merging, the parser calculates the semantic interval of the top two action groups. ,even though The semantic gap threshold of 0.18 or higher will still apply if the target object key, necessary slot value, or execution parameters of the two action groups differ. Set to 0 and write the candidate action set to the confirmation prompt queue; only when multiple candidates are textual equivalents of the same action node and the same slot value will they be included as a candidate action in the subsequent scoring;

[0124] The risk level of an action is determined by the risk node based on the action type, action occupancy flag, user confirmation history, and action revocability flag in the device's action status. Confirmed; the risk node marks information sending, payment, navigation switching, and external device control as risk level 3, contact selection and route query as risk level 2, and local query and volume adjustment as risk level 1; when or At that time, the risk node will Directly upgrade to 3 and disable automatic execution; if If a user has revoked a similar action within the past 7 days, the risk level increases by 1 and is then reduced to between 1 and 3; the risk level record is written to... For this field, a larger value indicates that more confirmation is required before execution.

[0125] The action risk table provides scoring ranges for different risk levels, and sets a lower limit for the execution range of risk level 1 for actions that only involve local queries or volume adjustment. In this specific implementation, the lower limit of the execution range for risk level 1 is 0.62, the lower limit of the confirmation range is 0.42, and the upper limit of the ignore range is 0.25. For actions involving information sending, payment, navigation switching, or external device control, a lower limit of the confirmation range of risk level 3 of 0.55 is set and direct automatic execution is prohibited. Actions of risk level 2 and 3 will not enter the direct execution branch even if the consistency score reaches the execution range of risk level 1. Each record in the action risk table also includes action type, revocable flag, number of user confirmation samples, and version number.

[0126] The identity-overlap-endpoint-semantic consistency gating module will , , Segment-level endpoint confidence Boundary stability Edit distance between N-best candidate transcribed texts Slot integrity Semantic uniqueness marker Confirmation of the set of exception markers Set of anomaly markers for score deweighting Action risk level and directly enforce the prohibition mark Write gated feature vector ; include or , , , as well as and , include , , and ; This is generated when model validation fails and a stable version is switched to. When using default scene parameters, missing voiceprint reference values, or invalid direction substitution strategies to participate in mask calculation, the mask is generated. When calculating boundary stability using the previous round of boundary estimation or the backoff value when there is no valid previous round of estimation, the following is generated: ;

[0127] Gating module calculates consistency score ,in Set of anomaly markers for score deweighting The pruning count is established with a value ranging from 0 to 1. Positive items represent the benefit component that supports execution, while negative items represent interference, overlap, action risk, and weighting anomaly penalty components. The weights are obtained from training on the confirmed sample set and stored in the gating weight table. The pruning interval is 0 to 1. For dimensionless consistency scoring, risk level 1 can reach the execution range under ideal consistency conditions, while risk levels 2 and 3 are restricted to confirmation rather than direct automatic execution by the action risk table.

[0128] The gating weight table is trained using the confirmation sample set. Each record in the confirmation sample set includes the candidate speech segment identifier, whether the user confirmed the execution, whether the user canceled the action, the action risk level, the final action result, and the corresponding gating feature vector. The training process uses logistic regression as the objective to optimize the performance of the confirmed execution samples. The weights are higher than those of canceled or ignored samples, and a monotonicity check is performed after each weight update. , and When increasing alone No increase required , , , and When increasing alone It does not decrease;

[0129] The gating module outputs interactive decisions according to a fixed priority: when and When the model is unavailable, the output is ignored as a pre-failure protection measure; this branch precedes the comparison of interference and confirmation anomalies in the normal speech segment. or( and When ), output confirmation, and do not enter the scoring execution branch; when Output confirmation if any of the confirmation exception flags is true; otherwise, output confirmation when... and The output is ignored when the time is right; otherwise, when and( or or or Output is temporarily stored and merged when ), otherwise when and and The output is ignored when the time is right; otherwise, when or Output confirmation immediately; otherwise, proceed to the scoring range for judgment.

[0130] In the scoring interval determination, the execution interval is only applicable to... Actions; if , , and Execute the output; if ,regardless Whether or not the value reaches 0.62, the direct execution branch is not entered, and... Output confirmation upon request; if The output is ignored; if If the candidate segment has a sentence end marker, then output confirmation; if If the candidate segment does not have a sentence end marker, the output is temporarily stored and merged; if Output confirmation, thus overriding In the entire range from 0 to 1;

[0131] When the wearer's target speaking probability Not less than the wearer determination threshold of 0.55 and the endpoint confidence level Less than the current scenario endpoint threshold, the current candidate instruction record has , or slot integrity Smaller than the corresponding intention When, the gating module outputs are temporarily stored and merged; when At that time, low endpoint confidence, or Temporary merging should not be triggered independently; instead, confirmation or ignore should be output based on the non-wearer interference branch or scoring interval. Temporary merging results should not trigger device actions; the system will process candidate transcribed text, slot sets, and... Voiceprint version, session identifier , , Write timestamps to the continuous conversation cache The cache retention time is set to 2500ms;

[0132] When the next voice segment enters the gating module, the continuous dialogue manager compares the wearer's target speaking probability, voiceprint version, session identifier, time interval, and slot set of the two segments. If the current voice segment and the next voice segment... If the values ​​of the two voice segments are all not less than 0.55, the voiceprint version is consistent with the session identifier, the time interval between the two segments is not greater than the merging time threshold of 1500ms, and the slot sets of the two voice segments are complementary, then the candidate transcribed texts of the two voice segments will be concatenated by timestamp, and the slot sets will be merged into a continuous dialogue command. The slot complementarity determination requires that the missing slots of two voice segments can be mutually supplemented and that no conflicting values ​​appear in the same slot key. If a conflicting value appears, an acknowledgment is output instead of execution.

[0133] Continuous dialogue instructions Re-enter the context graph resolver and update the historical round nodes; the updated candidates then recalculate the consistency score. Action risk level If the execution conditions are met after re-evaluation and the device action node is still in an executable state, the interaction controller outputs an execution command to the action manager; if the continuous dialogue buffer exceeds 2500ms without merging, then... Output confirmation and retain the candidate action set in real time. The timeout reason is ignored and recorded in the output; the confirmation is the interactive decision output of this method. After the confirmation, the user agrees, refuses or the timeout is processed by the action manager after the output of this method.

[0134] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

[0135] This invention utilizes a continuous processing chain between multi-microphone acquisition, target speech masking, overlapping speaker states, candidate endpoints, and context graphs to gradually transform the original speech stream into an interactive decision with constraints on identity credibility, endpoint credibility, and semantic integrity. This forms an executable technical processing procedure to address the problems of unclear speech boundaries and semantic comprehension biases in open environments.

[0136] This invention sets up an identity-overlap-endpoint-semantic consistency gating module before the recognition result enters the execution control, and introduces the action risk level and anomaly mark into the decision priority, so that actions with risk level 2 or 3 enter the confirmation path instead of being executed automatically directly; at the same time, by merging the voice segments of the same wearer, the same conversation and complementary slots through continuous dialogue caching, the omitted short sentences can return to the same context graph for re-parsing, thereby better ensuring the reliability of wearable voice control.

[0137] In the gating implementation, the controller generates an initial speech segment-level version lock before the initial speech segment enters endpoint detection and context graph parsing. The same lock version is used for the endpoint parameter table, speaker threshold table, overlap threshold table, boundary stability threshold table, gating weight table, action risk table, and slot integrity threshold table. The current candidate endpoint boundary, action risk, and consistency score only read the lock version. Missing or low-quality speaker templates, insufficient segment-level speaker confidence, weak space, model rollback, and direct execution prohibition are all included in the confirmation anomaly flag set. Action occupancy blocking, mask backoff, no executable candidate actions, and boundary stability backoff are included in the score deweighting anomaly flag set, thereby preventing low-stability boundaries or non-preemptible actions from bypassing confirmation and being executed directly.

Claims

1. A voice interaction method for AI glasses, characterized in that, include: S1. Collect multi-microphone speech streams from AI glasses, wearer's registered voiceprint, microphone array spatial features, environmental noise features, historical dialogue states, and device action states to generate a synchronous interactive observation sequence; S2. Input the synchronous interactive observation sequence into a self-supervised speech encoder to obtain frame-level speech representations. Generate a target speech mask based on wearer's registered voiceprint, voiceprint missing markers, and effective near-field direction markers. Separate wearer's target speech features and generate non-target residual signals based on the target speech mask; S3. Input the wearer's target speech features, non-target residual signals, and microphone array spatial features into the overlap-aware speaker state classifier, and output the speaker state sequence; S4. Input the wearer's target speech features and speaker state sequence into the streaming speech recognition network and endpoint detection branch, and generate candidate instruction boundaries, candidate transcribed text, endpoint confidence, and boundary stability; S5. Input the output of S4, speaker state sequence, historical dialogue states, and device action states into the context graph, generate action risk levels based on device action states, calculate consistency scores through the identity-overlap-endpoint-semantic consistency gating module, and output interactive decisions such as execute, confirm, ignore, or temporarily merge according to the decision priority of confirmation anomaly marker, interference state, semantic uniqueness, temporary storage conditions, action risk level, and consistency score.

2. The voice interaction method for AI glasses according to claim 1, characterized in that, S1 includes: The multi-microphone speech stream is framed according to the same clock index, and the channel energy, inter-channel phase difference, direction of arrival vector and noise spectrum envelope of each observation frame are recorded; the wearer's registered voiceprint is written into the voiceprint vector, and the voiceprint missing marker is generated when the voiceprint template is missing or the quality score is less than the voiceprint quality threshold determined by the registered sample quantile value; The historical dialogue state is written into a state record containing the previous round's intent, filled slots, and actions to be confirmed. The device action state is written into a state record containing the action type, currently executable action, action occupancy flag, action cooldown time, user confirmation history, and action revocability flag. The channel energy, inter-channel phase difference, direction of arrival vector, noise spectrum envelope, voiceprint vector, voiceprint missing flag, historical dialogue state, and device action state are combined into the synchronous interactive observation sequence according to the frame number.

3. The voice interaction method for AI glasses according to claim 1, characterized in that, S2 includes: The self-supervised speech encoder encodes the speech frames in the synchronous interactive observation sequence and outputs the frame-level speech representation. The frame-level speech representation and the voiceprint vector are used to calculate the voiceprint similarity, and the voiceprint similarity is set as the voiceprint missing reference value when the voiceprint missing marker is established. The directional coherence is calculated by combining the angle field in the direction of arrival vector with the center angle and half-width of the preset wearer near-field direction sector. When the peak coherence is lower than the effective directional threshold or the directional estimation is invalid, the directional coherence is set to 0 and a spatial weak marker is generated. The voiceprint similarity, the directional coherence, and the noise suppression coefficient determined by the noise spectrum envelope are input into the mask mapping layer to obtain the target speech mask with a value range of 0 to 1. The target speech mask is multiplied frame by frame with the frame-level speech representation to obtain the wearer's target speech features; The non-target residual signal is generated by multiplying the remaining mask component obtained by subtracting the target speech mask from 1, the frame-level speech representation, and the near-field direction component away from the wearer obtained by subtracting the directional coherence from 1. The wearer's target speech features and the non-target residual signal are then synchronously written into the target speech feature record.

4. The voice interaction method for AI glasses according to claim 1, characterized in that, S3 includes: receiving the non-target residual signal; the overlap-aware speaker state classifier performing temporal modeling on the wearer target speech features, the non-target residual signal, and the microphone array spatial features, outputting the non-speech probability, wearer speech probability, non-wearer speech probability, and overlapped speech probability corresponding to each observation frame; when the state probability ranked first by probability value is not less than the state confidence threshold determined by the quantile value of the labeled verification sample and the probability difference between different categories meets the state interval threshold, determining the frame-level state according to the category ranked first by probability value; and based on the speech activity detection results... The initial speech segments are first divided into silent candidate intervals and frame-level states. When the probability of the state with the highest probability value is less than the state confidence threshold or the difference between probabilities of different categories does not meet the state interval threshold, an insufficient confidence state marker is generated. When the proportion of the insufficient confidence state markers in the initial speech segments reaches the segment-level confidence proportion threshold, a segment-level insufficient confidence marker is generated. The wearer target speaking probability, non-wearer interference probability, overlapping speech probability, and segment-level insufficient confidence markers of each initial speech segment are retained as inputs to the identity-overlap-endpoint-semantic consistency gating module.

5. The voice interaction method for AI glasses according to claim 1, characterized in that, S4 includes: The streaming speech recognition network uses the wearer's target speech features as acoustic input and the initial speech segment in the speaker's state sequence as the recognition unit and state constraint to generate N-best candidate transcribed texts. The endpoint detection branch outputs word end markers, sentence end markers, segment-level endpoint confidence, and boundary stability for each candidate transcribed text. The boundary stability is calculated by converting the frame shift of the initial candidate boundary start and end frame offset of the same initial speech segment in the streaming update into a time offset. When the sentence end marker is valid, the endpoint duration is not less than the endpoint duration threshold determined by the historical confirmed sample quantile value, and the segment-level endpoint confidence is not less than the endpoint confidence threshold, an initial candidate boundary is generated. When the boundary stability of the preliminary candidate boundary is not less than the boundary stability threshold, the preliminary candidate boundary is determined as the candidate instruction boundary; When the duration of the endpoint is less than the endpoint duration threshold or the confidence level of the segment-level endpoint is less than the endpoint confidence threshold, the corresponding speech segment is marked as a segment to be merged and a temporary storage and non-execution state is set, and no direct execution prohibition flag is generated; When the boundary stability is less than the boundary stability threshold and the segment-level endpoint confidence is not less than the endpoint confidence threshold, the corresponding speech segment is marked as a candidate for confirmation, a direct execution prohibition flag is generated, and the speech segment is prohibited from entering the direct execution branch.

6. The voice interaction method for AI glasses according to claim 1, characterized in that, S5 include: The context graph includes intent nodes, slot nodes, historical round nodes, device action nodes, and risk nodes; The candidate transcribed text is mapped to intent slot candidates, and the omitted slots are completed according to the historical dialogue state to obtain the context graph matching result; when there is an executable candidate action that meets the slot completeness threshold and is allowed to be executed by the device action state after the context graph matching result is merged by equivalent representation, a semantically unique tag is generated. The equivalent representation merging requires the candidate action to be mapped to the same device action node, the same target object, and the same necessary slot value. When there is no executable candidate action that meets the conditions because the semantics do not match the device action, the slot integrity does not reach the slot integrity threshold, or the device action state does not allow execution, a no-executable-candidate-action flag is generated, and the confirmation, ignore, or temporary storage and merging is output based on the wearer's target speaking probability, the non-wearer's interference probability, the endpoint state, and the slot's fillable state. The identity-overlap-endpoint-semantic consistency gating module calculates the consistency score based on the wearer's target speaking probability, the non-wearer's interference probability, the overlapping speech probability, the endpoint confidence, the boundary stability, the semantic uniqueness flag, the confirmation anomaly flag set, the score reduction anomaly flag set, and the action risk level, and outputs the interaction decision based on the direct execution prohibition flag, the decision priority, and the preset score interval where the consistency score is located.

7. The voice interaction method for AI glasses according to claim 3, characterized in that, The self-supervised speech encoder is trained through masked speech frame prediction and multi-channel consistency prediction tasks; the training samples include wearer near-field speech, non-wearer far-field speech, commuting noise, office noise and outdoor wind noise. During training, the voiceprint vector of the same wearer is used as a positive sample condition, and the voiceprint vectors of different speakers are used as negative sample conditions, so that the frame-level speech representation output by the self-supervised speech encoder retains wearer identity-related information and spatial orientation-related information.

8. The voice interaction method for AI glasses according to claim 6, characterized in that, The identity-overlap-endpoint-semantic consistency gating module writes the wearer target speaking probability, the non-wearer interference probability, the overlapping speech probability, the endpoint confidence, the boundary stability, the normalized edit distance between N-best candidate transcribed texts, the slot integrity, the semantic uniqueness marker, the confirmation anomaly marker set, the score deweighting anomaly marker set, and the action risk level into the gating feature vector; The set of confirmed anomaly markers includes at least the voiceprint missing marker, the segment-level insufficient confidence marker, the spatial weak marker, the model rollback successful marker, and the direct execution prohibited marker; the set of score deweighting anomaly markers includes at least the mask rollback marker, the no executable candidate action marker, and the boundary stability rollback marker. When model validation fails and a stable version is switched to, the model rollback success flag is generated. When the mask mapping layer uses the default scene parameters, missing voiceprint reference values, or invalid direction substitution strategies to participate in the mask calculation, the mask rollback flag is generated. When the boundary stability is calculated using the rollback value from the previous round of boundary estimation or when there is no valid previous round of estimation, the boundary stability rollback flag is generated. The consistency score is obtained by passing the gated feature vector through a linear weight layer and an interval pruning layer. The weights of the linear weight layer are obtained by training the confirmed sample set. The interval pruning layer restricts the output to 0 to 1. The decision priority includes, in order: ignoring the output as a pre-failure protection when the model verification fails and rollback is not successful; directly executing the prohibition mark to trigger confirmation; triggering confirmation with a confirmation anomaly mark other than the temporary storage and non-execution state; ignoring when the non-wearer interference probability reaches the interference judgment threshold and the wearer's target speaking probability is less than the wearer's judgment threshold; confirming semantic non-uniqueness; triggering temporary storage merging when the segment or slot to be merged is to be merged; and determining the score interval. In the scoring interval determination, the execution interval is only applicable to actions with an action risk level of 1. When the action risk level is 1, the direct execution prohibition flag is not met, and the consistency score falls into the execution interval, the execution is output. When the action risk level is 2 or 3, regardless of whether the consistency score reaches the execution interval, the direct execution branch is not entered, and confirmation is output when the consistency score reaches the lower limit of the consistency confirmation interval determined by the confirmation sample quantile value. When the consistency score falls into the ignore interval, the ignore is output, and when it falls into the temporary storage interval, the temporary storage and merging is output.

9. The voice interaction method for AI glasses according to claim 8, characterized in that, When the interaction decision is temporary merging, the wearer's target speaking probability, voiceprint version, session identifier, sentence end marker, segment to be merged marker, intent slot candidate, and timestamp of the current speech segment are written into the continuous dialogue cache. If the current speech segment and the next speech segment both satisfy the following conditions: the wearer's target speaking probability is not less than the wearer's judgment threshold, the voiceprint version and session identifier are consistent, the time interval between the two speech segments is not greater than the merging time threshold, and the slot sets of the two speech segments are complementary, then the candidate transcribed text and slot sets of the two speech segments are merged into a continuous dialogue instruction, and the context graph is re-entered. The complementary slot set means that the missing slots of the two speech segments can complement each other and there are no conflicting values ​​for the same slot key. When there are conflicting values ​​for slot keys or the continuous dialogue cache exceeds the cache retention time without merging, an acknowledgment or ignore is output and the reason for not merging is recorded.

10. The voice interaction method for AI glasses according to claim 8, characterized in that, The action risk level is determined based on the action type, action occupancy flag, user confirmation history, and action revocability flag in the device action status. When the action occupancy flag indicates that it cannot be preempted, the action risk level is increased to 3 and a "no executable candidate action" flag is generated, prohibiting direct automatic execution. The "no executable candidate action" flag includes three situations: semantically not matched device action, slot integrity not reaching the slot integrity threshold, and device action status not allowing execution. For actions involving only local queries or volume adjustment and where the action occupancy flag does not indicate that they cannot be preempted, a risk level 1 execution range lower limit is set. For actions involving information sending, payment, navigation switching, or external device control, a risk level 3 confirmation range lower limit is set, and direct automatic execution is prohibited. When the semantic uniqueness flag is not valid, confirmation is output according to the decision priority. When the non-wearer interference probability is not less than the interference judgment threshold and the wearer target speaking probability is less than the wearer judgment threshold, ignore is output. When the non-wearer interference probability is not less than the interference judgment threshold and the wearer target speaking probability is not less than the wearer judgment threshold, confirmation is output.

Images