Intelligent glasses interaction system based on speech recognition

Abstract

The application discloses an intelligent glasses interaction system based on speech recognition, comprising: a speech collection module for generating a speech signal sequence; a feature analysis module for generating a speech feature vector sequence; an intention construction module for generating a candidate semantic path set; a counterfactual path generation module for generating a counterfactual path set; an entropy flow calculation module for generating a path uncertainty entropy flow sequence; a path modulation module for generating a target semantic path; a behavior generation module for generating a device behavior sequence; and an execution feedback module for executing scheduling control and generating interaction feedback information. The application realizes multi-path semantic analysis, counterfactual semantic discrimination, target path screening and behavior feedback closed-loop control in the intelligent glasses speech interaction process, and improves the semantic recognition accuracy, interaction stability and device response reliability in a complex speech scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent voice interaction technology, and in particular to an intelligent glasses interaction system based on voice recognition. Background Technology

[0002] As an important form of wearable smart terminal, smart glasses are gradually acquiring functions such as voice interaction, information prompts, device control, scene-assisted recognition, and human-machine collaborative processing. Compared to touch control, voice interaction has advantages such as hands-free operation, direct response, and strong adaptability to mobile scenarios, and is therefore widely used in smart glasses. Users can use voice commands to access interfaces, query information, switch functions, control external devices, and trigger tasks, thereby improving ease of use and interaction efficiency.

[0003] Most existing smart glasses voice interaction solutions rely on direct mapping of speech recognition results. This involves first converting ambient speech into text or feature sequences, then outputting a single semantic result according to preset rules or classification models, and finally generating corresponding control commands based on that semantic result. While this approach can achieve basic interaction when the speech is clear, the semantics are explicit, and the environment is stable, in practical applications, environmental noise, sentence omissions, colloquial expressions, semantic ambiguity, and contextual changes can easily lead to recognition errors, causing the system to generate incorrect commands or output incomplete interaction results.

[0004] Meanwhile, existing technologies typically focus on single-path semantic recognition, lacking the ability to systematically model multiple potential semantic interpretations that the same voice input may correspond to, making it difficult to distinguish between multiple candidate semantic paths with fine granularity. Especially in complex scenarios, existing solutions lack the ability to characterize uncertain changes, cannot effectively constrain the degree of fluctuation in the evolution of semantic paths, and are also unable to stably filter candidate results by combining path order relationships, resulting in insufficient robustness of interaction results.

[0005] Therefore, it is necessary to propose a speech recognition-based smart glasses interaction system that performs feature analysis, candidate semantic path construction, counterfactual path generation, uncertainty entropy flow calculation, path modulation, and behavior feedback control on speech signals to improve the accuracy and stability of smart glasses interaction in complex speech scenarios. Summary of the Invention

[0006] One objective of this invention is to propose a speech recognition-based intelligent glasses interaction system. This invention fully integrates speech feature parsing, candidate semantic path construction, counterfactual path generation, path uncertainty entropy flow calculation, sequence modulation constraints, and execution feedback control mechanisms. It describes in detail the entire process of intelligent glasses interaction from speech acquisition, semantic parsing, multi-path discrimination, target path generation to behavior output and feedback correction. It has the advantages of high semantic recognition accuracy, strong ambiguity handling capability, good path selection stability, and strong adaptability to complex scene interaction.

[0007] According to an embodiment of the present invention, a smart glasses interaction system based on voice recognition includes:

[0008] The voice acquisition module is used to acquire environmental voice signals and generate voice signal sequences;

[0009] The feature parsing module is used to perform noise reduction and feature extraction operations on the speech signal sequence to generate a speech feature vector sequence;

[0010] The intent building module is used to perform semantic parsing operations at the corresponding positions in the speech feature vector sequence to generate a set of candidate semantic paths.

[0011] The counterfactual path generation module is used to generate a set of counterfactual paths based on the speech feature vector sequence, where each counterfactual path corresponds to a potential semantic interpretation;

[0012] The entropy flow calculation module is used to construct a path uncertainty entropy flow sequence at the corresponding position of the candidate semantic path set, and to calculate the entropy flow change rate at the path evolution position;

[0013] The path modulation module is used to generate an order modulation factor based on the entropy flow change rate at the path combination position, establish non-commutative operation rules at the path sorting position, and perform order constraint calculations on the candidate semantic path set to generate the target semantic path.

[0014] The behavior generation module is used to perform instruction reconstruction operations at the corresponding position of the target semantic path to generate a device behavior sequence;

[0015] The execution feedback module is used to perform scheduling control on the device behavior sequence and generate interactive feedback information.

[0016] Optionally, the generation of the speech feature vector sequence by the feature parsing module includes the following steps:

[0017] The system receives a sequence of audio signals, converts it into a digital signal stream, performs noise reduction processing, extracts the frequency component of the signal through wavelet transform, and removes background noise.

[0018] The denoised signal stream is divided into frames, which are of equal duration. Each frame contains a certain length of speech data and there is overlap between the frames.

[0019] Mel frequency cepstral coefficients are extracted from each frame of signal to extract speech features, including fundamental frequency, energy, pitch, and formants, and then standardized.

[0020] The frequency domain of each frame of signal is analyzed using Fast Fourier Transform to obtain frequency component information, and the first 20 frequency components are extracted.

[0021] The extracted frequency features are combined with time-domain features, and principal component analysis is used to reduce the dimensionality of the extracted features, resulting in a sequence of dimensionality-reduced speech feature vectors.

[0022] Optionally, the execution of semantic parsing by the intent construction module to generate a set of candidate semantic paths specifically includes:

[0023] The speech feature vector sequence is segmented, and a semantic segment sequence is generated at the segment position. Semantic identifier vector and context association identifier are extracted at each semantic segment position.

[0024] A set of semantic mapping rules is established at the corresponding position of the semantic fragment, and the semantic identifier vector is mapped to the basic semantic unit. During the mapping process, the semantic category identifier and semantic intensity value are written to generate a sequence of semantic units.

[0025] Establish sequence connection relationships at corresponding positions in the semantic unit sequence, perform connection calculations on adjacent semantic units, generate semantic connection paths at connection positions, and record path order identifiers and connection weight values ​​at path positions;

[0026] Introduce contextual constraints at the corresponding positions of semantic connection paths, perform constraint filtering calculations on semantic connection paths, eliminate connection paths that do not satisfy contextual consistency, and generate a set of candidate connection paths;

[0027] Perform path expansion calculations at the corresponding positions in the candidate connection path set, combine and expand several semantic connection paths, generate several complete semantic paths at the expansion positions, and perform set construction operations on several complete semantic paths at the corresponding positions to generate a complete semantic path set.

[0028] Perform path filtering calculations on the complete set of semantic paths, sort them according to path confidence values ​​and path structure completeness, and output a set of candidate semantic paths.

[0029] Optionally, the counterfactual path generation module specifically includes:

[0030] A semantic perturbation construction operation is performed on the speech feature vector sequence, a perturbation parameter sequence is generated at the feature index position, and a perturbation type identifier and a perturbation amplitude value are written at the corresponding position. The perturbation type identifier includes at least a semantic replacement identifier, a semantic deletion identifier, and a semantic insertion identifier. A perturbation semantic segment sequence is generated based on the perturbation parameter sequence.

[0031] Semantic remapping rules are established at the corresponding positions of the perturbation semantic fragment sequence to map the perturbation semantic fragments into perturbation semantic units. During the mapping process, semantic category identifiers and semantic association identifiers are written to generate a perturbation semantic unit sequence.

[0032] Establish counterfactual connections at corresponding positions in the perturbation semantic unit sequence, perform path connection calculations between perturbation semantic units, generate counterfactual connection paths at connection positions, and record path sequence identifiers, path branch identifiers, and connection weight values ​​at path positions.

[0033] At the corresponding position of the counterfactual connection path, perform path bifurcation extension calculation, perform combination construction operation on the perturbation parameter sequence at the path bifurcation position, combine the perturbation type identifier and the perturbation amplitude value according to the index position to generate a perturbation combination set, generate several path branches based on the perturbation combination set at the path bifurcation position, write the branch source identifier and the branch level identifier at the branch position, perform path extension operation on each path branch to generate several counterfactual semantic paths;

[0034] Path constraint calculations are performed at the corresponding positions of several counterfactual semantic paths. Semantic consistency constraints and contextual rationality constraints are introduced at the constraint positions. Screening is performed on each counterfactual semantic path to eliminate paths that do not meet the constraints and generate a set of candidate counterfactual paths.

[0035] Path feature calculation is performed at the corresponding position in the candidate counterfactual path set. Complexity calculation is performed at the path index position based on the path length, number of path branches, and connection weight values ​​in the counterfactual semantic path to generate the path structure complexity value.

[0036] At the path index position, perform difference calculation based on the perturbation semantic unit sequence and the semantic unit sequence to generate a semantic deviation value;

[0037] At the path index position, a path score is calculated based on the path disturbance amplitude, path structure complexity, and semantic deviation, generating a path score result. The path score result is then sorted, and a counterfactual path priority sequence is generated at the sorting position.

[0038] Path selection calculation is performed at the corresponding position of the counterfactual path priority sequence. The first few counterfactual semantic paths are selected according to a preset quantity threshold. Set construction operation is performed at the corresponding position to generate a counterfactual path set.

[0039] Optionally, the entropy flow calculation module specifically includes:

[0040] Extract the path probability parameters corresponding to each semantic path at the path index position of the candidate semantic path set, and perform normalization processing on the path probability parameters at the corresponding positions to generate a path probability sequence.

[0041] Extract the number of path branches and the path length of each semantic path from the path index position of the candidate semantic path set, and perform joint statistical calculations on the number of path branches and the path length at the corresponding positions to generate path structure distribution parameters.

[0042] Extract the path score results corresponding to each semantic path at the path index position in the candidate semantic path set, perform scale unification processing on the path score results at the corresponding position, and generate a path weight sequence;

[0043] At the path index position in the candidate semantic path set, entropy value calculation is performed based on the path probability sequence, path structure distribution parameters, and path weight sequence to generate a path uncertainty entropy value sequence;

[0044] Differential calculations are performed at the path index positions of the path uncertainty entropy value sequence according to the semantic path order identifier, and entropy value change sequences are generated at adjacent path index positions;

[0045] Normalization is performed at the path index position of the entropy change sequence, combined with the path sequence identifier, to generate the entropy flow change rate sequence.

[0046] Perform a sequence mapping operation at the path index position of the entropy flow rate of change sequence to map the entropy flow rate of change sequence to the corresponding position in the candidate semantic path set, generating a path uncertainty entropy flow sequence.

[0047] Optionally, the path modulation module specifically includes:

[0048] Extract the path uncertainty entropy flow sequence and path score result from the path index position of the candidate semantic path set. Perform interval division calculation on the entropy flow change rate value at the corresponding position to generate the entropy flow interval identifier sequence. Perform association mapping between the entropy flow interval identifier sequence and the path score result at the corresponding position to generate the path modulation parameter sequence.

[0049] At the path index position of the candidate semantic path set, the sequential modulation factor generation calculation is performed based on the path modulation parameter sequence. At the corresponding position, the sequential modulation factor value is generated according to the entropy flow interval identifier and the path score result. At the path index position, the sequential modulation factor value is subjected to directional constraint processing in combination with the path sequence identifier to generate a direction-distinguished modulation factor sequence.

[0050] Establish path combination relationships based on the direction-distinguished modulation factor sequence at the path index position of the candidate semantic path set. Perform path order constraint calculation at the combination position to generate path combination constraint results. Perform sorting calculation based on the path combination constraint results and path score results at the path index position to output the target semantic path.

[0051] Optionally, the behavior generation module generates a sequence of device behaviors, specifically including:

[0052] Extract semantic unit sequences and path order identifiers from the path index positions of the target semantic path, establish instruction mapping rule sets at the corresponding positions, map the semantic unit sequences to basic operation instruction units, write device function identifiers and operation type identifiers during the mapping process, and generate instruction unit sequences.

[0053] Extract operation parameter information from the unit index position of the instruction unit sequence, perform parameter parsing calculation at the corresponding position, convert the operation parameter information into a parameter value sequence, and perform combination construction operation with the instruction unit sequence at the corresponding position to generate an operation instruction sequence;

[0054] Extract instruction dependency identifiers from the instruction index positions of the operation instruction sequence, perform dependency resolution calculations at the corresponding positions, determine the sequence and constraint relationships between each operation instruction, and generate an instruction dependency structure sequence.

[0055] Sequential arrangement calculations are performed at the instruction index positions of the operation instruction sequence, and constraint processing is performed at the corresponding positions in combination with the path sequence identifier and the instruction dependency structure sequence to generate the device behavior sequence.

[0056] Optionally, the execution feedback module specifically includes:

[0057] Read each operation instruction at the instruction index position in the device behavior sequence, execute the device control instruction issuance calculation at the corresponding position, map the operation instruction into the device execution signal, and generate the device execution result sequence at the execution position;

[0058] The device status information is collected at the result index position of the device execution result sequence. At the corresponding position, the device status information is parsed and calculated to generate a status feature sequence. At the corresponding position, the status feature sequence is matched and calculated with the device behavior sequence to generate an execution consistency result.

[0059] Error calculation is performed at the result index position of the consistency result, and the difference between the target behavior and the actual execution result is calculated at the corresponding position to generate an execution deviation sequence. Then, a feedback adjustment parameter sequence is generated at the corresponding position based on the execution deviation sequence.

[0060] Feedback control calculations are performed at the parameter index positions of the feedback adjustment parameter sequence, scheduling correction operations are performed on the device behavior sequence at the corresponding positions, an updated device behavior sequence is generated, and interactive feedback information is generated at the corresponding positions.

[0061] The beneficial effects of this invention are:

[0062] (1) By constructing a candidate semantic path set, a counterfactual path set, and a path uncertainty entropy flow sequence, this invention enables the voice interaction process of smart glasses to move beyond a single recognition result output and instead perform parallel analysis and constraint screening of multiple potential semantic interpretations corresponding to the same voice input. With the synergistic effect of path scoring, entropy flow change rate, and sequence modulation factor, the recognition fluctuations caused by semantic ambiguity, sentence omission, colloquial expressions, and complex contexts can be refined, thereby improving the accuracy of target semantic path selection and enhancing the stability and reliability of interaction results in complex scenarios.

[0063] (2) This invention connects the target semantic path with the device behavior sequence generation process, and combines the execution result parsing, execution consistency calculation, execution deviation generation, and feedback adjustment parameter construction processes to form a closed-loop interactive mechanism from semantic understanding to behavior control and then to feedback correction. This structure can promptly detect the difference between the target behavior and the actual result during device execution, and schedule and correct the device behavior sequence, reducing the impact of erroneous instruction propagation and improving the response efficiency, control accuracy, and continuous interaction capability of smart glasses in dynamic environments. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0065] Figure 1 This is a flowchart of a smart glasses interaction system based on voice recognition proposed in this invention.

[0066] Figure 2 This is a schematic diagram illustrating the path exchange violation implementation process of a speech recognition-based smart glasses interaction system proposed in this invention.

[0067] Figure 3 This is a schematic diagram illustrating the path uncertainty entropy flow evolution process of a speech recognition-based smart glasses interaction system proposed in this invention. Detailed Implementation

[0068] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0069] refer to Figures 1-3 A smart glasses interaction system based on voice recognition, comprising:

[0070] The voice acquisition module is used to acquire environmental voice signals and generate voice signal sequences;

[0071] The feature parsing module is used to perform noise reduction and feature extraction operations on the speech signal sequence to generate a speech feature vector sequence;

[0072] The intent building module is used to perform semantic parsing operations at the corresponding positions in the speech feature vector sequence to generate a set of candidate semantic paths.

[0073] The counterfactual path generation module is used to generate a set of counterfactual paths based on the speech feature vector sequence, where each counterfactual path corresponds to a potential semantic interpretation;

[0074] The entropy flow calculation module is used to construct a path uncertainty entropy flow sequence at the corresponding position of the candidate semantic path set, and to calculate the entropy flow change rate at the path evolution position;

[0075] The path modulation module is used to generate an order modulation factor based on the entropy flow change rate at the path combination position, establish non-commutative operation rules at the path sorting position, and perform order constraint calculations on the candidate semantic path set to generate the target semantic path.

[0076] The behavior generation module is used to perform instruction reconstruction operations at the corresponding position of the target semantic path to generate a device behavior sequence;

[0077] The execution feedback module is used to perform scheduling control on the device behavior sequence and generate interactive feedback information.

[0078] In this embodiment, the feature parsing module generates a speech feature vector sequence by including the following steps:

[0079] The system receives a sequence of audio signals, converts it into a digital signal stream, performs noise reduction processing, extracts the frequency component of the signal through wavelet transform, and removes background noise.

[0080] The denoised signal stream is divided into frames, which are of equal duration. Each frame contains a certain length of speech data and there is overlap between the frames.

[0081] Mel frequency cepstral coefficients are extracted from each frame of signal to extract speech features, including fundamental frequency, energy, pitch, and formants, and then standardized.

[0082] The frequency domain of each frame of signal is analyzed using Fast Fourier Transform to obtain frequency component information, and the first 20 frequency components are extracted.

[0083] The extracted frequency features are combined with time-domain features, and principal component analysis is used to reduce the dimensionality of the extracted features, resulting in a sequence of dimensionality-reduced speech feature vectors.

[0084] In this embodiment, the execution of semantic parsing by the intent construction module to generate a candidate semantic path set specifically includes:

[0085] The speech feature vector sequence is segmented, and a semantic segment sequence is generated at the segment position. Semantic identifier vector and context association identifier are extracted at each semantic segment position.

[0086] A set of semantic mapping rules is established at the corresponding position of the semantic fragment, and the semantic identifier vector is mapped to the basic semantic unit. During the mapping process, the semantic category identifier and semantic intensity value are written to generate a sequence of semantic units.

[0087] Establish sequence connection relationships at corresponding positions in the semantic unit sequence, perform connection calculations on adjacent semantic units, generate semantic connection paths at connection positions, and record path order identifiers and connection weight values ​​at path positions;

[0088] Introduce contextual constraints at the corresponding positions of semantic connection paths, perform constraint filtering calculations on semantic connection paths, eliminate connection paths that do not satisfy contextual consistency, and generate a set of candidate connection paths;

[0089] Perform path expansion calculations at the corresponding positions in the candidate connection path set, combine and expand several semantic connection paths, generate several complete semantic paths at the expansion positions, and perform set construction operations on several complete semantic paths at the corresponding positions to generate a complete semantic path set.

[0090] Perform path filtering calculations on the complete set of semantic paths, sort them according to path confidence values ​​and path structure completeness, and output a set of candidate semantic paths.

[0091] Specifically, generating a complete set of semantic paths includes:

[0092] Each semantic connection path in the candidate connection path set is recursively concatenated according to the connectability relationship between the tail semantic unit and the head semantic unit. Paths that satisfy the context continuity constraint are expanded one by one to generate a complete semantic path set.

[0093] The execution path filtering calculation specifically includes:

[0094] The context consistency value, semantic integrity value, and path confidence value can be calculated for each semantic path in the complete semantic path set. Semantic paths that do not meet the preset threshold conditions can be removed. The remaining semantic paths are then sorted according to the comprehensive score results, and the semantic paths ranked in the top 3 to top 10 are selected as the candidate semantic path set.

[0095] In this embodiment, the counterfactual path generation module specifically includes:

[0096] A semantic perturbation construction operation is performed on the speech feature vector sequence, a perturbation parameter sequence is generated at the feature index position, and a perturbation type identifier and a perturbation amplitude value are written at the corresponding position. The perturbation type identifier includes at least a semantic replacement identifier, a semantic deletion identifier, and a semantic insertion identifier. A perturbation semantic segment sequence is generated based on the perturbation parameter sequence.

[0097] Semantic remapping rules are established at the corresponding positions of the perturbation semantic fragment sequence to map the perturbation semantic fragments into perturbation semantic units. During the mapping process, semantic category identifiers and semantic association identifiers are written to generate a perturbation semantic unit sequence.

[0098] Establish counterfactual connections at corresponding positions in the perturbation semantic unit sequence, perform path connection calculations between perturbation semantic units, generate counterfactual connection paths at connection positions, and record path sequence identifiers, path branch identifiers, and connection weight values ​​at path positions.

[0099] At the corresponding position of the counterfactual connection path, perform path bifurcation extension calculation, perform combination construction operation on the perturbation parameter sequence at the path bifurcation position, combine the perturbation type identifier and the perturbation amplitude value according to the index position to generate a perturbation combination set, generate several path branches based on the perturbation combination set at the path bifurcation position, write the branch source identifier and the branch level identifier at the branch position, perform path extension operation on each path branch to generate several counterfactual semantic paths;

[0100] Path constraint calculations are performed at the corresponding positions of several counterfactual semantic paths. Semantic consistency constraints and contextual rationality constraints are introduced at the constraint positions. Screening is performed on each counterfactual semantic path to eliminate paths that do not meet the constraints and generate a set of candidate counterfactual paths.

[0101] Path feature calculation is performed at the corresponding position in the candidate counterfactual path set. Complexity calculation is performed at the path index position based on the path length, number of path branches, and connection weight values ​​in the counterfactual semantic path to generate the path structure complexity value.

[0102] At the path index position, perform difference calculation based on the perturbation semantic unit sequence and the semantic unit sequence to generate a semantic deviation value;

[0103] At the path index position, a path score is calculated based on the path disturbance amplitude, path structure complexity, and semantic deviation, generating a path score result. The path score result is then sorted, and a counterfactual path priority sequence is generated at the sorting position.

[0104] Path selection calculation is performed at the corresponding position of the counterfactual path priority sequence. The first few counterfactual semantic paths are selected according to a preset quantity threshold. Set construction operation is performed at the corresponding position to generate a counterfactual path set.

[0105] Specifically, establishing semantic remapping rules at corresponding positions in the perturbed semantic segment sequence includes:

[0106] Read the perturbation type identifier, perturbation amplitude value, and original semantic fragment identifier corresponding to each perturbation semantic fragment. Perform classification processing on the perturbation semantic fragments according to the perturbation type identifier to form a set of semantic replacement fragments, a set of semantic deletion fragments, and a set of semantic insertion fragments. Record the fragment sequence identifier and the relationship between adjacent fragments at the corresponding position of each perturbation semantic fragment to form a perturbation fragment position index table.

[0107] A semantic candidate unit index table is established at the corresponding position of each perturbation semantic segment. The semantic category identifier, context association identifier, and position order identifier are written in the index table. The range of candidate semantic units corresponding to each perturbation semantic segment is determined by combining the semantic categories and connection relationships of adjacent segments before and after the perturbation semantic segment, so that each perturbation semantic segment corresponds to at least one set of optional semantic units.

[0108] Semantic matching calculations are performed on each perturbed semantic segment. The perturbed semantic segment is compared with the candidate semantic unit set item by item, and the semantic similarity value, contextual coherence value, and positional continuity value are calculated respectively. The target mapping unit is determined based on the above values, and the perturbed semantic unit is generated at the corresponding position. For semantic replacement segments, the original semantic unit is replaced with the target mapping unit. For semantic deletion segments, the original semantic unit is removed and the connection relationship between adjacent semantic units is reconstructed. For semantic insertion segments, the new semantic unit is inserted into the corresponding sequential position.

[0109] Semantic category identifiers and semantic association identifiers are written at the corresponding positions of each generated perturbation semantic unit. The order of all perturbation semantic units is sorted and the connection is corrected to form a perturbation semantic unit sequence. Consistency checks are performed on adjacent units in the perturbation semantic unit sequence, and the mapping results that meet the context connection conditions are retained as input for the subsequent construction of counterfactual connection relationships.

[0110] Specifically, establishing counterfactual connections at corresponding positions in the perturbation semantic unit sequence includes:

[0111] In the process of establishing counterfactual connections at corresponding positions in the perturbation semantic unit sequence, the semantic category identifier, semantic association identifier, position order identifier, and source perturbation type identifier corresponding to each perturbation semantic unit are first read. The perturbation semantic units are then sorted according to their position order to form an ordered perturbation semantic unit queue. Subsequently, an initial connection determination table is established between adjacent perturbation semantic units. Forward connection identifiers, backward connection identifiers, and cross-unit jump identifiers are written into the determination table to characterize whether there is a connection basis between each perturbation semantic unit.

[0112] For adjacent and spaced units in an ordered perturbation semantic unit queue, perform connection feasibility calculations, calculating the semantic connectivity, context transition, and sequential continuity values ​​respectively. When the above values ​​meet the preset connection conditions, write a connection validity flag at the corresponding position and establish a counterfactual connection edge between the corresponding units. When the perturbation type is a semantic deletion flag, perform cross-position connection calculations directly on the perturbation semantic units before and after the deletion position to generate a jump connection relationship. When the perturbation type is a semantic insertion flag, perform bidirectional connectivity judgments on the newly added perturbation semantic unit with the preceding and following units respectively to generate an insertion connection relationship. When the perturbation type is a semantic replacement flag, re-perform connection calculations on the replaced perturbation semantic unit with adjacent units to generate a replacement connection relationship.

[0113] After determining the validity of the connections, the path sequence identifier, path branch identifier, and connection weight value are written to the corresponding positions of each established counterfactual connection edge. The path sequence identifier is used to represent the order of the connection edges in the entire counterfactual semantic path, the path branch identifier is used to distinguish the connection branches generated by different perturbation methods, and the connection weight value is used to represent the credibility of the current connection relationship. Then, all connection edges are summarized and organized to form a counterfactual connection relationship table.

[0114] Based on the counterfactual connection table, connection correction processing is performed on the perturbed semantic unit sequence, conflict resolution calculation is performed on the connection edges with connection conflicts, connection relationships with high connection weight values ​​and satisfying the contextual rationality conditions are retained, and invalid connection relationships that do not meet the sequential continuity constraints or semantic connection constraints are eliminated, generating the final counterfactual connection relationship as the basis for subsequent counterfactual connection path construction and path branching expansion calculation.

[0115] Specifically, performing path fork expansion calculations at the corresponding positions of the counterfactual connection paths includes:

[0116] Read the path sequence identifier, path branch identifier, connection weight value and disturbance type identifier of each connection path at the corresponding position of the counterfactual connection path, and perform sorting processing on each counterfactual connection path according to the path sequence identifier to form an initial path queue.

[0117] Read the disturbance parameter sequence at the corresponding position of each counterfactual connection path, extract the disturbance type identifier and disturbance amplitude value related to the current connection path at the path bifurcation position, perform combination calculation according to the position index, generate a disturbance combination set corresponding to the current connection path, and establish a bifurcation candidate table at the corresponding position of each disturbance combination.

[0118] At the corresponding positions in each fork candidate table, write different perturbation combinations from the perturbation combination set into the current counterfactual connection path, perform branch replication calculation on the current counterfactual connection path to generate multiple path branches, and write the branch source identifier, branch level identifier, and branch inheritance identifier at the corresponding positions of each path branch.

[0119] At the corresponding positions of each path branch, path extension calculation is performed based on the connection weight value. The current path branch is concatenated with the subsequent connectable perturbation semantic units one by one. Path branches that meet the semantic connection conditions and contextual rationality conditions are retained, while path branches that do not meet the conditions are removed, generating several counterfactual semantic paths.

[0120] Specifically, performing path constraint calculations at the corresponding positions of several counterfactual semantic paths includes:

[0121] Read the semantic unit sequence, path order identifier, path branch identifier and connection weight value corresponding to each counterfactual semantic path, and establish a semantic consistency constraint table and a context rationality constraint table at the corresponding position of each path; then perform connection detection calculation on adjacent semantic units in each counterfactual semantic path to determine whether the semantic coherence relationship between the preceding and following semantic units is satisfied, and write constraint violation identifiers for paths that do not meet the semantic category connection conditions or semantic pointing conflict conditions.

[0122] For each counterfactual semantic path, a contextual rationality calculation is performed. Combining the sequential relationship, insertion position, deletion position, and replacement position of each semantic unit in the path, it is determined whether the path as a whole satisfies the conditions of statement structure continuity, semantic interpretation validity, and contextual transition stability. Counterfactual semantic paths that meet the constraints are retained, while counterfactual semantic paths with constraint violation flags and a number of violations exceeding a preset threshold are eliminated, generating a set of candidate counterfactual paths that meet the path constraints.

[0123] Specifically, performing path feature calculation at the corresponding position in the candidate counterfactual path set includes:

[0124] Read the path sequence identifier, path branch identifier, connection weight value, perturbation type identifier and perturbation amplitude value corresponding to each candidate counterfactual path, extract the number of semantic units, path fork number and path connection number contained in each candidate counterfactual path at the path index position, perform summary calculation on the number of semantic units, path fork number and cross-position connection number, and generate path length value, path branch number value and path structure complexity value.

[0125] Simultaneously, at the path index position, the perturbation semantic unit sequence corresponding to each candidate counterfactual path is compared with the original semantic unit sequence to calculate the corresponding difference.

[0126] At the path index position, the perturbation semantic unit sequence corresponding to each candidate counterfactual path is compared with the original semantic unit sequence one by one according to the semantic unit order, and the semantic difference calculation is performed to generate a semantic deviation value.

[0127] Generate semantic deviation values, and write the connection weight values, path structure complexity values, path length values, perturbation amplitude values, and semantic deviation values ​​into the corresponding path feature table to form the path feature calculation results for each candidate counterfactual path.

[0128] In this embodiment, performing the difference calculation specifically includes:

[0129] Read the perturbation semantic unit sequence and the original semantic unit sequence corresponding to the candidate counterfactual path, and establish a positional correspondence according to the semantic unit order; determine whether the semantic units in the two sequences are consistent at each corresponding position, record the replacement difference indicator for semantic replacement positions, record the missing difference indicator for semantic deletion positions, and record the new difference indicator for semantic insertion positions; perform difference statistics on the semantic category indicator, semantic association indicator and context connection relationship corresponding to each position to generate a position difference result sequence; then perform summary calculation on the position difference result sequence to generate the semantic deviation degree value corresponding to each candidate counterfactual path.

[0130] In this embodiment, the entropy flow calculation module specifically includes:

[0131] Extract the path probability parameters corresponding to each semantic path at the path index position of the candidate semantic path set, and perform normalization processing on the path probability parameters at the corresponding positions to generate a path probability sequence.

[0132] Extract the number of path branches and the path length of each semantic path from the path index position of the candidate semantic path set, and perform joint statistical calculations on the number of path branches and the path length at the corresponding positions to generate path structure distribution parameters.

[0133] Extract the path score results corresponding to each semantic path at the path index position in the candidate semantic path set, perform scale unification processing on the path score results at the corresponding position, and generate a path weight sequence;

[0134] At the path index position in the candidate semantic path set, entropy value calculation is performed based on the path probability sequence, path structure distribution parameters, and path weight sequence to generate a path uncertainty entropy value sequence;

[0135] Differential calculations are performed at the path index positions of the path uncertainty entropy value sequence according to the semantic path order identifier, and entropy value change sequences are generated at adjacent path index positions;

[0136] Normalization is performed at the path index position of the entropy change sequence, combined with the path sequence identifier, to generate the entropy flow change rate sequence.

[0137] Perform a sequence mapping operation at the path index position of the entropy flow rate of change sequence to map the entropy flow rate of change sequence to the corresponding position in the candidate semantic path set, generating a path uncertainty entropy flow sequence.

[0138] Specifically, performing joint statistical calculations on the number of path branches and the path length at the corresponding locations to generate path structure distribution parameters includes:

[0139] Read the number of path branches and the path length of each semantic path in the candidate semantic path set, build a path structure statistics table according to the path index position, and write the number of branches and the number of lengths of each semantic path into the path structure statistics table; then perform interval partitioning processing on the number of path branches and the path length of each semantic path, dividing the number of path branches into several branch level intervals and the path length into several length level intervals, and write the branch level identifier and the length level identifier at the corresponding position of each semantic path to form a path structure level sequence.

[0140] A joint distribution statistical table is established based on the combination relationship between branch level identifier and length level identifier. Combination and classification calculations are performed on each semantic path in the candidate semantic path set, and semantic paths with the same branch level identifier and length level identifier are grouped into the same statistical unit.

[0141] The number of semantic paths in each statistical unit is cumulatively calculated to generate the occurrence count value corresponding to each combination of branch level and length level. Then, the proportion of each occurrence count value is converted to generate the joint distribution proportion value corresponding to each combination position.

[0142] Perform parameter extraction calculations on the joint distribution ratio values, write the ratio components corresponding to each branch level interval, the ratio components corresponding to each length level interval, and the joint ratio components corresponding to each combined statistical unit into the parameter table to generate path structure distribution parameters. The path structure distribution parameters are used to characterize the distribution state of different path branch numbers and different path length combinations in the candidate semantic path set.

[0143] Specifically, performing sequence mapping operations at the path index position of the entropy flow change rate sequence includes:

[0144] Read the change rate values ​​corresponding to each path index position in the entropy flow change rate sequence and the path sequence identifiers corresponding to each semantic path in the candidate semantic path set, and establish an index correspondence according to the path sequence identifiers; write the entropy flow change rate values ​​into the attribute record items of the corresponding semantic path at each corresponding position to form a path change rate association table;

[0145] The path change rate association table is corrected by position matching. For missing index positions, the change rate values ​​of adjacent path index positions are used to fill in the missing index positions. For duplicate index positions, a unique change rate value is generated according to the preset merging rules. Finally, the corrected change rate values ​​are mapped one by one to the corresponding semantic path positions in the candidate semantic path set. The entropy flow identifier and change rate value are written in the corresponding positions to generate the path uncertainty entropy flow sequence.

[0146] In this embodiment, the path modulation module specifically includes:

[0147] Extract the path uncertainty entropy flow sequence and path score result from the path index position of the candidate semantic path set. Perform interval division calculation on the entropy flow change rate value at the corresponding position to generate the entropy flow interval identifier sequence. Perform association mapping between the entropy flow interval identifier sequence and the path score result at the corresponding position to generate the path modulation parameter sequence.

[0148] At the path index position of the candidate semantic path set, the sequential modulation factor generation calculation is performed based on the path modulation parameter sequence. At the corresponding position, the sequential modulation factor value is generated according to the entropy flow interval identifier and the path score result. At the path index position, the sequential modulation factor value is subjected to directional constraint processing in combination with the path sequence identifier to generate a direction-distinguished modulation factor sequence.

[0149] Establish path combination relationships based on the direction-distinguished modulation factor sequence at the path index position of the candidate semantic path set. Perform path order constraint calculation at the combination position to generate path combination constraint results. Perform sorting calculation based on the path combination constraint results and path score results at the path index position to output the target semantic path.

[0150] Specifically, the association mapping between the entropy flow interval identifier sequence and the path scoring result at the corresponding position includes:

[0151] Read the entropy flow interval identifier and path score result corresponding to each semantic path, and establish a corresponding relationship table according to the path index position;

[0152] At each path index position, the entropy flow interval identifier is written as the interval category item into the corresponding record unit, and the path score result is written as the score value item into the same record unit to form a path modulation association table.

[0153] Based on the preset modulation rules corresponding to different entropy flow interval identifiers, the path scoring results within the same interval are processed in a hierarchical manner to generate modulation level identifiers corresponding to each semantic path. Finally, the entropy flow interval identifier, path scoring results, and modulation level identifiers are written into the corresponding path parameter record to generate a path modulation parameter sequence.

[0154] Specifically, the calculation of sequential modulation factor generation based on the path modulation parameter sequence at the path index position in the candidate semantic path set includes:

[0155] Read the entropy flow interval identifier, path score result and modulation level identifier corresponding to each semantic path, and establish an order arrangement relationship according to the path order identifier;

[0156] At each path index position, the corresponding basic modulation coefficient range is determined according to the entropy flow interval identifier. The corresponding score correction value is determined according to the path score result. The basic modulation coefficient, score correction value and level adjustment value are written into the same path record item. The initial sequential modulation factor value corresponding to each semantic path is generated according to the preset calculation rules.

[0157] By combining the sequential relationship of each semantic path, the initial sequence modulation factor values ​​are corrected by order constraint calculation to generate the sequence modulation factor sequence corresponding to each semantic path.

[0158] Specifically, the process of performing directional constraint processing on the sequential modulation factor values ​​at the path index position in conjunction with the path sequence identifier to generate a direction-distinguished modulation factor sequence includes:

[0159] Read the path sequence identifier and initial sequence modulation factor value corresponding to each semantic path, and arrange each semantic path in the order of precedence according to the path sequence identifier to form an ordered path sequence; determine the sequential relationship between the current semantic path and the adjacent semantic paths at each path index position, mark the semantic path in the preceding position as the forward path, mark the semantic path in the following position as the backward path, and write the direction identifier at the corresponding position.

[0160] At each path index position, the initial sequential modulation factor value is corrected according to the direction identifier. Forward constraint value is written to the initial sequential modulation factor value corresponding to the forward path, and backward constraint value is written to the initial sequential modulation factor value corresponding to the backward path, generating the direction modulation result corresponding to each semantic path.

[0161] The modulation results in each direction are sorted according to the path sequence identifier to form a modulation factor sequence that distinguishes directions.

[0162] Specifically, establishing path combination relationships based on direction-distinguished modulation factor sequences at path index positions in the candidate semantic path set includes:

[0163] Read the modulation factor values, path order identifiers, and path scoring results corresponding to each semantic path, and arrange each semantic path in an ordered manner according to the path order identifier to form a path sorting table; record the adjacent relationships of each semantic path at the corresponding position in the path sorting table as the basis for subsequent path combination determination.

[0164] At each path index position, determine whether the sequential connection condition is met between adjacent semantic paths. Write a combined association identifier for semantic paths that meet the sequential connection condition, and write a combined exclusion identifier for semantic paths that do not meet the sequential connection condition.

[0165] Based on the modulation factor values ​​distinguished by the direction of each semantic path, the semantic paths with combination association identifiers are subjected to a sequential relationship determination to determine the preceding and following relationships between each semantic path and generate a path combination order table.

[0166] Write each path relationship in the path combination order table into the corresponding path record item, summarize and organize all semantic paths that meet the combination conditions to form a path combination relationship; the path combination relationship is used for subsequent path order constraint calculation.

[0167] In this embodiment, the behavior generation module specifically generates the device behavior sequence including:

[0168] Extract semantic unit sequences and path order identifiers from the path index positions of the target semantic path, establish instruction mapping rule sets at the corresponding positions, map the semantic unit sequences to basic operation instruction units, write device function identifiers and operation type identifiers during the mapping process, and generate instruction unit sequences.

[0169] Extract operation parameter information from the unit index position of the instruction unit sequence, perform parameter parsing calculation at the corresponding position, convert the operation parameter information into a parameter value sequence, and perform combination construction operation with the instruction unit sequence at the corresponding position to generate an operation instruction sequence;

[0170] Extract instruction dependency identifiers from the instruction index positions of the operation instruction sequence, perform dependency resolution calculations at the corresponding positions, determine the sequence and constraint relationships between each operation instruction, and generate an instruction dependency structure sequence.

[0171] Sequential arrangement calculations are performed at the instruction index positions of the operation instruction sequence, and constraint processing is performed at the corresponding positions in combination with the path sequence identifier and the instruction dependency structure sequence to generate the device behavior sequence.

[0172] Specifically, extracting operation parameter information at the unit index position of the instruction unit sequence and performing parameter parsing calculations at the corresponding positions includes:

[0173] Read the device function identifier, operation type identifier and parameter description content in the semantic unit corresponding to each instruction unit, and build a parameter extraction table according to the unit index position;

[0174] In the corresponding positions of each parameter extraction table, the numerical information, time information, object information and range information in the parameter description are identified, and a parameter fragment sequence is generated.

[0175] Perform parameter classification processing on each parameter segment sequence, determine the parameter name item corresponding to each parameter segment, and convert each parameter segment into the corresponding parameter name identifier and parameter value. Write the default parameter value for the missing parameter position according to the preset completion method.

[0176] Write the parameter name identifier, parameter value, and corresponding instruction unit into the same record item to form a parameter value sequence.

[0177] Specifically, the combination operation of the parameter value sequence and the instruction unit sequence at the corresponding position includes:

[0178] Read the device function identifier, operation type identifier, and parameter name identifier and parameter value in each parameter value sequence corresponding to each instruction unit, and establish a corresponding relationship table according to the unit index position;

[0179] At the corresponding position of each instruction unit, the parameter name identifier and parameter value related to the current instruction unit are written into the corresponding record item to form an instruction parameter record unit;

[0180] The execution content of each instruction parameter recording unit is integrated and processed to establish a correspondence between the device function identifier, operation type identifier, parameter name identifier, and parameter value in the same record structure; finally, the execution order of all instruction parameter recording units is arranged according to the unit index order to generate an operation instruction sequence.

[0181] Specifically, extracting instruction dependency identifiers at instruction index positions in the operation instruction sequence and performing dependency resolution calculations at the corresponding positions includes:

[0182] Read the device function identifier, operation type identifier, parameter name identifier, parameter value, and instruction dependency identifier corresponding to each operation instruction, and establish an instruction relationship table according to the instruction index position; record the sequential position and associated instruction number of each operation instruction in the corresponding position of the instruction relationship table to form the basic record of instruction relationship;

[0183] In each instruction relationship table, determine whether there is a preceding execution relationship, a subsequent execution relationship, a mutual exclusion relationship, or a concurrent permission relationship between the current operation instruction and other operation instructions at the corresponding position, and generate an instruction relationship identifier sequence; then, based on the instruction relationship identifier sequence, determine the order of each operation instruction to determine the execution order and constraint relationship position of each operation instruction.

[0184] Write a pre-execution constraint identifier for operation instructions with a pre-execution relationship at the corresponding position, write a subsequent constraint identifier for operation instructions with a subsequent execution relationship, write a mutual exclusion constraint identifier for operation instructions with a mutual exclusion relationship, and write a concurrency identifier for operation instructions with a concurrency permission relationship; organize all constraint relationships to form an instruction dependency structure sequence.

[0185] Specifically, the instruction index position of the operation instruction sequence is calculated for sequential arrangement, and constraint processing is performed at the corresponding position by combining the path sequence identifier and the instruction dependency structure sequence to generate the device behavior sequence, which includes:

[0186] Read the instruction index identifier, path order identifier, and instruction dependency structure sequence corresponding to each operation instruction, and perform initial sorting on all operation instructions according to the path order identifier to form an instruction sequence table;

[0187] Record the sequential arrangement relationship and corresponding constraint identifiers of each operation instruction at the corresponding position in the instruction sequence table to generate the basic record of sequence arrangement;

[0188] At the corresponding position of each operation instruction, it is determined whether the current operation instruction meets the preconditions, subsequent constraints, mutual exclusion constraints and concurrency constraints based on the instruction dependency structure sequence. For operation instructions that meet the constraints, the current sorting position is retained, and for operation instructions that do not meet the constraints, the position is adjusted.

[0189] Operation instructions with preconditions are adjusted to follow the corresponding preconditions; operation instructions with mutual exclusion constraints are separated and arranged; and operation instructions with concurrency constraints are written with the same-level execution flag.

[0190] Organize the execution order of all operation instructions after the position adjustment is completed, and write the execution order identifier, concurrent execution identifier and constraint effective identifier in the corresponding position of each instruction to form an ordered behavior record sequence;

[0191] Output the ordered sequence of behavior records as a sequence of device behavior.

[0192] In this embodiment, the execution feedback module specifically includes:

[0193] Read each operation instruction at the instruction index position in the device behavior sequence, execute the device control instruction issuance calculation at the corresponding position, map the operation instruction into the device execution signal, and generate the device execution result sequence at the execution position;

[0194] The device status information is collected at the result index position of the device execution result sequence. At the corresponding position, the device status information is parsed and calculated to generate a status feature sequence. At the corresponding position, the status feature sequence is matched and calculated with the device behavior sequence to generate an execution consistency result.

[0195] Error calculation is performed at the result index position of the consistency result, and the difference between the target behavior and the actual execution result is calculated at the corresponding position to generate an execution deviation sequence. Then, a feedback adjustment parameter sequence is generated at the corresponding position based on the execution deviation sequence.

[0196] Feedback control calculations are performed at the parameter index positions of the feedback adjustment parameter sequence, scheduling correction operations are performed on the device behavior sequence at the corresponding positions, an updated device behavior sequence is generated, and interactive feedback information is generated at the corresponding positions.

[0197] Specifically, performing state parsing calculations on the device state information at the corresponding locations to generate a state feature sequence includes:

[0198] Read the device status information corresponding to the execution results of each device. The device status information includes device on / off status information, function execution result information, parameter return values ​​and execution time information. Establish a status data table according to the result index position.

[0199] The device status information is classified and organized in the corresponding positions of each status data table, and the status values, status category information and time sequence information are extracted to form a status segment sequence.

[0200] The state values ​​in each state segment sequence are processed in a unified manner, the state category information is classified and organized, the execution time information is sorted in order, and the state values, state category information and execution time information are written into the same state record unit at the corresponding positions to form a state feature record item.

[0201] Perform a sequence permutation on all state feature records to generate a state feature sequence.

[0202] Specifically, the process of performing error calculation at the result index position of the consistency result, calculating the difference between the target behavior and the actual execution result at the corresponding position to generate an execution deviation sequence, and generating a feedback adjustment parameter sequence based on the execution deviation sequence at the corresponding position includes:

[0203] Read the target behavior information and actual execution result information corresponding to each result index position in the execution consistency results, and build a behavior comparison table according to the result index position; extract the target operation content, target parameter value, target execution order and actual operation content, actual parameter value, and actual execution order at the corresponding positions in each behavior comparison table, compare the target behavior information and actual execution result information item by item, and generate content difference results, parameter difference results and order difference results; then summarize and organize each difference result to form the execution deviation record item corresponding to each result index position, and arrange them according to the result index order to generate an execution deviation sequence.

[0204] Read the content difference results, parameter difference results, and sequence difference results corresponding to each execution deviation record item in the execution deviation sequence, and determine the deviation type and degree at each corresponding position; generate content adjustment items for content difference results, parameter adjustment items for parameter difference results, and sequence adjustment items for sequence difference results; then write the content adjustment items, parameter adjustment items, and sequence adjustment items into the same parameter record unit to form feedback adjustment parameters corresponding to each result index position, and organize them according to the result index order to generate a feedback adjustment parameter sequence.

[0205] Specifically, the process of performing feedback control calculations at the parameter index positions of the feedback adjustment parameter sequence, performing scheduling correction calculations on the device behavior sequence at the corresponding positions to generate updated device behavior sequences, and generating interactive feedback information at the corresponding positions includes:

[0206] Read the content adjustment item, parameter adjustment item and sequence adjustment item corresponding to each parameter index position in the feedback adjustment parameter sequence, and read the operation instruction content, parameter value and execution sequence information corresponding to each behavior unit in the device behavior sequence, and establish a correspondence table according to the parameter index position and behavior index position;

[0207] At each corresponding position, determine whether the current device behavior unit has a content correction requirement, parameter correction requirement, or sequence correction requirement. Write a correction mark for the device behavior unit with a correction requirement to form a behavior correction record table.

[0208] In each behavior correction record table, the operation instruction content is replaced according to the content adjustment item, the parameter value is adjusted according to the parameter adjustment item, and the execution order information is rearranged according to the sequence adjustment item to generate a corrected behavior unit sequence. All corrected behavior unit sequences are then reorganized according to the execution order to form an updated device behavior sequence.

[0209] The system reads the correction result information corresponding to each behavior unit in the updated device behavior sequence, extracts the corrected operation content, corrected parameter values, and corrected execution order information at each corresponding position, writes the differences before and after correction into the feedback record unit, and forms a feedback content sequence. The system then organizes the execution order of all feedback content sequences, generates interactive feedback information, and outputs the interactive feedback information to the smart glasses interactive interface.

[0210] Example 1:

[0211] To verify the feasibility of this invention in practice, it was applied to a smart glasses interaction scenario oriented towards mobile operations, information prompts, and near-eye device control. In this scenario, the wearer is in a state of alternating continuous walking, turning around, looking up to observe, and short-term pauses. Simultaneously, there are conversations, mechanical noises, wind noise, and prompts in the surrounding environment. Voice input is often accompanied by pauses, omissions, colloquial substitutions, and context jumps. Traditional smart glasses voice interaction solutions typically retain only a single path recognition result. Once the voice is partially distorted, the system easily maps the recognition result directly to a single control command, leading to situations such as accidental opening, closing, page switching, or execution not according to the true intention. This is especially true in complex statements with preceding and following constraints, such as "open navigation and lower brightness," "change photo to video recording," and "show the list first, then contact the target," where the probability of misunderstanding is even higher.

[0212] This embodiment employs the speech acquisition module, feature parsing module, intent construction module, counterfactual path generation module, entropy flow calculation module, path modulation module, behavior generation module, and execution feedback module described in the claims to fully process the real interaction process. The speech acquisition module continuously acquires environmental speech signals to form a speech signal sequence; the feature parsing module first performs noise reduction and frame segmentation on the original speech, and then extracts the fundamental frequency, energy, pitch, formants, and frequency components to form a speech feature vector sequence. The purpose of this processing is not simply to obtain a recognized text, but to completely preserve the acoustic basis required for subsequent semantic judgment, so that segments of the same sentence disturbed by noise can still be reinterpreted in subsequent stages.

[0213] In this implementation, the intent construction module generates a sequence of semantic segments based on the speech feature vector sequence, then maps the semantic segments to semantic units, and forms multiple candidate semantic paths through connection calculation. For example, when a user utters the statement "Turn on navigation, adjust the brightness to half, and remind me when we reach the intersection," the system does not immediately compress the entire sentence into a single control result. Instead, it first forms multiple candidate connection paths related to navigation activation, brightness adjustment, and reminder triggering, and then filters out a set of valid complete semantic paths based on contextual consistency. This avoids misinterpreting "remind me" as "stop" or "adjust the brightness to half" as "adjust the brightness to normal" when there is local pronunciation ambiguity, thus preventing the entire sentence from becoming invalid.

[0214] The counterfactual path generation module plays a crucial role in this implementation. The system doesn't just consider the candidate semantic paths themselves; instead, it proactively constructs perturbed semantic segments based on the speech feature vector sequence, remapping possible replacements, deletions, and insertions, and generating a set of counterfactual paths. Taking "Show me the next task" as an example, when "next task" is obscured by noise, the system simultaneously constructs potential explanatory paths such as "current task," "next page," and "task task," and then calculates features based on path length, number of branches, connection weights, semantic deviation, and perturbation amplitude. After this process, the system can retain semantic explanations that are truly possible but briefly perturbed by noise, while excluding pseudo-explanatory paths with messy structures and significant deviations from the original sentence.

[0215] After obtaining candidate semantic paths and counterfactual paths, the entropy flow calculation module processes the path probability parameters, path structure distribution parameters, and path scoring results of each path to generate a sequence of path uncertainty entropy flows. Here, entropy flow is not an abstract description, but rather reflects the degree of fluctuation of each semantic path during its evolution. If a path initially has a high score, but the entropy value changes significantly in the ranking of adjacent paths, it indicates that the path is greatly affected by local disturbances and lacks stability. Conversely, if the entropy flow changes relatively smoothly in the preceding and following positions of a path, it indicates that the path is more suitable as the final control basis. The path modulation module generates a sequence modulation factor based on this and establishes a non-commutative constraint relationship in conjunction with the path sequence identifier, ensuring that statements with clear sequential meanings are not reversed due to locally high-scoring segments.

[0216] After determining the target semantic path, the behavior generation module maps semantic units to basic operation instruction units, then parses parameter information to form a complete sequence of operation instructions. The execution feedback module further sends the operation instructions to the corresponding interface display, shooting control, volume control, brightness control, and reminder control units of the smart glasses, and collects device status information to compare the consistency between the target behavior and the actual execution result. When a deviation is detected between the device behavior sequence and the actual execution, the system generates feedback adjustment parameters to correct the interpretation of subsequent similar instructions. For example, in a scenario where "turn off notification sound" and "turn off alert sound" are used interchangeably three times in a row, the execution feedback module will adjust the relevant mapping weights in subsequent semantic segments to align with the actual execution result, thereby reducing false triggering.

[0217] To verify the effectiveness of this invention, a voice interaction set including daily control, information query, path reminder, shooting switching, interface jump, object calling, notification management, and combined control was constructed in this embodiment. A total of 4800 valid voice commands were collected, including 2100 short commands, 1700 compound commands, and 1000 commands with significant background noise. The comparison scheme uses a common single-path semantic recognition plus rule control method, while this invention adopts the above-mentioned complete interaction process. During the test, the semantic recognition accuracy, compound command complete execution rate, false trigger rate, average response time, secondary correction rate, success rate in complex noise scenarios, and continuous interaction stability were recorded. The results show that in simple static scenarios, both schemes can complete basic control, but once entering scenarios with large noise variations and obvious dependencies between commands, the advantages of this invention will quickly become apparent.

[0218] Based on multiple rounds of testing, the semantic recognition accuracy of this invention remained consistently above 96% in ordinary scenarios and above 92% in complex noisy scenarios, significantly higher than the comparative schemes. The complete execution rate of compound instructions increased from 78.4% to 93.6%, the false trigger rate decreased from 6.8% to 1.9%, and the secondary correction rate decreased from 18.7% to 6.1%. In sentences containing sequence constraints, this invention, through path modulation and direction differentiation, makes the execution relationship of sentences such as "display before sending," "take a picture before uploading," and "navigate before reminding" more stable, with stability still above 95% even after more than ten rounds of continuous interaction. This demonstrates that this invention can not only understand single sentences but also understand semantic order and contextual dependencies well in a real-world wearing environment.

[0219] Further observation reveals that this invention also exhibits good fault tolerance even when users use highly colloquial expressions. In tests, intentionally including non-standard expressions such as "Please turn this back on," "Turn off that notification," and "Turn down the navigation volume," the comparison solutions often required users to supplement the object or rephrase their statements. However, this invention, through the combined effect of candidate semantic paths, counterfactual paths, and execution feedback, can still identify the correct target object in most cases. For a small number of statements that cannot be directly determined, the system prioritizes providing the executable path closest to the current context, rather than directly outputting obviously erroneous control commands. Therefore, the overall interaction experience is smoother, and the user interruption rate is lower.

[0220] In summary, this embodiment demonstrates that the present invention can complete a full closed-loop process, from voice acquisition, feature parsing, candidate semantic construction, counterfactual path deduction, entropy flow change measurement, sequential modulation screening, behavior generation, to execution feedback correction, under complex voice input conditions. Compared to existing methods that rely solely on single-path recognition results, the present invention is more suitable for practical wearable smart glasses scenarios, especially for interactive environments with strong noise, numerous sentence omissions, frequent switching of control objects, and a high proportion of complex commands.

[0221] Table 1: Comparison of the Implementation Effects of Voice Interaction in Smart Glasses

[0222] Test Scenario Sample size (number of samples) Semantic recognition accuracy (comparison / invention) Complete execution rate of compound instructions (comparison / invention) False trigger rate (comparison / invention) Average response time (ms) (Comparison / Invention) Static and quiet scenes - basic controls 600 94.8% / 97.6% 88.1% / 95.4% 2.6% / 1.1% 812 / 768 Static and quiet scenes - information retrieval 420 93.9% / 96.8% 85.7% / 94.2% 3.1% / 1.4% 856 / 801 Mild Noise Scene - Interface Switching 560 91.5% / 96.1% 82.4% / 92.8% 4.7% / 1.8% 905 / 838 Low Noise Scene - Notification Management 430 90.6% / 95.3% 80.9% / 91.6% 5.0% / 2.0% 918 / 846 Walking Scene - Navigation Control 520 88.7% / 94.9% 79.1% / 92.5% 5.8% / 2.1% 962 / 881 Walking Scene - Reminder Trigger 380 87.9% / 94.1% 77.6% / 91.8% 6.2% / 2.4% 979 / 896 Complex Noisy Scenes - Shooting Switch 470 85.4% / 92.7% 74.8% / 90.4% 7.4% / 2.5% 1015 / 913 Complex Noise Scenarios - Object Calling 360 84.8% / 92.3% 73.9% / 89.7% 7.9% / 2.8% 1038 / 928 Composite instruction scenarios - Sequential control 560 83.7% / 93.4% 71.5% / 94.1% 8.2% / 1.9% 1086 / 952 Continuous interaction scenarios - more than ten rounds 500 82.9% / 92.6% 70.6% / 93.2% 8.6% / 2.1% 1112 / 968

[0223] Table 1 reflects the actual performance of the present invention under different usage conditions. First, let's look at the semantic recognition accuracy. In 10 scenarios, the accuracy of the compared schemes ranged from 82.9% to 94.8%, with an average of 88.42%; the corresponding range for the present invention was 92.3% to 97.6%, with an average of 94.58%, an average improvement of 6.16 percentage points. The most significant improvements occurred in two scenarios: "composite instruction scenarios - sequential control" and "continuous interaction scenarios - more than ten rounds." The former improved from 83.7% to 93.4%, an increase of 9.7 percentage points, and the latter from 82.9% to 92.6%, an increase of 9.7 percentage points. This indicates that when a sentence contains simultaneous sequential relationships, control object switching, and contextual continuity, the candidate semantic path, counterfactual path, and path modulation mechanism in the present invention can effectively suppress the bias caused by local high-scoring misjudgments. Secondly, regarding the complete execution rate of compound instructions, the average execution rate of the comparative scheme was 78.46%, while that of this invention averaged 92.57%, an average improvement of 14.11 percentage points. Specifically, the rate increased from 71.5% to 94.1% in sequential control scenarios (an improvement of 22.6 percentage points), and from 70.6% to 93.2% in continuous interaction scenarios (an improvement of 22.6 percentage points). This indicates that this invention not only "understands" the instructions but also correctly assigns multiple actions within the same instruction to the device's behavior sequence. Next, regarding the false trigger rate, the average rate of the comparative scheme was 5.95%, while that of this invention averaged 2.01%, a decrease of 3.94 percentage points. In complex noise scenarios—object calls—the false trigger rate of the comparative scheme reached 7.9%, while that of this invention decreased to 2.8%, a decrease of 5.1 percentage points. In sequential control scenarios, the rate decreased from 8.2% to 1.9%, a decrease of 6.3 percentage points. These results demonstrate that the entropy flow change rate and direction-distinguishing modulation factor can effectively suppress unstable paths, thereby reducing the direct execution of erroneous operations. Finally, looking at the average response time, the average response time for the comparison solution in 10 scenarios was 968.3ms, while the average response time for this invention was 879.1ms, a reduction of 89.2ms on average. Although this invention incorporates candidate path and counterfactual path analysis in its processing, the actual overall interaction did not slow down due to more stable subsequent behavior generation and execution feedback. On the contrary, the overall response was more coherent because of the reduction in secondary confirmation and repeated commands. The values ​​in the table show that this invention improves performance in quiet scenarios, and the improvement is more significant in walking, complex noise, and continuous interaction scenarios. This is highly consistent with the real-world wearing environment of smart glasses, directly demonstrating the good practicality and stability of this invention.

[0224] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A smart glasses interaction system based on voice recognition, characterized in that, include: The voice acquisition module is used to acquire environmental voice signals and generate voice signal sequences; The feature parsing module is used to perform noise reduction and feature extraction operations on the speech signal sequence to generate a speech feature vector sequence; The intent building module is used to perform semantic parsing operations at the corresponding positions in the speech feature vector sequence to generate a set of candidate semantic paths. The counterfactual path generation module is used to generate a set of counterfactual paths based on the speech feature vector sequence, where each counterfactual path corresponds to a potential semantic interpretation; The entropy flow calculation module is used to construct a path uncertainty entropy flow sequence at the corresponding position of the candidate semantic path set, and to calculate the entropy flow change rate at the path evolution position; The path modulation module is used to generate an order modulation factor based on the entropy flow change rate at the path combination position, establish non-commutative operation rules at the path sorting position, and perform order constraint calculations on the candidate semantic path set to generate the target semantic path. The behavior generation module is used to perform instruction reconstruction operations at the corresponding position of the target semantic path to generate a device behavior sequence; The execution feedback module is used to perform scheduling control on the device behavior sequence and generate interactive feedback information.

2. The intelligent glasses interaction system based on voice recognition according to claim 1, characterized in that, The feature parsing module generates a sequence of speech feature vectors by including the following steps: The system receives a sequence of audio signals, converts it into a digital signal stream, performs noise reduction processing, extracts the frequency component of the signal through wavelet transform, and removes background noise. The denoised signal stream is divided into frames, which are of equal duration. Each frame contains a certain length of speech data and there is overlap between the frames. Mel frequency cepstral coefficients are extracted from each frame of signal to extract speech features, including fundamental frequency, energy, pitch, and formants, and then standardized. The frequency domain of each frame of signal is analyzed using Fast Fourier Transform to obtain frequency component information, and the first 20 frequency components are extracted. The extracted frequency features are combined with time-domain features, and principal component analysis is used to reduce the dimensionality of the extracted features, resulting in a sequence of dimensionality-reduced speech feature vectors.

3. The intelligent glasses interaction system based on voice recognition according to claim 2, characterized in that, The execution of semantic parsing by the intent construction module to generate a candidate semantic path set specifically includes: The speech feature vector sequence is segmented, and a semantic segment sequence is generated at the segment position. Semantic identifier vector and context association identifier are extracted at each semantic segment position. A set of semantic mapping rules is established at the corresponding position of the semantic fragment, and the semantic identifier vector is mapped to the basic semantic unit. During the mapping process, the semantic category identifier and semantic intensity value are written to generate a sequence of semantic units. Establish sequence connection relationships at corresponding positions in the semantic unit sequence, perform connection calculations on adjacent semantic units, generate semantic connection paths at connection positions, and record path order identifiers and connection weight values ​​at path positions; Introduce contextual constraints at the corresponding positions of semantic connection paths, perform constraint filtering calculations on semantic connection paths, eliminate connection paths that do not satisfy contextual consistency, and generate a set of candidate connection paths; Perform path expansion calculations at the corresponding positions in the candidate connection path set, combine and expand several semantic connection paths, generate several complete semantic paths at the expansion positions, and perform set construction operations on several complete semantic paths at the corresponding positions to generate a complete semantic path set. Perform path filtering calculations on the complete set of semantic paths, sort them according to path confidence values ​​and path structure completeness, and output a set of candidate semantic paths.

4. The intelligent glasses interaction system based on voice recognition according to claim 3, characterized in that, The counterfactual path generation module specifically includes: A semantic perturbation construction operation is performed on the speech feature vector sequence, a perturbation parameter sequence is generated at the feature index position, and a perturbation type identifier and a perturbation amplitude value are written at the corresponding position. The perturbation type identifier includes at least a semantic replacement identifier, a semantic deletion identifier, and a semantic insertion identifier. A perturbation semantic segment sequence is generated based on the perturbation parameter sequence. Semantic remapping rules are established at the corresponding positions of the perturbation semantic fragment sequence to map the perturbation semantic fragments into perturbation semantic units. During the mapping process, semantic category identifiers and semantic association identifiers are written to generate a perturbation semantic unit sequence. Establish counterfactual connections at corresponding positions in the perturbation semantic unit sequence, perform path connection calculations between perturbation semantic units, generate counterfactual connection paths at connection positions, and record path sequence identifiers, path branch identifiers, and connection weight values ​​at path positions. At the corresponding position of the counterfactual connection path, perform path bifurcation extension calculation, perform combination construction operation on the perturbation parameter sequence at the path bifurcation position, combine the perturbation type identifier and the perturbation amplitude value according to the index position to generate a perturbation combination set, generate several path branches based on the perturbation combination set at the path bifurcation position, write the branch source identifier and the branch level identifier at the branch position, perform path extension operation on each path branch to generate several counterfactual semantic paths; Path constraint calculations are performed at the corresponding positions of several counterfactual semantic paths. Semantic consistency constraints and contextual rationality constraints are introduced at the constraint positions. Screening is performed on each counterfactual semantic path to eliminate paths that do not meet the constraints and generate a set of candidate counterfactual paths. Path feature calculation is performed at the corresponding position in the candidate counterfactual path set. Complexity calculation is performed at the path index position based on the path length, number of path branches, and connection weight values ​​in the counterfactual semantic path to generate the path structure complexity value. At the path index position, perform difference calculation based on the perturbation semantic unit sequence and the semantic unit sequence to generate a semantic deviation value; At the path index position, a path score is calculated based on the path disturbance amplitude, path structure complexity, and semantic deviation, generating a path score result. The path score result is then sorted, and a counterfactual path priority sequence is generated at the sorting position. Path selection calculation is performed at the corresponding position of the counterfactual path priority sequence. The first few counterfactual semantic paths are selected according to a preset quantity threshold. Set construction operation is performed at the corresponding position to generate a counterfactual path set.

5. The intelligent glasses interaction system based on voice recognition according to claim 4, characterized in that, The entropy flow calculation module specifically includes: Extract the path probability parameters corresponding to each semantic path at the path index position of the candidate semantic path set, and perform normalization processing on the path probability parameters at the corresponding positions to generate a path probability sequence. Extract the number of path branches and the path length of each semantic path from the path index position of the candidate semantic path set, and perform joint statistical calculations on the number of path branches and the path length at the corresponding positions to generate path structure distribution parameters. Extract the path score results corresponding to each semantic path at the path index position in the candidate semantic path set, perform scale unification processing on the path score results at the corresponding position, and generate a path weight sequence; At the path index position in the candidate semantic path set, entropy value calculation is performed based on the path probability sequence, path structure distribution parameters, and path weight sequence to generate a path uncertainty entropy value sequence; Differential calculations are performed at the path index positions of the path uncertainty entropy value sequence according to the semantic path order identifier, and entropy value change sequences are generated at adjacent path index positions; Normalization is performed at the path index position of the entropy change sequence, combined with the path sequence identifier, to generate the entropy flow change rate sequence. Perform a sequence mapping operation at the path index position of the entropy flow rate of change sequence to map the entropy flow rate of change sequence to the corresponding position in the candidate semantic path set, generating a path uncertainty entropy flow sequence.

6. The intelligent glasses interaction system based on voice recognition according to claim 5, characterized in that, The path modulation module specifically includes: Extract the path uncertainty entropy flow sequence and path score result from the path index position of the candidate semantic path set. Perform interval division calculation on the entropy flow change rate value at the corresponding position to generate the entropy flow interval identifier sequence. Perform association mapping between the entropy flow interval identifier sequence and the path score result at the corresponding position to generate the path modulation parameter sequence. At the path index position of the candidate semantic path set, the sequential modulation factor generation calculation is performed based on the path modulation parameter sequence. At the corresponding position, the sequential modulation factor value is generated according to the entropy flow interval identifier and the path score result. At the path index position, the sequential modulation factor value is subjected to directional constraint processing in combination with the path sequence identifier to generate a direction-distinguished modulation factor sequence. Establish path combination relationships based on the direction-distinguished modulation factor sequence at the path index position of the candidate semantic path set. Perform path order constraint calculation at the combination position to generate path combination constraint results. Perform sorting calculation based on the path combination constraint results and path score results at the path index position to output the target semantic path.

7. The intelligent glasses interaction system based on voice recognition according to claim 6, characterized in that, The behavior generation module specifically generates the device behavior sequence including: Extract semantic unit sequences and path order identifiers from the path index positions of the target semantic path, establish instruction mapping rule sets at the corresponding positions, map the semantic unit sequences to basic operation instruction units, write device function identifiers and operation type identifiers during the mapping process, and generate instruction unit sequences. Extract operation parameter information from the unit index position of the instruction unit sequence, perform parameter parsing calculation at the corresponding position, convert the operation parameter information into a parameter value sequence, and perform combination construction operation with the instruction unit sequence at the corresponding position to generate an operation instruction sequence; Extract instruction dependency identifiers from the instruction index positions of the operation instruction sequence, perform dependency resolution calculations at the corresponding positions, determine the sequence and constraint relationships between each operation instruction, and generate an instruction dependency structure sequence. Sequential arrangement calculations are performed at the instruction index positions of the operation instruction sequence, and constraint processing is performed at the corresponding positions in combination with the path sequence identifier and the instruction dependency structure sequence to generate the device behavior sequence.

8. The intelligent glasses interaction system based on voice recognition according to claim 7, characterized in that, The execution feedback module specifically includes: Read each operation instruction at the instruction index position in the device behavior sequence, execute the device control instruction issuance calculation at the corresponding position, map the operation instruction into the device execution signal, and generate the device execution result sequence at the execution position; The device status information is collected at the result index position of the device execution result sequence. At the corresponding position, the device status information is parsed and calculated to generate a status feature sequence. At the corresponding position, the status feature sequence is matched and calculated with the device behavior sequence to generate an execution consistency result. Error calculation is performed at the result index position of the consistency result, and the difference between the target behavior and the actual execution result is calculated at the corresponding position to generate an execution deviation sequence. Then, a feedback adjustment parameter sequence is generated at the corresponding position based on the execution deviation sequence. Feedback control calculations are performed at the parameter index positions of the feedback adjustment parameter sequence, scheduling correction operations are performed on the device behavior sequence at the corresponding positions, an updated device behavior sequence is generated, and interactive feedback information is generated at the corresponding positions.

Images