Intelligent glasses AI voice interaction method

Abstract

The application provides an intelligent glasses AI voice interaction method, comprising: simultaneously collecting user gesture images and voice signals, obtaining user historical interaction records, extracting key point coordinates and pointing direction information of the gesture images, and obtaining hand distance information, generating a gesture motion path, simultaneously extracting frequency spectrum information and tone peak values of the voice signals, and forming a voice beat sequence; extracting a spatial semantic pattern of gesture voice fusion data, identifying a core object of a pointing instruction of the user according to the pointing coordinates of the gesture motion path and the tone peak values of the voice beat sequence; after identifying the core object of the pointing instruction, determining the user's intention in combination with the spatial semantic pattern and the user historical interaction records; outputting a complete intention analysis result to an intelligent glasses display module to perform corresponding operations, and simultaneously feeding back to a collection module to adjust next-time capturing parameters including a collection frequency and an identification sensitivity, thereby improving response speed and accuracy.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to an AI voice interaction method for smart glasses. Background Technology

[0002] As an important branch of wearable devices, smart glasses provide users with an immersive experience by integrating voice, vision, and gesture interaction, demonstrating key value in education, healthcare, and industrial maintenance. Their core lies in understanding the user's natural interaction intentions and capturing the coordinated expression of voice and actions to achieve efficient and intuitive command execution. However, existing methods have significant limitations in handling multimodal interactions, primarily manifested in insufficient real-time collaborative analysis of voice and gestures in dynamic interactions. Many systems rely on predefined gesture templates or fixed voice commands, leading to inaccurate recognition of user intentions in complex scenarios, especially when user behavior is personalized, often resulting in misjudgment or slow response. The core challenge lies in accurately capturing and integrating the dynamic consistency between user gesture trajectories and voice rhythm. The diversity of gesture trajectories requires the system to track the direction and speed of finger movements in real time and match them with changes in tone and pauses in speech. For example, when a user points to an object and says "zoom in on this," the rapid swipe of the gesture is difficult to capture accurately, leading the system to misunderstand the specific object being referred to. Furthermore, this lack of dynamic consistency directly affects the efficiency of multimodal information fusion. Because the temporal relationship between gestures and speech is not fully analyzed, the system struggles to synthesize the semantics of both in a short time to form a complete user intent. Therefore, how to capture the collaborative features of gesture trajectories and speech rhythms in real time in complex dynamic scenarios, and accurately understand user directional commands through precise fusion of their temporal relationship, has become a key issue in the research of natural interaction for smart glasses. Summary of the Invention

[0003] This invention provides an AI voice interaction method for smart glasses, mainly including: The system acquires user gesture images and voice signals, obtains user historical interaction records, extracts key point coordinates and pointing direction information from the gesture images, generates a gesture motion path, and extracts the spectral information and pitch peaks from the voice signals to form a voice beat sequence. It performs time-series synchronization processing on the gesture motion path and the voice beat sequence, identifies the temporal correlation between gesture actions and voice pronunciation, extracts speed change nodes in the gesture motion path and pause points in the voice beat sequence, aligns them using timestamps, and generates a dynamic consistency score. If the dynamic consistency score exceeds a preset threshold, it fuses the pointing direction information of the gesture motion path with the keyword positions in the voice beat sequence to generate gesture-voice fusion data. It analyzes the spatial semantic pattern of the gesture-voice fusion data, identifies the core object of the user's pointing command based on the pointing coordinates of the gesture motion path and the pitch peaks of the voice beat sequence, combines the core object, the spatial semantic pattern, and the user's historical interaction records to determine the user's intent, generates an execution command containing the target object, operation type, and execution timing based on the user intent, and transmits the execution command to the smart glasses display module for execution, while simultaneously adjusting the acquisition capture parameters. Furthermore, the process of acquiring user gesture images and voice signals, obtaining user historical interaction records, extracting key point coordinates and pointing direction information from the gesture images, generating gesture movement paths, and extracting the spectral information and pitch peaks of the voice signals to form a voice beat sequence includes: The system captures an RGB image stream of the gesture image, records the voice signal through a microphone array, and reads the gesture trajectory template and voice command template from the user's historical interaction records in local storage. Based on the RGB image stream, it calculates the three-dimensional coordinates of key points on the hand, determines the pointing direction vector of the line connecting the index fingertip and the wrist, calculates the hand depth value through binocular parallax, and generates an absolute position sequence of the gesture movement path. It performs spline interpolation on the absolute position sequence, connects the key point coordinates to generate a smooth path, calculates the path curvature change rate, and marks the turning points to form the gesture movement path. It performs time-frequency analysis on the voice signal, generates a spectrogram, calculates the weighted average frequency of each frame of the spectrum, detects the abrupt change point of the spectral centroid as the pitch peak, locates the falling edge of the energy envelope, and segments to form the voice beat sequence. Furthermore, the step of generating gesture-speech fusion data by fusing the directional information of the gesture movement path with the keyword positions of the speech beat sequence based on the dynamic consistency score being higher than a preset threshold includes: The speech beat sequence is subjected to speech recognition to generate a text sequence. Nouns, verbs, and adjectives are extracted as candidate keywords, and the start and end timestamps of the candidate keywords are recorded. High-frequency command words are extracted from the user's historical interaction records and matched with the candidate keywords to generate a keyword sequence containing keyword text and time position. Based on the dynamic consistency score being higher than a preset threshold, the pointing direction vector and spatial coordinate sequence corresponding to the keyword sequence time are extracted from the gesture movement path, and the average pointing direction is calculated. The keyword text, time position, average pointing direction, and spatial coordinates are combined into a structured data record, arranged in chronological order, with time interval markers inserted, and the keyword semantic attributes and gesture spatial attributes are encoded. These are then concatenated to form a fusion feature vector, and the gesture speech fusion data is output.

[0004] Furthermore, the step of performing time-series synchronization processing on the gesture motion path and the speech beat sequence, identifying the temporal correlation between the gesture action and the speech pronunciation, extracting speed change nodes in the gesture motion path and pause points in the speech beat sequence, and generating a dynamic consistency score through timestamp alignment includes: Calculate the displacement difference between adjacent frames in the gesture motion path to generate an instantaneous velocity sequence, detect the extreme points of velocity in three consecutive frames, mark them as velocity change nodes, and record the timestamps; extract the audio energy envelope from the speech beat sequence, calculate the background noise level, and mark the time points where the energy is lower than the background noise level as the pause points; construct a time correlation matrix, calculate the time interval between the velocity change nodes and the pause points, statistically analyze the strong correlation node comparison ratio, and generate the dynamic consistency score. Furthermore, the construction of the time correlation matrix, calculation of the time interval between the velocity change node and the pause point, statistical analysis of the strong correlation node ratio, and generation of the dynamic consistency score include: Using the timestamps of the speed change nodes as row indices and the timestamps of the pause points as column indices, the time correlation matrix is ​​constructed, and the absolute values ​​of the time intervals are calculated and filled into the matrix elements. Node pairs with intervals less than a preset synchronization window are marked as strongly correlated, and the ratio of the number of strongly correlated node pairs to the total number of node pairs is calculated. The time offset sequences of strongly correlated node pairs are extracted, the standard deviation is calculated, and the dynamic consistency score is generated by combining the speed difference and frequency difference covariance. Furthermore, the spatial semantic pattern of the gesture-speech fusion data is analyzed, and based on the directional coordinates of the gesture movement path and the pitch peaks of the speech beat sequence, the core object of the user's pointing command is identified, including: The semantic and spatial vectors in the gesture-speech fusion data are analyzed, and the cosine similarity between the semantic vector and the preset category center is calculated to determine the semantic label. The pointing direction vector in the spatial vector is converted into screen coordinates, and the keyframes corresponding to the pitch peak are extracted to construct a spatial semantic mapping table. The screen coordinates are clustered to generate potential object regions, the region density is calculated, and the core objects are marked in combination with the amplitude of the pitch peak. The identifier and coordinates of the core objects are output. Furthermore, by combining the core object, the spatial semantic pattern, and the user's historical interaction records, the user's intent is determined, including: The frequency of operations involving the same object category is statistically analyzed from the user's historical interaction records to construct a frequency matrix; the similarity between the operation type and the verb tag is calculated, and a confidence score is generated by combining the frequency values ​​to determine the user's intent and parameters. Furthermore, the user intent generates execution instructions that include the target object, operation type, and execution sequence, including: The operation type of the user intent is mapped to an execution action code, and a basic execution instruction is generated by combining the screen coordinates of the target object; the displacement vector and instantaneous velocity are extracted from the gesture motion path, and pause points and turning points are marked; the beat duration is extracted, the action segment and beat are matched, a time-sequential execution unit sequence is generated, and the execution instruction is output.

[0005] Furthermore, the step of transmitting the execution command to the smart glasses display module for execution, while simultaneously adjusting the acquired capture parameters, includes: The execution command is transmitted to the display module, a highlight box is superimposed, the interface operation is triggered, and the execution delay and recognition accuracy are recorded. Based on the execution delay and recognition accuracy, the performance deviation and accuracy deviation are calculated, the acquisition frequency and recognition sensitivity are adjusted, and the capture parameters are updated.

[0006] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: This invention discloses an AI voice interaction method for smart glasses, addressing the problems of asynchronous timing between gestures and voice signals and inaccurate intent recognition in traditional human-computer interaction scenarios. By simultaneously acquiring user gesture images and voice signals, extracting the coordinates of key gesture points, pointing directions, and movement paths, and combining this with voice spectrum information and beat sequences, time synchronization processing is performed to identify the temporal correlation between gesture actions and voice pronunciation, calculating a dynamic consistency score. When the consistency score exceeds a threshold, gesture pointing and voice keywords are fused to generate gesture-voice fusion data, extracting spatial semantic patterns, and combining this with the user's historical interaction records to accurately determine the user's intent. Finally, an intent parsing result containing the target object, operation type, and execution timing is generated and output to the smart glasses for execution, with feedback to optimize acquisition parameters. This invention significantly improves the recognition accuracy and response speed of interactive commands through multimodal temporal collaboration and semantic fusion of gestures and voice, optimizing the natural interactive experience of smart devices. Attached Figure Description

[0007] Figure 1 This is a flowchart of an AI voice interaction method for smart glasses according to the present invention.

[0008] Figure 2 This is a schematic diagram of an AI voice interaction method for smart glasses according to the present invention. Detailed Implementation

[0009] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0010] like Figure 1-2 This embodiment of an AI voice interaction method for smart glasses may specifically include: S101. Collect user gesture images and voice signals, and simultaneously obtain user historical interaction records. Extract the key point coordinates and pointing direction information of the gesture images to generate gesture movement paths. Extract the spectrum information and pitch peaks of the voice signals to form a voice beat sequence.

[0011] The smart glasses' binocular camera and microphone array are activated for multi-source acquisition. The binocular camera captures an RGB image stream of the hand at a preset frame rate, while the microphone array records voice signals at a preset sampling rate. Simultaneously, gesture trajectory templates and voice command templates from the user's historical interaction records are retrieved from local storage. A pose estimation algorithm identifies the 3D coordinates of key hand points. The pointing direction vector is calculated based on the line connecting the index fingertip and wrist. The depth value of the hand from the camera is measured using the binocular parallax principle, obtaining the absolute position sequence of the hand in 3D space. Trajectory fitting is performed on this absolute position sequence. Spline interpolation is used to connect the coordinates of key points in adjacent frames to generate a smooth motion path. The rate of change of path curvature is calculated to identify gesture turning points and form the gesture motion path. Simultaneously, time-frequency analysis is performed on the recorded voice signal to obtain a spectrogram. The weighted average frequency of each frame's spectrum is calculated to obtain the spectral centroid trajectory. Abrupt changes in the spectral centroid are detected as pitch peaks, and the descent edge of the energy envelope curve is used to locate the speech beat boundary. From each beat segment segment segmented by the speech beat boundary, the spectrum is processed by a Mel filter bank and feature coefficients are extracted by discrete cosine transform to construct a speech beat sequence. The Euclidean distance between adjacent beat vectors is calculated to determine rhythmic stability. If the distance is less than a preset stability threshold, it is marked as a regular beat; otherwise, it is marked as a changing beat. Simultaneously, the instantaneous velocity is calculated from the gesture motion path, and the velocity maxima are identified as action emphasis points, establishing a temporal correspondence between action emphasis points and changing beats. Based on the temporal correspondence and the beat feature vector sequence, a gesture-speech synchronization matrix is ​​constructed. The matrix rows represent gesture keyframe indices, the columns represent speech beat indices, and the matrix element values ​​are the absolute values ​​of time differences. The optimal alignment scheme is solved using a bipartite graph matching algorithm. The keypoint coordinates and pointing vectors of the gesture motion path are temporally fused with the spectral features and beat markings of the speech beat sequence to output a multimodal data stream containing spatial trajectory information and audio rhythm information.

[0012] For example, in one implementation, the smart glasses achieve stereoscopic acquisition of gesture images through a binocular camera array. The left and right cameras are 65 mm apart, simulating the distance between human eyes. Each camera has a field of view of 120 degrees, and the overlapping area of ​​the two cameras' fields of view forms a stereoscopic vision perception zone. When the user's hand enters the overlapping area, the left and right cameras simultaneously capture hand images. Due to the difference in viewing angle, the pixel position of the same hand feature point is horizontally offset in the two images. This disparity value is inversely proportional to the depth distance. The pose estimation algorithm first detects the hand region in the left image and extracts the pixel coordinates of 21 key points such as the palm, wrist, and finger joints through a convolutional neural network. Then, it searches for matching points on the corresponding epipolar line in the right image, calculates the disparity value based on the pixel distance between the matching points, and combines the camera intrinsic parameter matrix and baseline distance to calculate the actual coordinates of each key point in three-dimensional space using the triangulation principle.

[0013] Specifically, the microphone array adopts a four-element ring layout, distributed at the four corners of the eyeglass frame, forming a spatial sound pickup network. After each microphone independently collects audio signals, beamforming technology is used to enhance the speech signal in the target direction and suppress environmental noise. The historical interaction records are stored in a local cache, containing the user's most recently performed gesture type, duration, movement speed, and corresponding voice command text and audio features. After reading these records, the system statistically analyzes high-frequency gesture patterns and common speech phrases to construct a personalized interaction template library. When a similar gesture initiation or speech start is detected, the corresponding template is preloaded to accelerate the subsequent recognition process. When processing gesture trajectories, the spline interpolation method first arranges the discrete keypoint coordinate sequence in chronological order, with every five consecutive points forming an interpolation interval. Within each interval, a cubic spline function is constructed to ensure that the function value, first derivative, and second derivative are continuous at the nodes, thereby generating a smooth spatial curve. Curvature calculation is achieved by solving the second derivative of the curve. The rate of change of curvature is the derivative of curvature with respect to time. When the rate of change exceeds a preset threshold, it is marked as a gesture inflection point. These inflection points usually correspond to conscious changes in direction or pauses by the user. Time-frequency analysis uses short-time Fourier transform to divide the speech signal into overlapping time windows, each 25 milliseconds long with an overlap rate of 50%. A fast Fourier transform is performed on each window to obtain the spectrum. The spectra of all windows are arranged in chronological order to form a time-frequency spectrogram.

[0014] For example, the spectral centroid is the center of gravity of the spectrum, reflecting the brightness characteristics of the sound. During calculation, the amplitude of each frequency component is used as a weight, and the frequency value as the position; a weighted average is used to obtain the spectral centroid value. The spectral centroid is calculated over consecutive time frames, forming a spectral centroid trajectory curve. When the user's pitch rises, high-frequency components increase, and the spectral centroid value rises; when the pitch falls, low-frequency components increase, and the spectral centroid value falls. By detecting local extrema of the spectral centroid trajectory, the position of pitch peaks is identified; these peaks typically correspond to stressed or emotionally emphasized parts of the sentence. The energy envelope is obtained by calculating the root mean square energy of each frame of the signal; as the energy value falls from its peak to its trough, the midpoint of the falling edge is marked as the beat boundary. The Mel filter bank simulates the auditory characteristics of the human ear, with dense filters in the low-frequency range and sparse filters in the high-frequency range. In specific implementation, linear frequencies are first converted to Mel frequencies, and 40 triangular filters are evenly placed in the Mel frequency domain, each filter covering a specific frequency range. The spectrum is passed through this set of filters to obtain 40-dimensional Mel-frequency spectral features. A discrete cosine transform is then performed, retaining the first 13 coefficients as Mel-frequency cepstral coefficients. These coefficients capture the timbre characteristics of the speech, providing good differentiation between speaker and content. The feature vectors of adjacent beats are similar using Euclidean distance; a smaller distance indicates a more stable rhythm.

[0015] In one possible implementation, gesture speed is calculated based on changes in keypoint positions between adjacent frames. Taking the index fingertip as an example, its three-dimensional coordinates at consecutive moments are recorded, the displacement vector at adjacent moments is calculated, and divided by the time interval to obtain the instantaneous velocity vector. The magnitude of the velocity reflects the speed of the movement, while the direction reflects the trend of the movement. When the velocity magnitude reaches a local maximum, it indicates that the user has performed a rapid pointing or swiping action at that moment. The emphasis of this action often appears synchronously with the accented beats in speech, reflecting the coordination of multimodal interaction.

[0016] Preferably, in the process of constructing the gesture-speech synchronization matrix, keyframes are first extracted from the gesture sequence, including velocity extreme value frames, direction transition frames, and pause frames. Simultaneously, beat frames are extracted from the speech sequence, including accent frames, pause frames, and intonation transition frames. The timestamps of the gesture keyframes and the speech beat frames are compared, and the absolute value of the time difference between each pair of frames is calculated and filled into the corresponding position in the matrix. Each element in the matrix represents a possible matching relationship; the smaller the time difference, the higher the matching probability. The bipartite graph matching algorithm uses the set of gesture keyframes and the set of speech beat frames as two vertex sets of a bipartite graph, and the reciprocal of the matrix elements as the edge weights, searching for a perfect match that maximizes the total weight. The algorithm iteratively updates the matching scheme through augmenting path search until no new augmenting path can be found. The matching result determines the speech segment corresponding to each gesture action, achieving accurate alignment of cross-modal information. Through this alignment mechanism, the system can accurately understand the specific object referred to by "this" when a user points to a virtual object and says "zoom in on this," thereby generating the correct control command.

[0017] S102. Perform time synchronization processing on the gesture movement path and speech beat sequence, identify the temporal correlation between gesture action and speech production, extract gesture speed change nodes from the gesture movement path, identify speech pause points from the speech beat sequence, and align the gesture speed change nodes and speech pause points with timestamps to determine the dynamic consistency score of gesture-speech coordination.

[0018] The displacement difference between adjacent frames is calculated from the gesture motion path to obtain the instantaneous velocity value at each moment. Local maxima and minima are detected by comparing the velocity values ​​of three consecutive frames, and extreme points are marked as velocity change nodes. The timestamp and velocity change amplitude of each node are recorded. Simultaneously, the audio energy envelope is extracted from the speech beat sequence, and the average energy of the past ten frames is calculated as the background noise level. When the energy value is continuously lower than the background noise level for more than a preset duration, it is marked as a speech pause point, and the start time and duration of the pause point are obtained. For the velocity change nodes and speech pause points, a time correlation matrix is ​​constructed. The row index of the matrix corresponds to the timestamp of the gesture velocity change node, and the column index corresponds to the timestamp of the speech pause point. The absolute value of the time interval between each pair of nodes is calculated and filled into the matrix elements. If the interval is less than a preset synchronization window threshold, it is marked as a strong correlation; otherwise, it is marked as a weak correlation. The ratio of the number of strongly correlated node pairs to the total number of node pairs is used as the preliminary correlation degree. Based on the strongly correlated node pairs in the temporal correlation matrix, the corresponding time offset sequences are extracted. The standard deviation of the offset sequences is calculated to assess temporal stability. If the standard deviation is less than a preset stability threshold, the velocity difference before and after the velocity change node in the gesture motion path is calculated as the change intensity, and the frequency difference before and after the pause point in the speech beat sequence is calculated as the pitch change intensity. The correlation metric is obtained by calculating the ratio of the covariance of the two change intensities to the product of their respective variances. Based on the correlation metric and the preliminary correlation degree, the correlation metric is multiplied by a preset feature weight, and the preliminary correlation degree is multiplied by a preset time weight. The two are added together to obtain the dynamic consistency score of gesture-speech coordination. The row and column indices of the strongly correlated node pairs are extracted from the temporal correlation matrix and arranged in chronological order to form an alignment mark sequence. The synchronization processing result containing the dynamic consistency score and the alignment mark sequence is output.

[0019] For example, in one implementation, the instantaneous velocity value is calculated based on the positional changes of gesture keypoints in three-dimensional space. Smart glasses capture hand images at a frequency of 60 frames per second, with each frame containing the three-dimensional coordinates of 21 keypoints. For the keypoint of the index fingertip, its coordinates in frame t are recorded as (xt, yt, zt), and in frame t+1 as (xt+1, yt+1, zt+1). The Euclidean distance between the two points is divided by the frame interval of 16.7 milliseconds to obtain the instantaneous velocity scalar value at that moment. By performing the same calculation on all keypoints, a complete velocity field is formed. Local extrema detection uses a three-point comparison method: when the velocity value in frame t is simultaneously greater than that in frames t-1 and t+1, it is marked as a maximum; otherwise, it is marked as a minimum. These extrema reflect the acceleration and deceleration transitions in the gesture movement and are key indicators of the user's conscious control over the rhythm of the gesture.

[0020] Specifically, in the process of extracting the audio energy envelope, the speech signal is divided into short time windows of 10 milliseconds. Within each window, the square root of the sum of the squares of all sampling points is calculated to obtain the energy value of that window. The energy values ​​of consecutive windows constitute the energy envelope curve. The background noise level is calculated by statistically analyzing the energy values ​​over the past 100 milliseconds, and its 75th percentile is taken as the noise benchmark. When the energy values ​​of five consecutive windows are all lower than the noise benchmark, it is considered that a speech pause has occurred, and the start timestamp and duration of the pause are recorded.

[0021] It should be noted that the construction process of the temporal correlation matrix reflects the temporal correspondence between gestures and speech. The number of rows in the matrix equals the number of detected gesture speed change nodes, and the number of columns equals the number of speech pauses. For the i-th gesture node occurring at time ti, and the j-th speech pause occurring at time tj, the value of matrix element Mij is |ti-tj|. The synchronization window threshold is set to 200 milliseconds. This time window is based on the human perception of synchronization tolerance; when the time difference between visual and auditory events is less than 200 milliseconds, the brain perceives them as occurring simultaneously. Therefore, when Mij is less than 200 milliseconds, the node pair is marked as strongly correlated, indicating that the gesture and speech pause are highly synchronized in time; otherwise, it is marked as weakly correlated. The preliminary correlation degree is obtained by calculating the proportion of strongly correlated node pairs to the total number of node pairs, reflecting the overall temporal coordination.

[0022] For example, in a scenario where a user says "make this object bigger" while simultaneously making a magnifying gesture, there is usually a short pause when the user says "this," and the speed at which the finger points to the target object reaches its peak; when saying "magnify," there is another pause, and the gesture begins the expansion movement from a static state, with the speed increasing from zero. The time offset sequence records the time difference values ​​of all strongly correlated node pairs. If the user's gestures and speech are well-coordinated, these differences will be relatively stable with a small standard deviation; conversely, if the gestures lag behind or precede the speech, the standard deviation of the offset sequence will increase.

[0023] Preferably, the calculation of the change intensity focuses on the dynamic features before and after the node. For gesture speed change nodes, the average speed of the 50 milliseconds before and after the node is extracted, and the absolute value of the difference between the two is taken as the speed change intensity. For speech pause points, the mean fundamental frequency of the 100 milliseconds before and after the pause is analyzed, and the difference in fundamental frequency reflects the pitch change intensity. The change intensities of all strongly correlated node pairs are arranged in chronological order to form two sequences: the gesture change intensity sequence and the speech change intensity sequence.

[0024] In one possible implementation, the correlation metric is calculated using statistical methods to assess the synchronicity of the two sequences. First, the variance σ of the gesture change intensity sequence is calculated. 2 h and the variance σ of the speech change intensity sequence 2v, then calculate the covariance cov(h,v) of the two sequences, and the correlation measure is equal to cov(h,v) / (σh×σv). This value ranges from -1 to 1. A value close to 1 indicates a high degree of consistency between the gesture and speech patterns, with a significant change in speech when the user makes a strong gesture; a value close to 0 indicates independent changes and a lack of coordination; a negative value indicates a reverse correlation, which is less common in actual interactions. Furthermore, the calculation of the dynamic consistency score considers both time and feature dimensions. The feature weights are dynamically adjusted based on the confidence level of the correlation measure. When the number of strongly correlated node pairs is greater than 10, the feature weight is set to 0.6; when the number of node pairs is between 5 and 10, the weight is set to 0.4; when there are fewer than 5 pairs, the weight drops to 0.2, and the corresponding time weight is 1 minus the feature weight. This dynamic weighting mechanism avoids over-reliance on unstable statistical features when the sample size is small.

[0025] Understandably, the encoding method of the alignment marker sequence directly affects subsequent intent parsing. Each element in the sequence contains three parts of information: a gesture node index, a voice node index, and association strength. Through this structured representation, the specific content of each synchronization point can be traced. When it is necessary to understand the user's intent, the relevant gestures and voice segments can be quickly located, achieving accurate multimodal information fusion. This temporal synchronization mechanism enables smart glasses to accurately capture the user's interaction intent, maintaining a high recognition accuracy even in complex environments.

[0026] The starting and ending points of the gesture are marked in the gesture movement path. The acceleration and deceleration points of the gesture in the spatial movement process are identified as speed change nodes. At the same time, the start and end points of speech are located from the speech beat sequence. The natural pauses and breathing gaps in the speech are detected as speech pause points. The time correspondence between the start time of the gesture and the start time of speech is established. The distribution positions of the gesture speed change nodes and speech pause points on the time axis are matched to confirm whether the duration of the gesture is synchronized with the duration of speech. The time consistency between the acceleration and deceleration of the gesture and the rhythm of speech pauses and breathing is verified.

[0027] The gesture's starting point is identified by detecting the moment when the speed increases from zero in the speed curve of the gesture's movement path, and the ending point is identified by the moment when the speed drops to zero and remains stationary. Acceleration points are identified by calculating the speed difference between adjacent frames, where the difference changes from negative to positive, and deceleration points are identified by the moment the difference changes from positive to negative. The timestamps of these acceleration and deceleration points are recorded to form a speed change node sequence. Simultaneously, the energy curve of the speech beat sequence is used to locate the moment when the energy suddenly increases from the silence threshold as the start of speech, and the moment when the energy drops below the silence threshold as the end point. For the speech segment between the start and end points, natural pauses are identified by detecting breaks in the fundamental frequency trajectory. A pause is marked when the fundamental frequency value is continuously missing for more than a preset duration. Breathing intervals are identified by analyzing the proportion of low-frequency energy. When the proportion of energy below a preset frequency exceeds a preset ratio and the duration matches breathing characteristics, a breathing point is marked. Pauses and breathing points are merged to form a speech pause point set, and a time correspondence between the gesture's starting point timestamp and the speech's starting point timestamp is established. Based on the time correspondence, the time span from the start to the end of the gesture is calculated as the gesture duration, and the time span from the start to the end of the speech is calculated as the speech duration. If the absolute value of the difference between the two durations is less than a preset synchronization threshold, the speed change node sequence is divided into preset time windows, and the number of nodes in each window is counted as a distribution value. The same division and statistics are performed on the speech pause point set, and the correlation coefficient of the two sets of distribution values ​​is calculated as the overlap. When the overlap exceeds a preset overlap threshold, the time difference between each gesture acceleration point and its nearest speech pause point is calculated to form a first interval sequence, and the time difference between each gesture deceleration point and its nearest breathing gap is calculated to form a second interval sequence. The mean and variance of the two interval sequences are calculated respectively. If the mean is less than a preset rhythm threshold and the variance is less than a preset stability threshold, it is determined that the gesture and speech have time consistency, and time sequence verification information containing the time correspondence and consistency determination results is output.

[0028] For example, in one implementation, the detection of the gesture start and end points relies on the zero-point recognition of the velocity curve. The smart glasses continuously track the movement speed of key hand points, and when an increase in speed from a stationary state is detected, that moment is recorded as the gesture start point. Specifically, a velocity threshold of 5 millimeters per second is set; values ​​below this threshold are considered stationary. When the velocity values ​​of three consecutive frames exceed the threshold, the timestamp of the first frame is marked as the start point. The determination of the end point uses the opposite logic: when the velocity drops below the threshold and remains below it for three frames, it is marked as the end point.

[0029] Specifically, acceleration and deceleration points are identified by analyzing the sign changes of speed differences. For speed differences between two adjacent frames, a positive value indicates acceleration, and a negative value indicates deceleration. When the difference sequence changes from negative to positive, the transition point is the acceleration point, indicating that the user begins to speed up their gesture; the transition point from positive to negative is the deceleration point, indicating that the user begins to slow down their action. These speed change nodes usually correspond to the user's intention transition, such as from pointing to selecting. Detection of fundamental frequency trajectory breaks is a key technology for recognizing speech pauses. The fundamental frequency reflects the frequency of vocal cord vibration and exhibits a continuously changing trajectory during normal speech. When the user pauses, the vocal cords stop vibrating, and the fundamental frequency value cannot be detected, appearing as a break in the trajectory. The fundamental frequency value of each 10-millisecond speech segment is extracted using an autocorrelation function. When no valid fundamental frequency is detected for 50 consecutive milliseconds, it is determined to be a natural pause. This pause usually occurs between word boundaries or semantic units and is a natural rhythmic point in the user's language organization. The identification of breathing gaps is based on changes in spectral energy distribution; breathing sounds are mainly concentrated in the low-frequency band below 500 Hz. Calculate the spectrum of each frame of speech, and count the proportion of energy in the frequency band below 500 Hz to the total energy. When this proportion exceeds 80% and lasts for more than 100 milliseconds, it is identified as a breathing gap.

[0030] For example, in a scenario where a user says "select the blue icon on the left" and simultaneously points with a gesture, there will be a short pause in the speech between "select" and "left," while the gesture transitions from a ready state to a pointing action, creating an acceleration point. There might be a breathing gap between "blue" and "icon," during which the gesture remains pointing but decreases in speed, creating a deceleration point. The time correspondence is established by recording the timestamps of these key moments. The timestamp of the gesture's starting point and the timestamp of the speech's starting point form an initial mapping pair, reflecting the synchronicity of the user's interaction.

[0031] It should be noted that the synchronization judgment of duration involves a comparison of the overall time span. The duration of a gesture is calculated from the start to the end point, while the duration of speech is calculated from the start to the end of utterance. The difference between the two reflects the degree of coordination between the gesture and speech in the overall rhythm. The preset synchronization threshold is usually set at 500 milliseconds, which takes into account the natural error range of human multimodal coordination.

[0032] Preferably, a fixed-length sliding window method is used to divide the time window. The entire interaction period is divided into windows of 200 milliseconds each, with a 100-millisecond overlap between windows. For each window, the number of speed change nodes contained within it is counted as the gesture distribution value, and the number of speech pauses is counted as the speech distribution value. The two sets of distribution values ​​form two time series, and their similarity is assessed by calculating the Pearson correlation coefficient. A correlation coefficient close to 1 indicates that the rhythms of the gestures and speech are highly synchronized, while a correlation coefficient close to 0 indicates that they change independently.

[0033] In one possible implementation, the interval sequence is constructed using the nearest neighbor matching principle. For each gesture acceleration point, the nearest speech pause point is searched on the time axis, and the time difference between the two is calculated as the interval value. All interval values ​​are arranged in chronological order to form the first interval sequence. The second interval sequence, consisting of gesture deceleration points and breathing gaps, is constructed using the same method. The mean of the interval sequence reflects the average time difference between gestures and speech, ideally close to zero; the variance reflects the stability of the time difference, with a smaller variance indicating that the user has maintained a stable and coordinated rhythm. Furthermore, consistency determination comprehensively considers multiple dimensions of indicators. In addition to the statistical characteristics of the interval sequence, it also includes whether the overlap exceeds a preset threshold of 0.7, indicating that the gesture and speech rhythms remain synchronized for at least 70% of the time window. When all indicators meet the preset conditions, the system determines that the gesture and speech have temporal consistency, and the output temporal verification information includes a detailed time mapping table and various consistency indicator values, providing a reliable temporal basis for subsequent intent understanding.

[0034] S103. If the dynamic consistency score is higher than the preset consistency threshold, the direction information in the gesture motion path and the keyword position in the speech beat sequence are fused to generate gesture-speech fusion data. Otherwise, the original data from the acquisition stage is returned and gesture-speech fusion data is re-generated.

[0035] The speech beat sequence is decoded using speech recognition, converting the speech signal into a text sequence. Nouns, verbs, and adjectives are extracted from the text as candidate keywords. The start and end timestamps of each candidate keyword in the speech signal are recorded. High-frequency command words are extracted from the user's historical interaction records to construct a command word set. The candidate keywords are matched with the command word set, and successful matches are marked as valid keywords, forming a keyword sequence containing keyword text and time position. The dynamic consistency score is determined to be higher than a preset consistency threshold. If it is higher than the threshold, gesture data for the corresponding time period is extracted from the gesture movement path based on the time position of each keyword in the keyword sequence. The pointing direction vector sequence and spatial coordinate sequence within that time period are obtained, and the average value of the pointing directions is calculated as the main pointing direction. The keyword text, occurrence time, main pointing direction, and spatial coordinates are combined into a structured data record. The structured data records are arranged in chronological order, with time interval markers inserted between adjacent records. The semantic attributes of keywords are encoded into numerical vectors, and the spatial attributes of gestures, including direction and position, are encoded into spatial vectors. The semantic vectors and spatial vectors are concatenated to form a fusion feature vector, and the gesture-speech fusion data is output. If the dynamic consistency score is lower than the threshold, the original collected gesture image and speech signal are returned, the sampling frequency is increased, the recognition window is expanded, and the recognition process is re-executed to obtain the gesture-speech fusion data.

[0036] For example, in one implementation, the speech recognition process employs continuous speech recognition technology, segmenting the speech beat sequence into 10-millisecond frame lengths and extracting Mel-frequency cepstral coefficients as acoustic features from each frame. A Hidden Markov Model (HMM) is used to convert the acoustic feature sequence into a phoneme sequence, which is then decoded into text using a language model. Within the text sequence, part-of-speech tagging tools are used to identify nouns, verbs, and adjectives, which typically carry the core semantics of the instruction. Each identified word retains its temporal position information within the original speech signal, including start and end times, with millisecond-level accuracy.

[0037] Specifically, the command word set is constructed based on users' historical interaction data. The system analyzes command words that appeared more than 5 times in the past 100 successful interactions, such as operation words like "zoom in," "zoom out," "select," "move," and "delete," as well as directional words like "left," "right," "above," and "below." The matching of candidate keywords with the command word set uses an edit distance algorithm, allowing for a certain degree of pronunciation deviation; an edit distance of less than 2 is considered a successful match.

[0038] It should be noted that the threshold for the dynamic consistency score is typically set at 0.7, an empirical value derived from a large amount of user test data. When the score is higher than this threshold, it indicates good coordination between gestures and speech, allowing for reliable data fusion. When extracting gesture data for corresponding time periods, considering that gestures usually precede speech, the gesture data extraction window is formed by extending the keyword time position forward by 100 milliseconds and backward by 50 milliseconds.

[0039] For example, the semantic vector encoding employs word embedding technology, mapping each keyword to a 128-dimensional vector space. The spatial vector contains a 3D unit vector indicating the direction and 3D coordinates indicating the spatial location, totaling 6 dimensions. The two types of vectors are concatenated to form a 134-dimensional fused feature vector. This encoding method preserves the richness of semantic information while accurately representing the spatial features of the gesture.

[0040] Preferably, when the dynamic consistency score falls below a threshold, the system's adaptive adjustment mechanism is activated. The sampling frequency is increased from 60Hz to 120Hz, making gesture capture more precise; the recognition window is expanded from 200 milliseconds to 300 milliseconds, giving the user more reaction time. When re-executing the recognition process, the system also lowers the confidence threshold for speech recognition, accepting more candidate keywords and improving the matching success rate. Through this adaptive mechanism, even in the event of an initial recognition failure, the system can still obtain effective fusion data through parameter adjustments.

[0041] S104. Extract the spatial semantic pattern of the gesture and speech fusion data, and identify the core object of the user's pointing command based on the pointing coordinates of the gesture movement path and the pitch peak of the speech beat sequence.

[0042] Semantic and spatial vectors are parsed from gesture-speech fusion data. The cosine similarity between the semantic vector and the center vector of a preset category is calculated, and the category with the highest similarity is used as the semantic label. The pointing direction in the spatial vector is multiplied by the intrinsic parameter matrix of the smart glasses camera to obtain two-dimensional screen coordinates. Simultaneously, the timestamps corresponding to the pitch peaks are extracted from the speech beat sequence. Keyframes in the fusion data are located based on the timestamps, and semantic labels and screen coordinates of the keyframes are extracted to construct a spatial semantic mapping table. For the screen coordinates in the spatial semantic mapping table, a clustering algorithm is used to aggregate coordinate points with time intervals less than a preset time threshold and spatial distances less than a preset distance threshold into candidate regions. The density value is obtained by dividing the number of coordinate points in each candidate region by the region area. Regions with density values ​​exceeding a preset clustering threshold are marked as potential object regions. At the same time, the amplitude values ​​of the pitch peaks are extracted, and the time corresponding to the peak with the largest amplitude value is taken as the core time. The semantic label sequence within a preset time period before and after the core time is obtained. Based on the potential object region and semantic label sequence, image patches corresponding to the potential object region are extracted from the smart glasses scene image. The color histogram and edge features of the image patches are extracted, and object boundaries are identified through contour detection. Object categories are identified through a classifier, and string matching is performed between the object categories and labels in the semantic label sequence. Successfully matched objects are marked as candidate core objects. The Euclidean distance between the geometric center of the candidate core object and the gesture pointing coordinates is calculated. Based on the Euclidean distance and the peak amplitude value of the tone, the reciprocal of the distance is multiplied by the peak amplitude value after normalization to obtain an importance score. The candidate core object with the highest score is selected as the core object of the user's pointing command. The unique identifier, screen position coordinates, and object category name of the core object are extracted, and a recognition result containing complete information about the core object is output.

[0043] For example, in one implementation, cosine similarity calculation involves the ratio of the vector inner product to its magnitude. The semantic vector is typically a 128-dimensional floating-point array, with each dimension representing the strength of a semantic feature. Preset category center vectors are obtained by clustering a large number of samples, including typical interaction categories such as "selection," "operation," and "navigation." During calculation, the semantic vector to be classified is multiplied by each category center vector, and then divided by the product of the magnitudes of the two vectors. The resulting value ranges from -1 to 1, with values ​​closer to 1 indicating higher similarity. The category with the highest similarity is selected as the label for the semantic vector; this label determines the semantic context for subsequent processing. Specifically, the conversion from the pointing direction vector to screen coordinates needs to consider the optical parameters of the smart glasses. The camera intrinsic parameter matrix includes parameters such as focal length, principal point coordinates, and distortion coefficients, which are calibrated at the factory. The intrinsic parameter matrix can be:

[0044] K represents the camera intrinsic parameter matrix, f x and f y c represents the focal length parameters along the x-axis and y-axis, respectively. x and c y These represent the x and y coordinates of the principal point in the image coordinate system, respectively. This matrix contains the camera's internal geometric parameters. The 3D pointing vector is first normalized, then multiplied by the intrinsic parameter matrix to obtain homogeneous coordinates on the image plane, and then perspective division is performed to obtain the 2D pixel coordinates. Since the display screen of smart glasses is usually offset from the camera's field of view, coordinate system alignment is also required to map the pixel coordinates in the camera coordinate system to the display screen coordinate system.

[0045] It's important to note that the clustering algorithm uses a density-based approach when processing screen coordinates. The algorithm first defines a neighborhood radius, typically set to 5% of the screen width; points within this radius are considered neighbors. For each coordinate point, the number of points in its neighborhood is counted. If the number exceeds a minimum threshold, the point is marked as a core point. Core points and all their neighbors form a cluster; multiple connected core point neighborhoods are merged into the same cluster. Each cluster corresponds to a candidate region, and the region's boundary is determined by the smallest bounding rectangle of all points within the cluster. The density value is calculated by dividing the number of points in the cluster by the area of ​​the bounding rectangle, expressed as pixels per square meter.

[0046] For example, in a scenario where a user points to and says "open that app," the intonation peak typically occurs on the indicator word "that," and the magnitude of the peak reflects the user's emphasis. The system extracts data from 200 milliseconds before and after the peak as the core time period, during which the gesture direction is most explicit. The semantic label sequence contains all keywords identified within this time period, such as "open," "that," and "app," and these labels provide semantic clues for subsequent object category matching.

[0047] Preferably, image feature extraction employs multiple complementary methods. The color histogram quantizes the color space of the image patch into 256 intervals, counting the number of pixels in each interval to form a color distribution feature. Edge features detect abrupt changes in pixel intensity using gradient operators; the horizontal and vertical gradient magnitudes are combined to form an edge intensity map. Contour detection, based on the edge map, identifies closed boundary lines through connected component analysis; each boundary line corresponds to a potential object. The classifier uses a pre-trained convolutional neural network, taking the image patch as input and outputting a probability distribution of object categories. The category with the highest probability is used as the recognition result. When performing string matching with semantic tags, synonym substitution is allowed; for example, "application" can match "program" or "software".

[0048] In one possible implementation, the Euclidean distance calculation takes into account the characteristics of the screen coordinate system. The geometric center of the candidate core object is obtained by the arithmetic mean of the coordinates of all pixels within its contour, and the gesture pointing coordinates are the intersection of the pointing vector and the screen plane. The distance between two points is calculated in pixels; for a 1920×1080 resolution display, the diagonal length is approximately 2203 pixels. The reciprocal of the distance reflects the accuracy of the pointing; the closer the distance, the larger the reciprocal, indicating more accurate pointing. After normalization to a minimum and maximum, the reciprocal of the distance ranges from 0 to 1. Furthermore, the importance score calculation integrates information from both spatial and temporal dimensions. The spatial dimension is represented by the reciprocal of the distance, and the temporal dimension is represented by the peak amplitude of the tone. After normalization to a minimum and maximum, the peak amplitude ranges from 0 to 1, and the score obtained by multiplying it by the reciprocal of the distance considers both the spatial accuracy of the pointing and the degree of emphasis in the speech. When the scores of multiple candidate objects are close, the system also considers the object's salience on the screen, such as area size and brightness contrast.

[0049] Understandably, the unique identifier of the core object is obtained by encoding the object's feature vector using a hash function, ensuring that the same object has the same identifier when identified at different times. Screen position coordinates record the pixel positions of the object's center point and bounding box, and the object category name adopts a predefined category system, including interface element types such as "icon," "button," "text," and "image." This information constitutes the complete recognition result, enabling the system to accurately locate the core object that the user intends to point to, and achieve precise multimodal interaction.

[0050] S105. After identifying the core object pointing to the instruction, determine the user's intent by combining spatial semantic patterns and user history interaction records.

[0051] The system acquires the category attributes and screen position coordinates of the core object. It extracts the semantic tag sequence corresponding to the core object's time from the fused data containing semantic tags and spatial coordinates. Simultaneously, it retrieves historical operation records containing the same object category from the user's historical interaction records, counting the occurrence frequency of each operation type within that object category. A frequency matrix is ​​constructed, with the behavior object category as the behavior, the column as the operation type, and the element value as the occurrence frequency. Based on this frequency matrix, it extracts all operation frequency values ​​for the row corresponding to the current object category. Combining this with verb tags from the semantic tag sequence, it calculates the string similarity between each candidate operation and the verb tag. The operation confidence score is obtained by multiplying the operation frequency value by the similarity value. If the highest confidence score exceeds a preset intent threshold, the corresponding operation type is identified as the user intent, and the confidence score value of that intent is recorded. Based on the user intent and the confidence score, it searches for historical execution instances of the same intent from the historical interaction records. It extracts the operation parameters from these instances, including execution direction, execution amplitude, and duration. The current core object's position coordinates and category attributes are weighted and averaged with the historical parameters, with the weight determined by the confidence score value. The system outputs a complete user intent description containing the operation type, target object identifier, and set of execution parameters.

[0052] For example, in one implementation, the construction of the frequency matrix relies on a systematic analysis of historical interaction records. The historical records store each user interaction chronologically, including the operation timestamp, object category, operation type, and operation result. First, object categories are categorized, unifying subcategories such as "application icons," "system icons," and "file icons" into the broader category of "icons." Operation types include basic operations such as "click," "long press," "drag," "zoom," and "rotate." Each element of the matrix records the frequency of combinations of a specific object category and a specific operation type, obtained by accumulating the matching items in the historical records.

[0053] Specifically, string similarity is calculated using a combination of edit distance and longest common subsequence. For verb tags extracted from semantic tag sequences, such as "open," "start," and "run," the system maintains a thesaurus, mapping semantically similar words to a unified operation type. Similarity values ​​are normalized between 0 and 1, with a perfect match being 1 and a complete mismatch being 0.

[0054] It should be noted that the calculation of operation confidence integrates both statistical and semantic information. The frequency value reflects the user's historical preferences, while the similarity value reflects the clarity of the current voice command. The product of the two values ​​takes into account both personalization and the accuracy of the command. The preset intent threshold is usually set to 0.6; a value below this indicates that the system cannot determine the user's intent and further confirmation is required.

[0055] For example, in a scenario where a user points to a document icon and says "open," the system searches the frequency matrix for all operation frequencies corresponding to the "document" category. It finds that "double-click to open" has the highest frequency (85 times), followed by "open from right-click menu" (12 times). Simultaneously, the semantic similarity between "open" and "double-click to open" is 0.9, and the similarity with "open from right-click menu" is 0.7. The frequencies and similarities are then multiplied using min-max normalization to obtain a confidence score. The action corresponding to the highest confidence score is selected as the user's intent.

[0056] Preferably, the weighted average of the execution parameters takes into account individual differences and scene adaptability. Parameters in historical execution instances include the speed, amplitude, and duration of gesture movements, which reflect the user's operating habits. The weights are determined by normalizing the confidence values; the higher the confidence level, the greater the weight of the current recognition result, and the smaller the weight of the historical average.

[0057] S106. Generate execution instructions for object manipulation based on user intent, identify the dynamic path of the gesture trajectory and the rhythm of the voice, determine the instruction execution time through the dynamic path of the gesture trajectory and the rhythm of the voice, and generate a complete intent parsing result containing the target object, operation type and execution sequence.

[0058] Based on the operation type and target object identifier in the user's intent, the operation type is mapped to a predefined execution action code. The screen coordinates, category attributes, and execution action code of the target object are combined into a basic execution instruction. Simultaneously, the spatial coordinates of key point sequences are extracted from the gesture trajectory. The instantaneous velocity is obtained by dividing the displacement vector between adjacent key points by the time interval. The displacement vector sequence and the instantaneous velocity sequence are combined to construct dynamic path data. For the dynamic path data, pause points are identified by detecting local minima of instantaneous velocity, and turning points are identified by detecting changes in displacement vector direction exceeding a preset angle threshold. Pause points and turning points are used as action segment markers. The start time and duration of each beat are extracted from the speech beat sequence. The difference between the action segment marker time and the beat start time is calculated. If the difference is less than a preset synchronization threshold, a correspondence between the segment marker and the beat is established, and the execution start time of each action segment is determined. Based on the aforementioned correspondence and execution start time, the basic execution instruction is copied into an equal number of instruction copies according to the number of action segments. Each instruction copy is assigned the execution start time of the corresponding action segment as the trigger time, and the duration of the corresponding beat as the execution window. The instruction copies, trigger times, and execution windows are then bound together to form a sequence of time-sequential execution units. Through this sequence of time-sequential execution units, the execution action code, trigger time, and execution window of each unit are extracted. These are then combined with the target object identifier, screen position coordinates, and operation type name to assemble structured data containing complete execution information. A timestamp is inserted at the position aligned with the gesture and voice time, and a complete intent parsing result containing the target object, operation type, and execution sequence is output.

[0059] For example, the mapping of action codes employs a hierarchical encoding mechanism. Operation types are first divided into two main categories: basic operations and complex operations. Basic operations include single actions such as clicking, long-pressing, and dragging, encoded as 001 to 099; complex operations include actions requiring multi-point coordination, such as scaling, rotating, and sliding, encoded as 100 to 199. Each operation type corresponds to a unique three-digit code, facilitating subsequent instruction parsing and execution. The screen coordinates of the target object are represented using relative coordinates, with the screen center as the origin, and the horizontal and vertical coordinate values ​​normalized to the range of -1 to 1, avoiding compatibility issues between devices with different resolutions.

[0060] Specifically, the construction of dynamic path data involves information extraction from multiple dimensions. Key points in the gesture trajectory typically include 21 hand markers, with the system focusing on three key locations: the fingertip of the index finger, the fingertip of the thumb, and the center of the palm. The displacement vector between adjacent frames is obtained by calculating the coordinate difference between the corresponding key points in the current frame and the previous frame; the magnitude of the vector represents the distance traveled, and the direction represents the motion trend. The instantaneous velocity calculation takes into account the sampling frequency and is obtained by dividing the magnitude of the displacement vector by the frame interval, with units of pixels per second. To reduce the impact of noise, the system performs a moving average of the velocity values ​​from five consecutive frames to obtain a smooth velocity curve.

[0061] It's important to note that pause points and turning points employ different detection strategies. Pause point identification is based on local features of the velocity curve. When the instantaneous velocity is below a preset pause threshold for three consecutive frames, the intermediate frame is marked as a pause point. The pause threshold is typically set at 10% of the average velocity, but this percentage can be adjusted based on user habits. Turning point identification, on the other hand, is based on changes in the direction of displacement vectors. By calculating the angle between two adjacent displacement vectors, a change in direction is considered to have occurred when the angle exceeds 45 degrees. This dual detection mechanism accurately captures key feature points of gesture movements, providing a reliable basis for subsequent motion segmentation.

[0062] For example, when a user performs the action of "dragging an object to a specified location," the gesture trajectory exhibits a clear three-segment characteristic: the initial segment accelerates from a standstill to a stable movement, the middle segment maintains a relatively uniform dragging speed, and the final segment decelerates from movement to a standstill. The voice command "move this to there" also corresponds to three beats: "move this," "move to," and "there." A time alignment algorithm is used to align the initial segment with the first beat, the middle segment with the second beat, and the final segment with the third beat. During alignment, the difference between the timestamp of each action segment and the start time of the most recent beat is calculated; alignment is considered successful when the difference is less than 100 milliseconds.

[0063] Preferably, the sequential execution unit is constructed using template copying and parameter filling. The basic execution instruction serves as the template, containing the operation action code, target object identifier, and basic parameters. Based on the number of action segments, the system creates a corresponding number of instruction copies, each inheriting the basic information from the template. Trigger time allocation follows a chronological order principle; the trigger time for the first action segment is set to the current time, and the trigger times for subsequent action segments increase sequentially, with the increment equal to the duration of the corresponding beat. The execution window defines the effective execution period for each instruction copy; the window start time is the trigger time, and the window end time is the trigger time plus the beat duration.

[0064] In one possible implementation, the assembly of structured data follows predefined data. The data structure contains four main fields: `targetObject` stores the unique identifier and screen coordinates of the target object; `operationType` records the operation type name and execution action code; `executionTimeline` contains a sequence of time-sequential execution units, each recording the trigger time and execution window; and `syncMarkers` marks the synchronization points between gestures and speech, each marker containing a timestamp and synchronization type. Furthermore, the insertion position of the timestamp marker corresponds to the moment when the gesture action and the speech beat are successfully aligned. The marker types are divided into strong synchronization and weak synchronization. Strong synchronization indicates that the time difference between the gesture action and the speech beat is less than 50 milliseconds, while weak synchronization indicates a time difference between 50 and 100 milliseconds. These markers play a crucial role in the system's instruction execution; instructions at strong synchronization points have higher priority and are executed more precisely.

[0065] Understandably, the output format of the complete intent resolution result adopts a hierarchical structure. The top layer contains session identifiers and timestamps for tracking and playback; the middle layer contains core execution information, namely the target object, operation type, and execution sequence; and the bottom layer contains detailed execution parameters and synchronization flags.

[0066] S107. Output the complete intent parsing result to the smart glasses display module to perform the corresponding operation, and at the same time feed it back to the acquisition module to adjust the next capture parameters, including the acquisition frequency and recognition sensitivity, to improve the response speed and accuracy.

[0067] The target object identifier, operation type, and execution sequence from the complete intent parsing result are transmitted to the smart glasses display module via a preset communication interface. The display module overlays a semi-transparent highlight frame at the corresponding position based on the screen coordinates of the target object as an execution confirmation mark. Simultaneously, it triggers the corresponding interface operation response according to the execution sequence, records the time difference between the time the operation command is issued and the time the interface response is completed as the execution delay, and calculates the ratio of the number of correctly recognized commands to the total number of commands in this interaction as the recognition accuracy. Based on the execution delay and recognition accuracy, the difference between the execution delay and the preset standard delay is taken as the performance deviation, and the difference between the recognition accuracy and the preset target accuracy is taken as the accuracy deviation. If the performance deviation exceeds the preset delay threshold or the accuracy deviation is negative, the current acquisition frequency and recognition sensitivity value are extracted. The performance deviation is multiplied by a preset frequency adjustment coefficient to obtain the frequency adjustment amount, and the absolute value of the accuracy deviation is multiplied by a preset sensitivity adjustment coefficient to obtain the sensitivity adjustment amount. By adding the frequency adjustment amount to the current acquisition frequency to obtain the new acquisition frequency, and adding the sensitivity adjustment amount to the current recognition sensitivity to obtain the new recognition sensitivity, the new acquisition frequency is written into the frequency configuration parameters of the acquisition module, and the new recognition sensitivity is written into the threshold parameters of the recognition algorithm to obtain the updated capture parameter configuration and apply it to the next round of gesture voice acquisition and recognition process.

[0068] For example, in one implementation, the communication interface employs a shared memory mechanism to achieve high-speed data transmission. The complete intent parsing result is written to the shared memory buffer in the form of a structured data packet. The packet header includes a timestamp, data length, and checksum, ensuring transmission integrity and real-time performance. The display module detects the buffer status via polling, immediately reading and parsing new data upon detection. The semi-transparent highlight frame is rendered using an overlay blending mode with 50% transparency, a 3-pixel border width, and its color dynamically adjusted based on the operation type: green for clicks, blue for drags, and orange for zooms.

[0069] Specifically, the execution latency is measured with millisecond-level precision. The system records the current value of the system clock as the start time when sending an operation command, and records the end time when a anticipated change in the state of an interface element is detected. The difference between these two times is the execution latency. Accuracy statistics are based on the entire interaction session. Each time the user issues a command, the system determines whether the target object and operation type have been correctly identified. A correct identification count is incremented by one, and the accuracy rate is finally obtained by dividing the number of correct identifications by the total number of commands.

[0070] It should be noted that the adjustment coefficients were set considering both system stability and convergence speed. The frequency adjustment coefficient is typically set to 0.1, meaning that a delay deviation of one millisecond will result in a 0.1Hz adjustment of the acquisition frequency. The sensitivity adjustment coefficient is set to 0.05 to ensure the gradual nature of parameter adjustments. These coefficient values ​​are empirical values ​​derived from extensive experimental optimization, ensuring both rapid response to performance changes and preventing system oscillations.

[0071] For example, when the system detects an execution latency of 150 milliseconds, while the standard latency is 100 milliseconds, the performance deviation is 50 milliseconds. If the current acquisition frequency is 30Hz, after applying the frequency adjustment coefficient, the new acquisition frequency is adjusted to 35Hz. Meanwhile, if the recognition accuracy is 85%, lower than the target accuracy of 95%, the accuracy deviation is -10%, and after applying the sensitivity adjustment coefficient, the recognition sensitivity parameter increases by 0.5.

[0072] Preferably, a gradual strategy is adopted for parameter updates to avoid the impact of sudden changes on system stability. The new parameters do not immediately and completely replace the old parameters, but are gradually transitioned through a weighted average method, with the weight ratio increasing over time, and the parameters are fully updated within 5 acquisition cycles.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. The present invention has been described in detail with reference to preferred embodiments. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications and substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A smart glasses AI voice interaction method, characterized in that, include: The system collects user gesture images and voice signals, and simultaneously obtains user historical interaction records. It extracts the key point coordinates and pointing direction information of the gesture images to generate gesture movement paths, and extracts the spectral information and pitch peaks of the voice signals to form a voice beat sequence. After performing time-series synchronization processing on the gesture motion path and the speech beat sequence, the speed change nodes in the gesture motion path and the pause points in the speech beat sequence are extracted to generate a dynamic consistency score. Based on the dynamic consistency score being higher than a preset threshold, the pointing direction of the gesture motion path and the keyword positions in the speech beat sequence are fused to generate gesture-speech fusion data. The spatial semantic pattern of the gesture-speech fusion data is analyzed, and the core object of the user's pointing command is identified based on the pointing direction of the gesture motion path and the intonation peaks of the speech beat sequence. After identifying the core object pointing to the command, the user's intent is determined by combining spatial semantic patterns and user history interaction records; Based on the user's intent, an execution instruction containing the target object, operation type, and execution sequence is generated; the execution instruction is then transmitted to the smart glasses display module for execution.

2. The AI ​​voice interaction method for smart glasses according to claim 1, characterized in that, The process involves acquiring user gesture images and voice signals, simultaneously obtaining user historical interaction records, extracting key point coordinates and pointing direction information from the gesture images to generate gesture movement paths, and extracting the spectral information and pitch peaks from the voice signals to form a voice beat sequence, including: The system captures an RGB image stream of the gesture image, records the voice signal, and reads the gesture trajectory template and voice command template from the user's historical interaction record in local storage. Based on the RGB image stream, it calculates the three-dimensional coordinates of key points on the hand, determines the pointing direction vector of the line connecting the index fingertip and the wrist, and generates an absolute position sequence of the gesture movement path. It performs spline interpolation on the absolute position sequence, connects the key point coordinates to generate a smooth path, calculates the path curvature change rate, and marks the turning points to form the gesture movement path. It performs time-frequency analysis on the voice signal, generates a spectrogram, calculates the weighted average frequency of the spectrum for each frame, detects the abrupt change points of the spectral centroid as the pitch peaks, and forms the voice beat sequence.

3. The AI ​​voice interaction method for smart glasses according to claim 1, characterized in that, After performing time-series synchronization processing on the gesture motion path and the speech beat sequence, the velocity change nodes in the gesture motion path and the pause points in the speech beat sequence are extracted respectively, including: calculating the displacement difference between adjacent frames in the gesture motion path, generating an instantaneous velocity sequence, marking the extreme points of velocity in three consecutive frames as velocity change nodes, and recording the timestamps of the velocity change nodes; analyzing the audio energy envelope based on the speech beat sequence, calculating the background noise level, and marking the time points when the audio energy is lower than the background noise level as the pause points.

4. The AI ​​voice interaction method for smart glasses according to claim 3, characterized in that, Generating a dynamic consistency score includes: constructing a time correlation matrix using the timestamps of the speed change nodes as row indices and the timestamps of the pause points as column indices; calculating the absolute values ​​of time intervals and filling them into matrix elements; marking node pairs with intervals less than a preset synchronization window as strongly correlated, and calculating the ratio of the number of strongly correlated node pairs to the total number of node pairs as the preliminary correlation degree; extracting the time offset sequences of strongly correlated node pairs and calculating the standard deviation of the time offset sequences; if the standard deviation is less than a preset stability threshold, calculating the speed difference before and after the speed change node from the gesture motion path as the gesture change intensity, calculating the frequency difference before and after the pause point from the speech beat sequence as the pitch change intensity, calculating the covariance and variance of the sequences of the two change intensities of gesture change intensity and pitch change intensity, and combining the preliminary correlation degree to determine the dynamic consistency score of gesture-speech coordination.

5. The AI ​​voice interaction method for smart glasses according to claim 1, characterized in that, Based on the dynamic consistency score being higher than a preset threshold, gesture-speech fusion data is generated by fusing the pointing direction of the gesture movement path with the keyword position of the speech beat sequence. This includes: performing speech recognition on the speech beat sequence to generate a text sequence, extracting nouns, verbs, and adjectives as candidate keywords, and recording the start and end timestamps of the candidate keywords; extracting high-frequency command words from the user's historical interaction records, matching the candidate keywords, and generating a keyword sequence containing keyword text and time position; when the dynamic consistency score is higher than the preset threshold, extracting the pointing direction vector and spatial coordinate sequence corresponding to the keyword sequence time from the gesture movement path, and calculating the average pointing direction of the keywords; combining the keyword text, time position, average pointing direction, and spatial coordinates and inserting time interval markers to output the gesture-speech fusion data.

6. The AI ​​voice interaction method for smart glasses according to claim 5, characterized in that, Before outputting the gesture-speech fusion data, the process includes: encoding the semantic attributes of keywords into semantic vectors, encoding the spatial attributes of gestures, including pointing direction and position, into spatial vectors, and concatenating the semantic vectors and spatial vectors to form a fusion feature vector.

7. The AI ​​voice interaction method for smart glasses according to claim 6, characterized in that, The process of parsing the spatial semantic pattern of the gesture-speech fusion data, and identifying the core object of the user's pointing command based on the direction of the gesture movement path and the intonation peak of the speech beat sequence, includes: The semantic and spatial vectors in the gesture-speech fusion data are analyzed, and the cosine similarity between the semantic vector and the preset semantic category center is calculated to determine the semantic label. The pointing direction in the spatial vector is converted into screen coordinates, and keyframes corresponding to the pitch peak are extracted to construct a spatial semantic mapping table. For the screen coordinates in the spatial semantic mapping table, the screen coordinates are clustered to generate candidate regions. The density value is obtained by dividing the number of coordinate points in each candidate region by the region area. Regions with a density value exceeding a preset clustering threshold are marked as potential object regions. The core time period is determined based on the pitch peak. Within the core time period, image patches corresponding to the potential object regions are extracted from the smart glasses scene image. The core object of the user's pointing command is identified by combining the semantic label sequence within a preset time period before and after the core time.

8. The AI ​​voice interaction method for smart glasses according to claim 1, characterized in that, After identifying the core object pointing to the command, the user's intent is determined by combining spatial semantic patterns and user history interaction records, including: Retrieve historical operation records containing the same object category from the user's historical interaction records, count the number of times each operation type appears in the object category, and construct a frequency matrix with the behavior object category, column as operation type, and element value as the number of occurrences; based on the frequency matrix, extract all operation frequency values ​​in the row corresponding to the current object category, combine them with verb tags in the semantic tag sequence, calculate the string similarity between each candidate operation and the verb tag, and determine the user intent based on the operation frequency value and string similarity.

9. The AI ​​voice interaction method for smart glasses according to claim 1, characterized in that, The step of generating execution instructions containing the target object, operation type, and execution sequence according to the user intent includes: The user's intention operation type is mapped to an execution action code, and a basic execution instruction is generated by combining the screen coordinates of the target object. The displacement vector and instantaneous velocity are extracted from the gesture movement path, and pause points and turning points are marked. The pause points and turning points are used as action segment markers, and a correspondence between the segment markers and the beat is established. The execution start time of each action segment is determined. The basic execution instruction is copied into the same number of instruction copies according to the number of action segments. Each instruction copy is assigned the execution start time of the corresponding action segment as the trigger time, and the duration of the corresponding beat is assigned as the execution window. The instruction copies, trigger times, and execution windows are bound to form a time-sequential execution unit sequence to generate the execution instruction.

10. The AI ​​voice interaction method for smart glasses according to claim 9, characterized in that, After generating the execution instructions, the process also includes: The execution action code, trigger time, and execution window of each time-sequential execution unit are extracted from the sequence of execution units. Combined with the target object identifier, screen position coordinates, and operation type name, the data is assembled into structured data, and a complete intent parsing result containing the target object, operation type, and execution sequence is output.

Images