Intelligent glasses control method, intelligent glasses, computer device, computer readable storage medium and computer program product

Abstract

This application relates to a smart glasses control method, smart glasses, a computer device, a computer-readable storage medium, and a computer program product. The method includes: acquiring voice data of the wearer of the smart glasses via a first microphone and acquiring body motion data of the wearer via a first radar; performing voice feature recognition on the voice data to obtain voice keywords output by the wearer; performing motion feature recognition on the body motion data to obtain the wearer's gesture trajectory; identifying the wearer's control intention based on the voice keywords and gesture trajectory to obtain the wearer's target control intention for the smart glasses; and controlling the smart glasses according to the target control intention. This method enables stable, accurate, and low-latency natural interaction with the smart glasses.

Description

Technical Field

[0001] This application relates to the field of smart glasses technology, and in particular to a smart glasses control method, smart glasses, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of wearable computing and natural human-computer interaction technologies, smart glasses, as an emerging head-mounted interactive terminal, have been widely used in fields such as industrial inspection, outdoor operations, remote collaboration, and special equipment.

[0003] Currently, the mainstream interaction method for smart glasses is primarily single-modal interaction. For example, voice interaction uses a microphone to collect user voice commands. However, in practical applications, voice interaction is severely limited by environmental noise and echo interference. The fundamental challenge of visual gesture interaction lies in its reliance on ambient lighting and its inability to fundamentally distinguish between hands and complex backgrounds.

[0004] Therefore, the smart glasses control schemes in related technologies suffer from insufficient reliability in complex environments. Summary of the Invention

[0005] Therefore, it is necessary to provide a smart glasses control method, smart glasses, computer equipment, computer-readable storage medium, and computer program product that can achieve stable, accurate, and low-latency natural interaction for smart glasses, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a smart glasses control method, the method being applied to smart glasses; the smart glasses are equipped with a first microphone and a first radar; the first microphone includes a bone conduction microphone for detecting head vibration data when the wearer of the smart glasses outputs speech; the first radar includes a millimeter-wave radar for identifying body movement data of the wearer; the method includes:

[0007] The first microphone is used to collect the voice data of the wearer of the smart glasses, and the first radar is used to collect the body movement data of the wearer.

[0008] The speech data is subjected to speech feature recognition to obtain the speech keywords output by the wearer;

[0009] Motion feature recognition is performed on the body motion data to obtain the gesture trajectory of the wearer;

[0010] Based on the voice keywords and the gesture trajectory, the control intention of the wearer is recognized to obtain the target control intention of the wearer for the smart glasses;

[0011] The smart glasses are controlled according to the stated target control intent.

[0012] Secondly, this application also provides a smart glasses, which includes a first microphone and a first radar; the first microphone includes a bone conduction microphone for detecting head vibration data when the wearer of the smart glasses outputs speech; the first radar includes a millimeter-wave radar for recognizing body movement data of the wearer; the smart glasses include:

[0013] The acquisition module is used to acquire voice data of the wearer of the smart glasses through the first microphone, and to acquire body movement data of the wearer through the first radar;

[0014] The first recognition module is used to perform voice feature recognition on the voice data to obtain the voice keywords output by the wearer;

[0015] The second recognition module is used to perform motion feature recognition on the body motion data to obtain the gesture trajectory of the wearer;

[0016] The third recognition module is used to recognize the control intention of the wearer based on the voice keywords and the gesture trajectory, so as to obtain the target control intention of the wearer for the smart glasses;

[0017] A control module is used to control the smart glasses according to the target control intention.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps included in any of the foregoing method embodiments.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0021] The aforementioned smart glasses control method, smart glasses, computer equipment, computer-readable storage medium, and computer program products, through the bone conduction microphone and millimeter-wave radar built into the smart glasses, solve the problem of traditional interaction failure in harsh environments from the sensor level. Based on the above two highly robust sensing modalities, this application uses voice keywords to carry command categories and gesture trajectories to carry spatial parameters, realizing logical decoupling and joint decoding of command semantics and spatial parameters. The inherent defect of incomplete information in a single modality is compensated by the complementary modality introduced in the embodiments of this application, achieving stable, accurate, and low-latency natural human-computer interaction in harsh environments while protecting user privacy. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an application environment diagram of a smart glasses control method in one embodiment;

[0024] Figure 2 This is a flowchart illustrating a smart glasses control method in one embodiment;

[0025] Figure 3 This is a structural block diagram of smart glasses in one embodiment;

[0026] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0028] Explanation of related terms:

[0029] Smart glasses: A head-mounted wearable computing device that integrates a display module, camera module, audio module and various sensors, capable of providing the wearer with augmented reality, virtual reality or mixed reality experiences, and supports control through various interaction methods.

[0030] Bone conduction microphone: A sound pickup device that uses vibrations of the skull, cheekbones, or other skeletal structures to collect sound signals. Unlike traditional air conduction microphones, bone conduction microphones are not sensitive to environmental noise and can clearly capture the voice commands of the wearer in high-noise environments such as industrial workshops, outdoor work areas, and transportation hubs.

[0031] Millimeter-wave radar: Radar sensors operating in the 30GHz to 300GHz frequency band. Frequency-modulated continuous wave (FMCW) radar in the 60-77GHz band transmits electromagnetic waves modulated with a continuous frequency and receives the reflected echoes from targets, enabling precise measurement of target distance, velocity, angle, and other information. Millimeter-wave radar is unaffected by lighting conditions and has a certain penetrating ability through textiles and thin sheets, allowing for stable detection of hand movements.

[0032] Multimodal fusion: a technique that combines sensor data from two or more different modalities at the feature layer or decision layer.

[0033] Transformer model: A neural network architecture based on self-attention mechanism, consisting of multiple stacked encoder layers. The lightweight Transformer model contains 4 encoder layers, each with 8 attention heads, and a hidden layer dimension of 256. After compression and quantization, it is suitable for running on embedded devices.

[0034] Before describing the embodiments of this application, the relevant technologies and their existing problems will be further explained:

[0035] The mainstream interaction methods for smart glasses can be categorized into two types: voice interaction and visual gesture interaction. Voice interaction uses a microphone to collect user voice commands, converts them into text using speech recognition technology, and executes the corresponding operations. This method conforms to natural human communication habits, requires no hands, and has good recognition accuracy in quiet environments.

[0036] However, in practical applications, voice interaction is severely hampered by environmental noise and echo interference. In scenarios such as industrial workshops, transportation hubs, and outdoor streets, background noise often reaches 85dB to 110dB, strongly masking the user's voice signal. The pickup diaphragm of traditional air-conducting microphones is directly exposed to ambient air pressure waves, resulting in a severely degraded signal-to-noise ratio. Even with advanced voice enhancement algorithms, it is difficult to reliably recover voice commands from strong noise, leading to a sharp drop in recognition rate, frequent false triggers or rejections, and seriously affecting the usability of the device.

[0037] Correspondingly, visual gesture interaction uses a camera integrated into smart glasses to capture images of the hand and leverages computer vision technology to recognize preset gesture trajectories, enabling air-based control. This method is intuitive and suitable for expressing concrete commands such as spatial pointing and area selection. However, the fundamental challenge of visual gesture interaction lies in its reliance on ambient lighting and its inability to fundamentally distinguish the hand from complex backgrounds. In low-light conditions such as dim lighting, backlighting, or strong exposure, the camera's image quality deteriorates sharply, making it difficult to extract hand features. In scenarios with cluttered background textures, hand skin tones similar to the background, or partial occlusion, image segmentation algorithms suffer from pathological problems, struggling to accurately separate the hand area from the background at the pixel level. Furthermore, continuous operation of visual recognition algorithms consumes significant power on embedded devices, making it difficult to support all-day wear.

[0038] To address the reliability issues of single-modal interaction in complex environments, exploratory research on multimodal interaction has emerged. For example, some solutions attempt to fuse speech recognition and visual gesture recognition at the decision layer, i.e., recognizing them separately and then combining and matching them using a rule table. However, such solutions are essentially still simple parallel connections of two single-modal systems, failing to fundamentally solve the problems of noise and illumination interference at the sensor level. Furthermore, the predefined combination rules are insufficient to cover all possible command variations, limiting the system's scalability.

[0039] Therefore, how to achieve stable, accurate, and low-latency natural interaction of smart glasses in harsh environments with extreme noise, variable lighting, and complex backgrounds is a technical problem that urgently needs to be solved in this field.

[0040] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0041] The smart glasses control method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Taking smart glasses as an example, the smart glasses have a miniature bone conduction microphone embedded in the area where the temples contact the temporal bone of the head, serving as the first microphone; a 60-77GHz millimeter-wave radar antenna module is integrated on the front of the frame or the outer side of the temples, serving as the first radar; the smart glasses may also include a low-power embedded processor, an embedded AI acceleration unit, a local encrypted storage chip, a camera module, a display module, and an audio feedback module. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0042] In one exemplary embodiment, such as Figure 2 As shown, a smart glasses control method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0043] Step 202: Collect voice data of the wearer of the smart glasses through the first microphone, and collect body movement data of the wearer through the first radar.

[0044] In this application, bone conduction microphones are designed to effectively suppress ambient noise transmitted through the air. For example, in an industrial workshop environment with 85dB background noise, if the wearer speaks "take a picture" at a normal volume, the bone conduction microphone can still capture a clear vibration signal with a signal-to-noise ratio (SNR) of at least 25dB, while traditional air conduction microphones typically have an SNR below 5dB in the same environment. Therefore, this embodiment uses a bone conduction microphone built into the smart glasses to capture the wearer's sound signal. Specifically, when the wearer issues a voice command, the vocal cords vibrate and are transmitted to the ear and surrounding tissues through the skull, jawbone, and other skeletal structures. The bone conduction microphone is fitted tightly to the inside of the temple of the smart glasses, facing the temporal bone region, and converts the bone vibration into an electrical signal. Specifically, the sampling rate of the bone conduction microphone is set to 16kHz or higher, and the raw voice signal it captures is denoted as S_v, which is represented as a one-dimensional floating-point array that varies over time. To ensure the accuracy of subsequent recognition, the acquisition process is continuous, and the system maintains a circular data buffer with a length of 2 seconds to cover the duration of the complete voice command.

[0045] Correspondingly, the millimeter-wave radar antenna module operates in a frequency-modulated continuous wave (FMCW) mode. In a typical implementation, the center frequency of the radar transmitted signal is 60.5 GHz, the bandwidth is 4 GHz, the sweep period is 50 μs, and 256 points are sampled in each sweep period. The radar antenna is configured as 1 transmit and 3 receive or 2 transmit and 4 receive to achieve horizontal angle of arrival (AoA) estimation. When the wearer's hand moves within the radar's field of view, the electromagnetic waves emitted by the radar are reflected by the hand and captured by the receiving antenna. The raw radar echo data is denoted as S_r, and its data structure is a complex three-dimensional tensor, with dimensions represented as (Chirp number, sampling points, number of receiving antennas). Within a data frame (typically 50 ms), the radar can transmit 32 or 64 consecutive sweep pulses, forming a fast-time-slow-time two-dimensional data matrix.

[0046] It should be noted that millimeter-wave radar collects body motion data, not just hand data. In actual wearable scenarios, the radar's field of view covers the chest area of ​​the wearer. Through subsequent signal processing, the system can distinguish hand movements from other body movements (such as arm swinging or head turning during walking) based on distance, speed, and micro-Doppler features. In this embodiment, motion feature recognition focuses on the aerial hand gesture trajectory with a clear command intent.

[0047] Optionally, to ensure data time alignment accuracy, the microcontroller unit (MCU) of the smart glasses provides a unified hardware synchronization clock for the bone conduction microphone and millimeter-wave radar, controlling the sampling start time deviation to within 1ms. Each frame of voice data packet and each frame of radar data packet carry the same timestamp identifier to support subsequent multimodal feature alignment.

[0048] The challenge of visual gesture recognition technology lies in its reliance on ambient lighting and its inability to fundamentally distinguish between hands and complex backgrounds. Both RGB and depth cameras output signals containing a mixture of hand and background pixels, making the separation process essentially a complex image segmentation problem. This application addresses this by introducing millimeter-wave radar, shifting the perception dimension from optical reflection to radio frequency scattering. The electromagnetic waves emitted by the millimeter-wave radar actively illuminate the target, independent of any ambient light source. Simultaneously, the radar's range-Doppler processing mechanism, based on the Doppler velocity, can directly filter out the reflected energy of static backgrounds from the signal domain, eliminating the need for image segmentation algorithms to identify and remove the background. Furthermore, through the Doppler effect, millimeter-wave radar can output the instantaneous velocity of hand movements in real time and with precision. Velocity characteristics are revealed before the gesture displacement becomes apparent, reducing recognition latency. Moreover, in smart glasses applications, the intentions of fast and slow hand waves differ; therefore, the velocity information collected by millimeter-wave radar in this application helps achieve gesture precision grading.

[0049] Step 204: Perform speech feature recognition on the speech data to obtain the speech keywords output by the wearer.

[0050] The original speech signal S_v can be bandpass filtered, with the passband range set to 300Hz to 3400Hz. This frequency band covers the region where speech energy is most concentrated, while effectively filtering out low-frequency mechanical vibration noise (such as low-frequency shaking during walking) and high-frequency circuit noise. The speech feature tensor V is then input to an end-to-end speech recognition model deployed locally on the smart glasses. This speech recognition model uses a deep neural network structure, preferably a combination of a convolutional neural network and a connection-time classification (CTC) decoder, and has been fine-tuned for bone conduction speech characteristics. Specifically, the speech recognition model outputs a text sequence and its corresponding confidence score. Subsequently, the semantic keyword matching module compares the recognized text with a predefined set of keywords, thus obtaining the recognized speech keyword K (a string); and the recognition confidence score p_k of the keyword, ranging from [0,1]. For example, when the wearer says "take a picture," the speech recognition model outputs the text "take a picture," with a confidence score p_k = 0.92. The predefined keyword set is pre-configured based on the application scenarios of smart glasses. For example, based on the characteristics of the usage scenarios of smart glasses, the keyword set may include: taking a photo, recording a video, zooming in, zooming out, going back, confirming, closing, navigation, increasing volume, decreasing volume, etc.

[0051] Considering the drawback of air conduction microphones—that their pickup diaphragm is directly exposed to ambient air pressure waves—in scenarios such as industrial workshops, transportation hubs, and outdoor operations, environmental noise and speech signals are superimposed on the same physical medium, resulting in a severely degraded signal-to-noise ratio. This pushes the backend to the theoretical limit of recovering weak signals from strong noise. Therefore, in this embodiment, a bone conduction microphone is introduced to switch the signal pickup path from air conduction to solid conduction. The speech signal is transmitted to the microphone's sensitive element through the wearer's skull, jawbone, and other skeletal structures, while environmental noise, primarily in the form of air pressure waves, is difficult to effectively couple to the solid medium of the bone. This physical path isolation allows the bone conduction microphone to still acquire speech signals with a signal-to-noise ratio of no less than 20dB even in an industrial noise environment of 110dB.

[0052] Meanwhile, considering that smart glasses are used in scenarios such as industrial inspection and special operations, wearers typically need to perform hand operations simultaneously, making it impossible to free up fingers to press the call button or to shout loudly near their mouths. The bone conduction microphone, placed close to the temporal bone, responds to soft whispers, murmurs, and even silent reading achieved through muscle tension alone. This allows the embodiments of this application to support low-volume or even silent commands, meeting the needs for concealment and convenience in special scenarios.

[0053] Step 206: Perform motion feature recognition on the body motion data to obtain the gesture trajectory of the wearer.

[0054] In this process, a Fast Fourier Transform (FFT) is performed on the original radar echo data S_r in each frequency sweep period (fast time) dimension to convert the time-domain echo signal into a range-dimensional spectrum, obtaining the target's range information relative to the radar. The range resolution is determined by the signal bandwidth; in this embodiment, a bandwidth of 4 GHz corresponds to a range resolution of approximately 3.75 cm. Subsequently, a second FFT is performed on the complex sequence within the same range gate in the slow time dimension to extract the target's radial velocity information. The velocity resolution and the maximum unambiguous velocity are determined by the frame period and carrier frequency. Through the above two FFT transformations, a Range-Doppler Map (RDM) is obtained. This map is a two-dimensional complex matrix, with the horizontal axis representing the range gate and the vertical axis representing the Doppler frequency (corresponding to radial velocity). The amplitude of each pixel in the map represents the probability intensity of the presence of a reflecting target at that range and velocity.

[0055] Furthermore, to obtain the spatial orientation information of the target, embodiments of this application estimate the angle of arrival (AHA) of the phase difference between multiple receiving antennas. Digital beamforming (DBF) or multiple signal classification (MUSIC) algorithms are used to calculate the horizontal AHA on each effective detection unit of the range-Doppler spectrum. Thus, the two-dimensional RDM is extended into a three-dimensional range-Doppler-angle data tensor R, with dimensions represented as (D, V, A), corresponding to the number of range units, the number of Doppler units, and the number of angle units, respectively.

[0056] Millimeter-wave radar simultaneously receives reflected signals from moving targets (hands) and static backgrounds (walls, equipment, furniture, etc.). In the range-Doppler spectrum, the signal reflected from the static background appears as a strong energy point with a Doppler frequency of zero or near zero. By subtracting the corresponding pixels from the range-Doppler spectra of two consecutive frames, the energy is significantly attenuated after subtraction because the complex values ​​of the static target remain unchanged between frames; however, the echo from the moving hand retains relatively strong energy after subtraction due to phase changes with displacement. After static clutter suppression processing, most of the static background energy is filtered out in the output range-Doppler spectrum, retaining only the sparse energy points corresponding to the moving target. Constant false alarm rate (CFAR) detection is then performed on the suppressed spectra of multiple consecutive frames to extract the effective detection points corresponding to the hand. The positions of these detection points in the range-velocity-angle space change continuously with frame time, forming a dynamic point cloud sequence in three-dimensional space.

[0057] The dynamic point cloud sequence is then organized along the time axis to construct a four-dimensional feature tensor R_seq, with dimensions (T, D', V', A'), where T is the number of time frames, and D', V', and A' are the subsets of distance, Doppler, and angle units retained after CFAR detection, respectively. This four-dimensional tensor is input into a pre-trained gesture recognition neural network model. The model's output layer is a fully connected network with the same output dimension as the predefined number of gesture categories. After normalization using the Softmax function, the probability distribution of each gesture category is obtained. In the output of the gesture recognition model, in addition to the gesture category G, a 128-dimensional trajectory feature vector g is generated. This vector g represents the hidden state of the last time step of the LSTM, or the global motion feature encoding after attention weighting, used for subsequent multimodal fusion.

[0058] Step 208: Based on the voice keywords and the gesture trajectory, the control intent of the wearer is recognized to obtain the target control intent of the wearer for the smart glasses.

[0059] Among them, voice keywords provide the type of instruction the user wants to do, such as taking a picture, zooming in, or navigating; while gesture trajectories provide the spatial parameters and objects the user wants to manipulate, such as the position and size of the frame, the rate of change of the pinch distance, and the direction and speed of the swipe.

[0060] Taking the photo-taking command as an example: If only the voice keyword "take a photo" is recognized, the system can only execute the default central area photo capture, and cannot know which specific area the user wants to photograph. If only the gesture drawing a frame is recognized, the system can only perceive that the user has drawn a rectangle in the air, but cannot determine whether the action is an intention to take a photo, an intention to select text, or an intention to define a search area. Therefore, only when the voice keyword and the gesture trajectory are presented together can the semantic integrity of the intention be better guaranteed. Thus, compared with the control scheme of smart glasses based on single-modal data in related technologies, the voice modality and the gesture modality are naturally complementary in terms of information expression dimensions. Voice is good at expressing abstract command categories, temporal sequences, and logical relationships; gestures are good at expressing concrete spatial parameters, motion trajectories, and quantification amplitudes.

[0061] A single modality inherently possesses semantic ambiguity in specific scenarios. For example, the referent of a voice command is unclear; a gesture of drawing a circle could represent confirmation or rotation. This application's embodiments resolve ambiguity through cross-modal constraints: a gesture of drawing a circle paired with voice confirmation is decoded as a confirmation operation; a gesture of drawing a circle paired with voice rotation is decoded as rotation zoom. The voice modality constrains the semantic scope of the gesture modality, and the gesture modality provides a spatial anchor for the referent of the voice modality.

[0062] The specific intent recognition process may include: First, assessing whether there is a temporal and spatial correlation between the speech keywords and the gesture trajectory in the current frame. Regarding temporal correlation, assessing whether the time difference between the end time of the speech keyword and the key event time of the gesture trajectory is less than a preset threshold, typically 500 milliseconds. If the time difference is too large, it is determined to be two independent instructions, and forced fusion is not performed. Regarding spatial correlation, for instructions involving spatial positioning such as selection or pointing, assessing whether the center position of the gesture trajectory is logically related to the region of interest in the current display field of view.

[0063] Based on the voice keyword, one or more candidate intents associated with the keyword are retrieved from a predefined intent-instruction mapping table. Each candidate intent includes the expected shape constraint of the gesture trajectory. Taking taking a photo with the voice keyword as an example, the candidate intents returned by the mapping table include: full-frame photo, with the gesture shape constraint being no specific gesture or a confirmation gesture; frame-selection photo, with the gesture shape constraint being a rectangular trajectory and the trajectory closure reaching a preset threshold; and burst photo, with the gesture shape constraint being rapid back-and-forth waving and the movement speed and number of back-and-forth movements reaching preset thresholds.

[0064] The matching degree between the actual recognized gesture trajectory and the gesture shape constraints corresponding to each candidate intent is calculated. The candidate intent with the highest matching degree and exceeding the preset matching threshold is selected as the target control intent I. Simultaneously, the control parameter θ associated with this intent is parsed from the gesture trajectory. Parameter extraction rules are bound to the intent type: for the box-selection and photo-taking intent, the smallest bounding box of the rectangle is extracted and converted into coordinates and dimensions in the display coordinate system; for the zoom intent, the start and end spacing of the pinch gesture is extracted, and the zoom factor is calculated; for the swipe-to-turn-page intent, the horizontal displacement of the swipe is extracted, the direction is determined, and it is converted into a previous or next page command.

[0065] Optionally, when logical contradictions arise in the bimodal information, such as a voice command to shrink but a gesture trajectory to enlarge, the system should make a decision based on a preset conflict resolution strategy. The strategy adopted in this embodiment is: using voice keywords as the primary instruction category and gesture trajectory as a parameter correction reference. That is, the operation type specified by the voice is executed first, while simultaneously evaluating whether the gesture trajectory conforms to the typical parameter form of that operation type; if there is a significant deviation, the fusion confidence is reduced and the user is required to reissue the command.

[0066] Considering that in extremely noisy environments, although bone conduction microphones can capture speech, the confidence level of the speech recognition model may drop to a critical range. This application's embodiments introduce gesture trajectories as a second modality, forming a redundant channel for intent expression. Even if the speech recognition confidence level is in the critical range of 0.6 to 0.8, if the gesture trajectory highly matches the logical constraints of the speech keyword, the system can still determine the intent is credible, avoiding frequent rejections due to fluctuations in the quality of a single modality. This application's embodiments achieve logical decoupling of instruction semantics and spatial parameters by using speech keywords to carry instruction types and gesture trajectories to carry spatial parameters. Adding a new way to express spatial parameters, such as drawing triangles or arrows, does not require modifying the speech recognition model; adding a new instruction type, such as ranging or labeling, does not require adding new gesture categories, only supplementing constraint rules in the intent mapping table. This redundant confirmation mechanism significantly improves the robustness of the system without increasing the user's cognitive burden.

[0067] Step 210: Control the smart glasses according to the target control intention.

[0068] In order to convert the identified user intent into specific device operation commands and provide execution feedback to the user, the operating system of the smart glasses maintains an intent-service mapping table in this embodiment. When the intent recognition module outputs the target control intent I and associated parameters θ, the system service scheduler looks up the corresponding application programming interface (API) according to the mapping table and calls it.

[0069] Taking a frame-selection photo as an example: The wearer faces the device panel, says "take a photo," and draws a rectangle in the air with their hand. The system recognizes the intention as frame-selection photo capture, and the parameter θ contains the normalized coordinates and size of the rectangle in the field of view. After receiving this intention, the smart glasses' camera control service performs the following operations: activates the main camera sensor; based on the coordinate information in parameter θ, crops the specified area from the full-frame output of the image sensor; performs automatic focusing and automatic exposure optimization on the cropped image area; triggers image acquisition, encoding, and storage; and saves the image file to a local encrypted storage partition.

[0070] Let's take zoom control as an example. The user is viewing a map displayed on the smart glasses' projection screen and says "zoom in" while making a pinch-to-expand gesture. The system recognizes the intention as zoom in, with parameter θ being a zoom factor of 1.5. After receiving this intention, the display module control service performs the following operations: obtains the zoom reference point of the current display screen, defaulting to the geometric center of the gesture trajectory; adjusts the projection matrix scaling factor of the rendering engine to 1.5 times the original factor; re-renders the display frame and pushes it to the optical display module; the user observes the map interface being smoothly zoomed in.

[0071] Optionally, to enhance interaction certainty and user experience, the system provides clear execution confirmation feedback to the wearer via a feedback module after each instruction is executed. The feedback method selects at least one of the following forms based on the scenario and hardware capabilities: audio feedback, which plays short prompts or voice messages via bone conduction or air conduction speakers; visual feedback, which overlays icons or text prompts in the projection field of the display module; and haptic feedback, where, if the smart glasses integrate haptic feedback devices such as linear motors, specific vibration patterns can indicate the completion of the operation.

[0072] In this embodiment, the feedback module can employ a multimodal composite feedback strategy. Taking a photography scenario as an example: at the moment the virtual shutter is pressed, the display module presents a 0.1-second white screen flash effect, while the bone conduction speaker plays a low-frequency clicking sound optimized by a parametric equalizer. This sound is conducted directly to the inner ear through the bone structure, and the wearer can still clearly perceive it even in extremely noisy environments.

[0073] In summary, this embodiment of the application uses a bone conduction microphone and millimeter-wave radar built into smart glasses to collect robust voice data and hand movement data unaffected by lighting conditions, respectively; it extracts voice keywords and gesture trajectories through an independent modal recognition pipeline; it achieves reliable command screening through confidence judgment and gesture type verification; it achieves multimodal intent decoding through feature vectorization and spatiotemporal alignment; and finally, it completes closed-loop control through intent-service mapping and multimodal feedback. This solves the technical challenges of low recognition rate in noisy environments for single voice interaction and easy failure of single visual gesture interaction in complex backgrounds and lighting conditions, achieving stable, accurate, and low-latency natural human-computer interaction in demanding scenarios such as industrial inspection, outdoor operations, and special tasks.

[0074] In some embodiments, the step of performing motion feature recognition on the body motion data to obtain the gesture trajectory of the wearer includes:

[0075] The first radar collects the original echo signal corresponding to the wearer's hand;

[0076] The original echo signal is subjected to range-Doppler transformation to obtain a radar data spectrum corresponding to the wearer's hand; the radar data spectrum contains at least one of the distance information, velocity information and angle information of the wearer's hand.

[0077] Static clutter suppression processing is performed on the radar data spectrum to obtain a dynamic point cloud sequence with the moving hand of the wearer as the reflecting target;

[0078] The gesture trajectory is obtained by recognizing the continuous spatial motion path of the wearer's hand on the dynamic point cloud sequence.

[0079] The first radar is a millimeter-wave radar antenna module integrated into the temple or frame of the smart glasses, operating in the 60-77 GHz frequency band and employing a frequency-modulated continuous wave modulation (FMCM) system. The millimeter-wave radar continuously transmits linearly modulated electromagnetic waves and receives the echo signals reflected from the target. When the wearer's hand enters the radar's field of view, the electromagnetic waves emitted by the radar strike the hand's surface, generating a reflected echo. This echo signal is captured by the radar's receiving antenna, forming the original echo signal.

[0080] Considering that the hand is the primary means of transmitting air gesture commands and that hand movements have a clear directional intent, this application focuses on the hand as the data acquisition target. The radar's field of view is designed to match the wearing position and typical usage posture of the smart glasses, ensuring that the hand is within the radar's effective detection range in a natural operating posture. For example, the radar's horizontal field of view is set to ±60 degrees, and the vertical field of view is set to ±30 degrees, with an effective detection distance of 0.1 meters to 2.0 meters, thereby fully covering the hand movement space in the chest area of ​​the wearer.

[0081] The raw radar echo data is denoted as S_r, and its data structure is a complex three-dimensional tensor, with dimensions represented as (Chirp number, sampling points, number of receiving antennas). For example, if the center frequency of the radar transmitted signal is 60.5 GHz, the bandwidth is 4 GHz, the sweep period is 50 μs, and 256 points are sampled within each sweep period. The radar antenna configuration is either 1 transmit and 3 receive or 2 transmit and 4 receive to achieve horizontal angle of arrival estimation. Within a data frame (typically 50 ms), the radar transmits 32 or 64 consecutive sweep pulses, forming a fast-time-slow-time two-dimensional data matrix.

[0082] Range-Doppler transform is a signal processing method that converts the original time-domain echo signal into a joint range-velocity domain. It involves the cascaded application of two Fast Fourier Transforms (FFTs). Specifically, a first FFT is performed in each sweep cycle dimension. The sweep cycle corresponds to the fast time dimension, where the sampling points reflect the round-trip time delay of the electromagnetic wave from transmission to reception. Through the Fourier transform, the time-domain echo signal is converted into a range-dimensional spectrum, where the frequency corresponding to the spectral peak is proportional to the target distance. The range resolution is determined by the signal bandwidth; in this embodiment, a bandwidth of 4 GHz corresponds to a range resolution of approximately 3.75 cm. This means that two targets with a distance difference of 3.75 cm can be distinguished as two independent peaks in the range-dimensional spectrum.

[0083] A second Fast Fourier Transform (FFT) is performed on the complex sequence within the same range gate in the slow-time dimension. The slow-time dimension corresponds to a frame sequence consisting of multiple consecutive frequency sweep cycles, and the change of the complex sequence within the same range gate with slow time reflects the phase evolution of the target's reflected signal at that range. The radial velocity information of the target is extracted through the second Fourier Transform. The Doppler frequency is proportional to the radial velocity; a positive frequency indicates that the target is moving towards the radar, and a negative frequency indicates that the target is moving away from the radar.

[0084] The range-Doppler spectrum is obtained through the two Fourier transforms described above. This spectrum is a two-dimensional complex matrix, with the horizontal axis representing the range gate and the vertical axis representing the Doppler frequency. The amplitude of each pixel in the spectrum represents the probability intensity of the presence of a reflecting target at that distance and velocity, while the phase information preserves the fine features of the target's micro-motion.

[0085] Furthermore, to obtain the spatial orientation information of the target, embodiments of this application estimate the angle of arrival (AHA) of the phase difference between multiple receiving antennas. When multiple receiving antennas are arranged in space at a specific interval, the path difference of the echo from the same target reaching different antennas results in a phase difference, which is definitely related to the AHA of the target relative to the antenna array normal. Digital beamforming or multi-signal classification algorithms are used to calculate the horizontal AHA at each effective detection unit in the range-Doppler spectrum.

[0086] Therefore, the two-dimensional range-Doppler spectrum is expanded into a three-dimensional range-Doppler-angle data tensor R, whose dimensions are represented as (D, V, A), corresponding to the number of range cells, Doppler cells, and angle cells, respectively. This data tensor is the radar data spectrum referred to in the embodiments of this application, which fully depicts the instantaneous state of the hand target in the range, velocity, and angle dimensions.

[0087] Millimeter-wave radar receives reflected signals from both moving targets and static backgrounds when acquiring echo signals. Static backgrounds include, but are not limited to, walls, equipment, furniture, floors, fixed installations, and any other objects stationary within the radar's field of view. In the range-Doppler spectrum, signals reflected from static backgrounds appear as strong energy points with Doppler frequencies of zero or near zero. The intensity of these static clutter signals is typically much higher than the hand-reflected signal; if not suppressed, they will severely obscure the weak motion signals corresponding to the hand.

[0088] Therefore, this application employs moving target indication (MTI) technology to suppress static clutter in radar data spectra. MTI technology utilizes the physical difference that static target echoes exhibit coherence between frames, while the phase of moving target echoes changes with displacement. Specifically, it is implemented using a two-pulse canceller: complex subtraction is performed on the range-Doppler spectra of two consecutive frames at corresponding pixels.

[0089] Let the spectrum of the nth frame be R_n, and the spectrum of the (n-1)th frame be R_{n-1}. Then, the spectrum after cancellation is R_mti = R_n - R_{n-1}. For a static target, its complex value remains unchanged between frames, and the energy is significantly attenuated after subtraction, ideally reaching zero. For a moving hand, because its spatial position changes over time, the phase of the reflected signal also changes accordingly, and a relatively strong energy is still retained after subtraction. Through the cancellation operation, the static background energy is greatly suppressed, while the moving hand signal becomes prominent.

[0090] After static clutter suppression, most of the static background energy is filtered out in the output range-Doppler spectrum, retaining only the sparse energy points corresponding to the moving target. The spectrum then presents a clean signal environment dominated by the moving hand.

[0091] Effective detection points corresponding to the hand are extracted from each frame of the suppressed spectrograms across multiple consecutive frames. Each detection point contains information such as distance, Doppler frequency, angle of arrival, and signal strength. The positions of these detection points in the distance-velocity-angle space change continuously with frame time, forming a discrete set of points in three-dimensional space. Organizing the point sets of multiple consecutive frames in chronological order yields a dynamic point cloud sequence.

[0092] The essence of dynamic point cloud sequences is a sparse sampling representation of hand movements in the radar observation space. Unlike hand skeleton points or dense depth maps generated by visual methods, radar point clouds do not contain fine morphological details of the hand, but accurately preserve the spatial position trajectory and radial velocity temporal sequence of the hand. This representation is completely insensitive to factors such as illumination, background texture, skin color, and occlusion, and has high computational efficiency.

[0093] The dynamic point cloud sequence is a three-dimensional positional record of hand movements on the time axis, while the continuous spatial motion path is the complete motion trajectory fitted from these discrete points. The dynamic point cloud sequence is organized on the time axis to construct a four-dimensional feature tensor R_seq, with dimensions represented as (T, D', V', A'), where T is the number of time frames, and D', V', and A' are the subsets of distance, Doppler, and angle units retained after constant false alarm rate (CFAR) detection, respectively. Since the number of valid points detected in each frame may differ, density normalization or a fixed number of samples are required to form a regular tensor structure. This four-dimensional tensor is then input into a pre-trained gesture recognition neural network model. This application preferably employs a combined architecture of a convolutional neural network and a long short-term memory (LSTM) network, specifically designed to handle the spatiotemporal characteristics of radar point cloud sequences. The LSTM module consists of two layers of bidirectional LSTM networks, with a maximum of 128 hidden units. The spatial feature sequence output by the convolutional neural network is input into the long short-term memory network in chronological order. The bidirectional structure allows the features at each time step to simultaneously incorporate contextual information from past and future moments. The memory units and gating mechanism of the long short-term memory network effectively capture the dynamic evolution of gesture movements over time, such as the periodic repetition of a circling gesture, the unidirectional uniform motion of a swipe gesture, and the monotonically changing spacing of a pinch gesture. The model's output layer is a fully connected network with an output dimension equal to the number of predefined gesture categories. After normalization using the Softmax function, the probability distribution of each gesture category is obtained. The category corresponding to the highest probability is selected as the gesture category G.

[0094] Furthermore, the hidden state of the last time step of the Long Short-Term Memory network can be used as the trajectory feature vector g, which encodes the global temporal information of the entire gesture movement process. Optionally, an attention mechanism can be applied in the temporal dimension, learning a weight coefficient for the features at each time step, and the weighted sum is used to obtain the trajectory feature vector g. The trajectory feature vector g is a dense vector representation of the gesture trajectory. Compared to the discrete index that transmits the gesture category G, the trajectory feature vector g retains rich details such as continuous morphological information, velocity change information, and spatial position information of the gesture trajectory, providing higher-resolution input features for multimodal fusion.

[0095] Considering that visual gesture recognition requires continuous operation of deep neural networks for target detection, key point localization, or semantic segmentation, the power consumption on embedded devices such as smart glasses is relatively high, making it difficult to support all-day wear. The distance-Doppler transform, static clutter suppression, and constant false alarm rate detection used in this application are all lightweight signal processing algorithms that can run in real time on the microcontroller unit without waking up the high-performance processor. Only when a valid gesture movement is detected is the convolutional neural network-long short-term memory network model invoked for fine classification. This hierarchical wake-up architecture enables smart glasses to continuously sense hand movements with milliwatt-level power consumption, significantly extending device battery life.

[0096] In some embodiments, the step of recognizing the control intent of the wearer based on the voice keywords and the gesture trajectory to obtain the wearer's target control intent for the smart glasses includes:

[0097] Obtain the recognition confidence score of the voice keywords based on the trajectory type of the gesture trajectory;

[0098] Determine whether the recognition confidence of the voice keyword is higher than a first preset threshold, and whether the trajectory type of the gesture trajectory belongs to a predefined set of valid gestures;

[0099] If the recognition confidence of the voice keyword is higher than the first preset threshold and the trajectory type of the gesture trajectory belongs to the set of valid gestures, the step of recognizing the control intention of the wearer to obtain the target control intention of the wearer for the smart glasses is executed; otherwise, the currently collected voice data and body movement data are discarded.

[0100] Among them, the recognition confidence p_k of the voice keywords is a quantitative evaluation of the reliability of the recognition result by the model, with a value range of [0, 1]. The trajectory type of the gesture trajectory is the gesture category G output by the aforementioned gesture recognition model. This category is selected from a predefined set of valid gestures, which is configured according to the application scenario of the smart glasses. For example, the set of valid gestures may include gesture categories with clear instruction semantics, such as drawing a frame, swiping left, swiping right, swiping up, swiping down, pinching, unfolding, drawing a circle, and waving. Each type of gesture has its typical kinematic characteristics. For example, drawing a frame requires the trajectory to be approximately rectangular and the closure to meet the standard, pinching requires the distance between the thumb and index finger to change monotonically, and waving requires rapid reciprocating hand movements.

[0101] The first preset threshold is a configurable hyperparameter used to define whether the speech recognition result has reached an acceptable level of reliability. If the threshold is set too low, such as 0.60, a large number of low-confidence, potentially misrecognized voice commands will enter the subsequent process, leading to frequent false triggers. If the threshold is set too high, such as 0.95, even in a quiet environment, some voice commands with non-standard pronunciation or slight interference will be rejected, causing users to need to repeat commands and reducing interaction efficiency.

[0102] The predefined set of valid gestures is a complete set of gesture commands that the system can understand and respond to. This set is defined during the system design phase based on the product's functional positioning and is embedded in the output layer of the gesture recognition model. Any gesture trajectory that does not belong to this set, such as an unconscious, random wave of the hand, the action of adjusting clothing, or natural gestures when interacting with others, is judged as an invalid gesture and will not trigger the subsequent intent recognition process.

[0103] The current instruction is deemed valid only if both of the above conditions are met simultaneously, and the subsequent multimodal intent recognition process continues. If either condition is not met, the currently acquired frame of speech and radar data is discarded, and the process returns to the data acquisition step, awaiting the next round of instruction input. The dual verification mechanism in this embodiment is located after single-modal recognition and before multimodal fusion. Through pre-filtering, a large number of invalid or unreliable inputs are filtered out at a lower cost before entering the computationally expensive multimodal fusion model, avoiding waste of system resources.

[0104] Considering that smart glasses continuously collect voice signals and hand movements within the radar's field of view during wear, without an effective filtering mechanism, casual conversations and unconscious hand gestures could be misinterpreted as commands. This application's embodiments reduce the false trigger rate through dual verification using a voice confidence threshold and a valid gesture set. If a user accidentally says "take a photo" while talking to someone but doesn't make a framing gesture, or unintentionally draws a rectangle without saying any keywords, the photo-taking operation will not be triggered, effectively avoiding awkward scenarios and device malfunctions.

[0105] In some embodiments, the step of recognizing the control intent of the wearer to obtain the target control intent of the wearer regarding the smart glasses includes:

[0106] The voice keywords are mapped to word vectors, and the gesture trajectory is encoded into a fixed-dimensional trajectory feature vector;

[0107] Align and concatenate the word vectors and the trajectory feature vectors in the time dimension to obtain the multimodal fusion input vector corresponding to the wearer;

[0108] The multimodal fusion input vector is input into a pre-trained multimodal fusion neural network model to obtain the intent category representing the target control intent and the control parameters associated with the intent category.

[0109] Word vectors are a standard technique in natural language processing for mapping discrete symbols to a continuous vector space. A speech keyword K is a discrete string of symbols, lacking the continuity and measurability required for numerical computation. To input speech modal information into subsequent neural network models, the keyword symbols need to be converted into dense real-valued vector representations.

[0110] In this embodiment, the smart glasses locally store a pre-trained word embedding lookup table. This lookup table is a real matrix of dimension (V, D), where V is the size of the vocabulary of the predefined keyword set, and D is the embedding dimension of the word vectors. In this embodiment, D is set to 128. The vocabulary covers all the predefined keywords mentioned in the preceding steps, including taking a photo, recording a video, zooming in, zooming out, going back, confirming, closing, navigation, increasing volume, decreasing volume, etc., and can be dynamically expanded with system upgrades.

[0111] A lookup operation is performed in the word embedding lookup table using the speech keyword K as the key. The result is a 128-dimensional floating-point vector e_K. This vector corresponds to the meaning of K in the semantic space: keywords with similar semantics have similar word vectors that are also close in Euclidean space. For example, the cosine similarity between the word vectors of "enlarged" and "expanded" is high, while the cosine similarity between the word vector of "closed" and "closed" is low. It should be noted that the word embedding lookup table can be dynamically updated. In online mode, the smart glasses can receive incrementally updated word embedding parameters from the server to optimize the representation of domain-specific terms or user-personalized language. In offline mode, the factory-preset or the most recently synchronized static version is used.

[0112] Correspondingly, the gesture trajectory feature vector g encodes the following information about the gesture trajectory: the geometry of the trajectory in the distance-angle plane, the instantaneous velocity change curve of the hand movement, the closure and smoothness of the trajectory, and the acceleration pattern during gesture execution. Compared with discrete gesture categories G, the trajectory feature vector g preserves the continuous details of gesture motion, providing higher resolution information input for multimodal fusion.

[0113] Considering that the recognition of speech keywords is an event, corresponding to a specific point in time or a short period of time on the timeline—usually the moment when the last syllable of the keyword is recognized—while the gesture trajectory is a continuous process lasting from hundreds of milliseconds to several seconds, its start time, end time, and each stage of the movement have different informational value. Concatenating the word vector of the keyword with the final state vector of the entire gesture trajectory would result in the loss of the time period information most relevant to the speech command in the gesture trajectory. Therefore, to solve the above-mentioned spatiotemporal heterogeneity problem, the embodiments of this application adopt the following alignment strategy: First, obtain the timestamp T_k of the keyword recognition completion moment from the speech recognition module. This timestamp is based on the system clock and has a precision of milliseconds. Second, obtain the gesture trajectory feature vector sequence g_seq from the radar signal processing module. The gesture recognition model outputs a corresponding trajectory feature vector for each frame of radar data (frame period 50ms), so g_seq is a matrix of dimension (T, 128), where T is the number of frames the current gesture process lasts. Then, using T_k as the baseline, a preset time window is traced back, with a typical window length of 1.5 seconds, corresponding to 30 frames of radar data. The feature subsequence g_win within this time window is extracted from g_seq, with dimensions (L, 128), where L is the actual number of frames contained within the window, which may be less than or equal to 30 depending on the gesture execution speed. g_win is then aggregated and compressed along the time dimension. Max pooling aggregation can be used to retain the most significant feature responses of the gesture trajectory within this time window, suitable for command scenarios emphasizing gesture amplitude or speed peaks. Optionally, attention-weighted aggregation can also be used to adaptively locate the time segment in the gesture trajectory most relevant to the current voice keyword. For example, for a photo-taking command, the attention weights focus on the instant the gesture frame closes; for a zoom-in command, the weights focus on the end of the pinch-to-expand gesture.

[0114] After alignment, the word vector e_K is concatenated with the aligned trajectory feature vector g_align. Concatenation involves joining the two 128-dimensional vectors end-to-end to form a new 256-dimensional vector F = [e_K; g_align]. This concatenation operation preserves the independent information channels of the two modalities, enabling the subsequent fusion model to autonomously learn the interaction patterns between them, rather than forcibly compressing them at the input layer. The structure of the multimodal fusion neural network model consists of two main components: a feature interaction layer and a task output layer. The feature interaction layer is responsible for modeling the deep association between the word vector and the trajectory feature vector. In this embodiment, a multilayer perceptron is used as the basic architecture of the feature interaction layer. Specifically, the 256-dimensional input vector F is passed sequentially through three fully connected layers with 512, 256, and 128 hidden units in each layer, followed by a batch normalization layer and a ReLU activation function. Through multi-layer nonlinear transformations, the model can capture high-order interaction features between speech keywords and gesture trajectories.

[0115] Correspondingly, the task output layer contains two parallel branches: the intent classification branch consists of a fully connected layer followed by a Softmax activation function, with an output dimension of M, where M is the total number of predefined intent categories. The Softmax function normalizes the output vector into a probability distribution, with each dimension taking values ​​between 0 and 1, and the sum being 1. The index corresponding to the maximum probability is taken as the target control intent I. The predefined intent categories in this embodiment include, but are not limited to: box selection for taking a picture, taking a picture immediately, zooming in, zooming out, navigating to, increasing volume, closing the application, and confirming the operation.

[0116] The structure of the parametric regression branch is related to the intent category. For intent categories that do not require continuous parameters, this branch outputs a fixed placeholder or a null value. For intent categories that require continuous parameters, this branch outputs a real value for the corresponding dimension. Taking the box-selection for taking a photo intent as an example, the parametric regression branch outputs a 4-dimensional vector, corresponding to the normalized values ​​of the top-left corner x-coordinate, top-left corner y-coordinate, width, and height of the rectangle, with values ​​ranging from 0 to 1. Taking the zoom intent as an example, the parametric regression branch outputs a 1-dimensional scalar representing the zoom factor, with values ​​ranging from 0.5 to 3.0.

[0117] The loss function of the parametric regression branch is jointly optimized with that of the intent classification branch. During model training, the gradient of the parametric regression branch is only backpropagated if the intent is correctly classified; if the intent is misclassified, the gradient of the parametric regression branch is masked to avoid invalid updates to the parameter space of incorrect intents.

[0118] In related technologies, multimodal fusion of voice and gesture often remains at the decision level, i.e., after recognizing voice commands and gesture commands separately, they are combined and matched using a predefined rule table. This fusion method is essentially still a simple parallel connection of two single-modal systems, which is difficult to handle combinations not exhaustively listed in the rule table, and cannot utilize subtle interaction information between modalities. The embodiments of this application map voice keywords into continuous vectors and encode gesture trajectories into continuous vectors, and perform concatenation and nonlinear transformation in the same vector space, enabling the model to autonomously learn joint feature representations across modalities.

[0119] In some embodiments, the multimodal fusion neural network model includes an encoder model based on the Transformer architecture, and the method further includes:

[0120] On the server side, the initial Transformer model is trained using a dataset of voice and gesture pairing samples labeled with intent categories and control parameters to obtain a converged model;

[0121] The convergence model is compressed and quantized to obtain a lightweight model suitable for embedded device operation;

[0122] The lightweight model is deployed to the local processor of the smart glasses to perform the intent recognition in an offline environment.

[0123] The server side can be a cloud computing cluster with high-performance computing resources, large-capacity storage, and distributed training capabilities. Compared with the embedded processor on the smart glasses, the server side has several times the computing power and thousands of times the storage space, enabling iterative training of large-scale datasets and full convergence of complex model architectures.

[0124] Each sample in the voice and gesture pairing sample dataset can include: word vectors of voice keywords, a sequence of gesture trajectory feature vectors for the corresponding time period, and manually labeled intent categories and control parameters. The dataset construction process can be as follows: First, collect voice-gesture combination commands executed by volunteers in simulated real-world scenarios. The collection environment covers different noise levels, such as quiet offices, noisy workshops, and outdoor streets, and different lighting conditions, including bright, dim, and completely dark scenes. Volunteers represent different ages, genders, and accents to ensure data diversity.

[0125] The collected raw data undergoes preprocessing. The speech portion passes through a speech recognition pipeline, outputting keywords and their confidence scores; only samples with a confidence score higher than 0.7 are retained to control noise. The radar portion passes through a gesture recognition pipeline, outputting a trajectory feature vector sequence g_seq. To ensure the quality of the training data, samples with recognition errors or improper execution need to be manually removed.

[0126] For each valid sample, the annotation tool labels the intent category I and the associated control parameter θ. The annotation tool provides a visual interface that simultaneously displays voice text, gesture trajectory animation, and radar point cloud playback, comprehensively judging the user's true intent based on multimodal information. For samples involving spatial parameters, such as box selection for taking photos, the annotator needs to draw a rectangle on the video screen as the ground truth; for scaling intents, the scaling ratio before and after must be annotated.

[0127] The initial Transformer model can adopt a standard encoder architecture. In this embodiment, the Transformer encoder is configured as follows: the encoder has 4 layers, each with 8 attention heads, a hidden layer dimension of 256, an intermediate layer dimension of 1024, uses GELU as the activation function, layer normalization is placed before the residual connection, and the dropout rate is set to 0.1. The input sequence length is L+1, where L is the number of frames of the gesture trajectory feature vector within the time window, and 1 is the insertion of the speech keyword vector as the first position of the sequence. The output is the encoded vector corresponding to the first position of the sequence as the fused feature, which is fed into the subsequent classification and regression heads.

[0128] The training process utilizes the AdamW optimizer with an initial learning rate of 1e-4 and a weight decay coefficient of 0.01. Learning rate scheduling employs a warm-up followed by cosine annealing, with the warm-up steps comprising 10% of the total training steps. The loss function is a weighted sum of the cross-entropy loss for intent classification and the mean squared error loss for parametric regression, with the weight coefficients determined through hyperparameter search to be 0.7:0.3. The training batch size is 256, and distributed training is performed on eight NVIDIA A100 GPUs, with the model converging after approximately 200 epochs.

[0129] Compression and quantization are used to convert large-scale models trained on the server side into models suitable for running on resource-constrained embedded devices. Specifically, attention heads with lower importance in the Transformer encoder are pruned. The importance of an attention head is measured by the performance degradation caused by its masking on the validation set. The possibility of weight sharing between adjacent encoder layers is explored. This application's embodiment adopts a cross-layer weight sharing strategy, binding partial projection matrices of the first and second layers together. Weight sharing reduces the storage requirements of model parameters, and since the parameters of the shared layers are updated more frequently, it has a certain degree of regularization effect.

[0130] Deployment refers to integrating the compressed and quantized model files, inference engine, and necessary runtime libraries into the firmware or system image of the smart glasses, enabling offline intent recognition capabilities from the moment the device leaves the factory. Specifically, the quantized model files can be packaged and stored together with the model metadata file in the non-volatile memory partition of the smart glasses. The metadata file records information such as the model's input dimensions, output category mapping table, and quantization parameters (scaling factor, zero point), which are then used by the inference engine for dequantization during loading. An intent recognition service is encapsulated at the system service layer of the smart glasses. This service resides in the background as a system daemon, providing a unified intent recognition application interface. Upper-layer applications call this service through the bound interface, passing in voice keywords, text, and gesture trajectory feature vectors; the service returns the intent category and control parameters. Full-process integration testing and power consumption optimization are then performed.

[0131] This application's embodiments employ a division-of-labor architecture of large-scale model training on the server side and lightweight model deployment on the device side, balancing model performance and deployment feasibility. The server side utilizes massive computing power and data to train a high-precision model, while the device side inherits its knowledge through compression and quantization. Thanks to the application of model compression and quantization technology, there is no need to equip smart glasses with expensive high-performance computing modules, making this method widely applicable to mass-market consumer smart glasses products that are sensitive to cost and battery life, rather than being limited to special equipment where cost is not a concern.

[0132] In some embodiments, the method further includes:

[0133] Receives a working mode switching trigger event from the smart glasses; the mode switching trigger event is triggered based on an instruction or environmental state.

[0134] The working mode of the smart glasses is switched between offline and online modes according to the switching trigger event. In offline mode, the voice feature recognition, motion feature recognition, and control intent recognition are all completed locally on the smart glasses, and the raw data and intermediate feature data of the voice data and body motion data are restricted to local processing and encrypted storage, and are not uploaded to external networks. In online mode, the anonymized multimodal fusion feature vector or recognition result is uploaded to the server for model optimization or data analysis of the multimodal fusion neural network model.

[0135] The working mode switching trigger event is an external or internal signal that causes the smart glasses to switch from the current working mode to another working mode. Specifically, the working modes in this application can include offline mode and online mode. In offline mode, all data processing is completed locally, and data does not flow out of the device; in online mode, anonymized feature data can be uploaded to the server for model optimization and data analysis.

[0136] Mode switching can be triggered by explicit commands. Wearers can actively switch operating modes via voice commands, gesture commands, or physical buttons. Voice commands include saying "Enter offline mode" or "Activate online mode," while gesture commands include drawing a circle and tapping to confirm.

[0137] Optionally, implicit triggering based on environmental conditions is also possible. For example, the system can automatically determine the current environmental characteristics based on sensor input and automatically trigger mode switching when preset conditions are met. For instance, when the wireless network connection detection module detects that the device is connected to a trusted wireless network in the home or office, it automatically switches to online mode to synchronize data and receive model updates. When the ambient light sensor detects continuous low illumination and the accelerometer detects prolonged stillness, the system determines that the device may be in a stored state and automatically switches to offline mode to save power.

[0138] In offline mode, the smart glasses constitute a completely independent computing system that does not exchange data with any external entity. Specifically, in offline mode, after the raw audio stream of voice data is converted from analog to digital by the bone conduction microphone, it is sent to the local memory buffer. After voice feature recognition processing, the raw audio sample is immediately and securely erased from memory. The raw echo signal of radar data is not written to persistent storage media after recognition. Even if the user performs memory dump or log export operations in offline mode, these intermediate features are excluded from the dump scope. Intent recognition results are not logged, not added to historical queries, and not used for user behavior analysis. The system only retains the most recent recognition result for confirmation display in the feedback module, and it is immediately cleared when the user switches applications or enters standby mode.

[0139] Correspondingly, the online mode is a data collaboration mechanism based on explicit user authorization and a trusted environment. The online mode does not simply remove all constraints from the offline mode; rather, it establishes a strict set of data anonymization and minimal collection standards. Specifically, the online mode uploads at most one of the following two types of data, requiring user authorization item by item: anonymized multimodal fusion feature vectors and recognition results and user feedback. The uploaded content does not contain any identifiers that can identify the user; it only includes an anonymous device identifier generated locally by the device and rotated periodically. All data uploaded in the online mode is encrypted using transport layer security protocols at the transport link layer, and column-level encryption is used during server-side storage. Access control is based on the principle of least privilege.

[0140] This application's embodiments construct a hierarchical system ranging from "absolute privacy" to "controllable sharing" by clearly distinguishing between offline and online modes and implementing strict anonymization processing in the online mode. Users can dynamically adjust their working mode according to the risk level of the current environment.

[0141] In some embodiments, controlling the smart glasses according to the target control intent includes:

[0142] The application programming interface corresponding to the target control intent is invoked according to the control parameters to perform device control operations on the smart glasses; the device control operations include at least one of controlling the camera module of the smart glasses to take pictures, adjusting the display content of the display module of the smart glasses, and controlling the audio module of the smart glasses to play announcements;

[0143] In response to the completion of the device control operation, the smart glasses provide execution confirmation feedback to the wearer through the feedback module.

[0144] Here, the target control intent I and its associated parameter θ are the output results of the multimodal intent recognition module in step 208. Intent I is an enumeration type that represents the macroscopic operation category that the user wants the smart glasses to perform; parameter θ is the specific execution parameter bound to the intent, and its data structure is a variable-length dictionary that is dynamically generated according to different intent types.

[0145] The operating system kernel of smart glasses maintains an intent-service mapping table. This table defines the correspondence between each intent category and system service, as well as the conversion rules for parameters θ to service interface parameters. The mapping table is loaded into memory when the system starts and supports dynamic updates at runtime to adapt to different versions of application functionality iterations.

[0146] Taking the intention to capture a frame as an example, when the intention recognition module outputs I=CAPTURE_FRAME, the parameter θ contains the normalized coordinates and size of the rectangle in the field of view, exemplarily {"x": 120, "y": 80, "w": 400, "h": 300}. The system service scheduler locates the camera control service according to the mapping table and calls its provided CaptureFrameWithRect interface. This interface receives four integer parameters, representing the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width, and the height of the rectangle, respectively. The scheduler converts the values ​​in the θ dictionary into interface parameters in sequence and initiates a remote procedure call. After receiving the call request, the camera control service executes the following sequence of device control operations: First, it detects the current state of the main camera sensor. If it is in a sleep state, it executes the wake-up and initialization process, including sensor power-on, I2C bus configuration, and output format settings. Second, it obtains the full-frame output of the current frame from the image signal processor. Then, based on the coordinate information in the parameter θ, it crops the specified area in the output buffer. The cropping operation does not involve pixel format conversion or scaling; it only generates a new buffer view through pointer offset and length calculation to minimize processing latency. Next, automatic exposure and autofocus optimization are performed on the cropped image area. Since the area selected by the user with a gesture is the focus area, this embodiment locks the metering and focus areas into this rectangular region, instead of the default central area as the focus area and other areas receiving equal attention.

[0147] Taking zoom control intent as an example: When the intent recognition module outputs I=ZOOM_IN, the parameter θ is {“scale”: 1.5}. The system service scheduler locates the display rendering service and calls its SetZoomFactor interface. The display rendering service obtains the zoom reference point of the current display screen. In this embodiment, the geometric center of the gesture trajectory is taken as the reference point by default, and the coordinates of this reference point are already included in the parameter θ. The rendering engine adjusts the scaling factor of the projection matrix to 1.5 times the original factor, and at the same time adjusts the viewport translation to scale around the reference point to keep the user's focus area at the center of the field of view. After the adjustment is completed, the rendering engine regenerates the display frame and pushes it to the optical display module through the display interface. The wearer observes that the map interface or viewfinder is smoothly magnified, and the center of magnification is consistent with the area selected by the gesture, forming an intuitive spatial mapping.

[0148] Taking volume control intent as an example: When the intent recognition module outputs I=VOLUME_UP, the parameter θ is {“delta”: +2}. The system service scheduler locates the audio service and calls its AdjustVolume interface. The audio service reads the current system volume value, increases it by 2 levels, and updates the digital gain parameter of the audio mixer. If the adjusted volume exceeds the hardware-supported upper limit, it is automatically truncated to the maximum value. At the same time, the audio service generates a short prompt tone sample, which is played through the bone conduction speaker, allowing the user to perceive the volume change through hearing.

[0149] It should be noted that the application programming interfaces (APIs) invoked in this application embodiment can be system-level service interfaces or callback interfaces registered by third-party applications. This application embodiment is not limited to the native functions built into smart glasses, but rather supports third-party developers in registering their application functions to the multimodal command system through a standardized intent-service mapping mechanism. For example, a third-party map application can register a callback function corresponding to the NAVIGATE_TO intent. When the user speaks "navigation" and points to the destination with a gesture, the system passes the destination identifier in the parameter θ to the application, which then completes route planning and navigation announcement.

[0150] Execution confirmation feedback is a crucial link in the human-computer interaction closed loop. Its purpose is to clearly convey to the user that the instruction has been received, understood, and executed by the system, reducing user uncertainty and the frequency of repetitive instructions. The feedback module is a collection of hardware and software components in smart glasses responsible for generating perceptible feedback signals. Specifically, depending on the physical form of the feedback signal, the feedback module may include at least one of the following: Audio feedback modules include bone conduction speakers and / or air conduction speakers. The content of the audio feedback can be pre-recorded short prompts, such as the click of a camera shutter, the ticking sound of volume adjustment, or the ding-dong sound of successful operation; it can also be voice-synthesized announcements, such as "Photo saved" or "Navigation started."

[0151] The visual feedback module includes a see-through optical display module and / or a side screen indicator. The see-through optical display module can overlay virtual images onto the wearer's real field of view, and feedback information is presented in the form of icons, text, animations, etc., at designated locations in the field of view. For example, after taking a photo, a white border animation flashes at the edge of the viewfinder for 0.3 seconds; when adjusting the volume, a progress bar is displayed at the edge of the field of view, the length of which is proportional to the current volume value; when an operation is rejected, a red dot is displayed in the center of the gaze with a shaking effect.

[0152] The haptic feedback module, comprising a linear resonant motor or piezoelectric ceramic actuator, is integrated into the inner temple of the glasses, in contact with the head. Haptic feedback is independent of auditory and visual channels, making it particularly suitable when the wearer is in a noisy environment or needs to maintain visual focus. For example, a single vibration pulse at 150Hz lasting 100ms is generated when a photo is taken; three short pulses are generated when a command is rejected; and directional vibrations to the left or right are generated when navigation approaches a turning point.

[0153] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0154] Based on the same inventive concept, this application also provides a smart glasses control device for implementing the smart glasses control method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more smart glasses control device embodiments provided below can be found in the limitations of the smart glasses control method described above, and will not be repeated here.

[0155] In one exemplary embodiment, such as Figure 3 As shown, a smart glasses 300 is provided, which includes a first microphone and a first radar; the first microphone includes a bone conduction microphone for detecting head vibration data when the wearer of the smart glasses outputs speech; the first radar includes a millimeter-wave radar for recognizing body movement data of the wearer.

[0156] The smart glasses 300 includes: a data acquisition module 302, used to acquire voice data of the wearer of the smart glasses through the first microphone, and to acquire body movement data of the wearer through the first radar;

[0157] The first recognition module 304 is used to perform voice feature recognition on the voice data to obtain the voice keywords output by the wearer;

[0158] The second recognition module 306 is used to perform motion feature recognition on the body motion data to obtain the gesture trajectory of the wearer;

[0159] The third recognition module 308 is used to recognize the control intention of the wearer based on the voice keywords and the gesture trajectory, so as to obtain the target control intention of the wearer for the smart glasses;

[0160] The control module 310 is used to control the smart glasses according to the target control intention.

[0161] The various modules in the aforementioned smart glasses 300 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0162] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a smart glasses control method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0163] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0164] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps included in any of the foregoing method embodiments.

[0165] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps included in any of the foregoing method embodiments.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0170] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for controlling smart glasses, characterized in that, The method is applied to smart glasses; the smart glasses are equipped with a first microphone and a first radar; the first microphone includes a bone conduction microphone for detecting head vibration data when the wearer of the smart glasses outputs speech; the first radar includes a millimeter-wave radar for recognizing body movement data of the wearer; the method includes: The first microphone is used to collect the voice data of the wearer of the smart glasses, and the first radar is used to collect the body movement data of the wearer. The speech data is subjected to speech feature recognition to obtain the speech keywords output by the wearer; Motion feature recognition is performed on the body motion data to obtain the gesture trajectory of the wearer; Based on the voice keywords and the gesture trajectory, the control intent of the wearer is recognized to obtain the target control intent of the wearer regarding the smart glasses, including: Obtain the recognition confidence score of the voice keywords and the trajectory type of the gesture trajectory; The system determines whether the recognition confidence of the voice keyword is higher than a first preset threshold, and whether the trajectory type of the gesture trajectory belongs to a predefined set of valid gestures; wherein, the set of valid gestures includes gesture categories with instruction semantics; the set of valid gestures is predefined according to the function of the smart glasses; If the recognition confidence of the voice keyword is higher than the first preset threshold and the trajectory type of the gesture trajectory belongs to the set of valid gestures, the target control intention of the wearer for the smart glasses is determined based on the currently obtained voice keyword and gesture trajectory; otherwise, the currently collected voice data and body movement data are discarded. The smart glasses are controlled according to the stated target control intent.

2. The method according to claim 1, characterized in that, The step of performing motion feature recognition on the body motion data to obtain the gesture trajectory of the wearer includes: The first radar collects the original echo signal corresponding to the wearer's hand; The original echo signal is subjected to range-Doppler transformation to obtain a radar data spectrum corresponding to the wearer's hand; the radar data spectrum contains at least one of the distance information, velocity information and angle information of the wearer's hand. Static clutter suppression processing is performed on the radar data spectrum to obtain a dynamic point cloud sequence with the moving hand of the wearer as the reflecting target; The gesture trajectory is obtained by recognizing the continuous spatial motion path of the wearer's hand on the dynamic point cloud sequence.

3. The method according to claim 1, characterized in that, The step of recognizing the control intent of the wearer to obtain the target control intent of the wearer regarding the smart glasses includes: The voice keywords are mapped to word vectors, and the gesture trajectory is encoded into a fixed-dimensional trajectory feature vector; Align and concatenate the word vectors and the trajectory feature vectors in the time dimension to obtain the multimodal fusion input vector corresponding to the wearer; The multimodal fusion input vector is input into a pre-trained multimodal fusion neural network model to obtain the intent category representing the target control intent and the control parameters associated with the intent category.

4. The method according to claim 3, characterized in that, The multimodal fusion neural network model includes an encoder model based on the Transformer architecture, and the method further includes: On the server side, the initial Transformer model is trained using a dataset of voice and gesture pairing samples labeled with intent categories and control parameters to obtain a converged model; The convergence model is compressed and quantized to obtain a lightweight model suitable for embedded device operation; The lightweight model is deployed to the local processor of the smart glasses to perform the intent recognition in an offline environment.

5. The method according to claim 4, characterized in that, Each sample in the speech and gesture pairing sample dataset contains word vectors of speech keywords, a sequence of gesture trajectory feature vectors for the corresponding time period, and manually labeled intent categories and control parameters. On the server side, the initial Transformer model is trained using a dataset of voice and gesture pairing samples labeled with intent categories and control parameters to obtain a converged model, including: The raw data is collected by combining voice and gesture commands in simulated real-world usage scenarios, which cover different noise levels, different lighting conditions, and different user groups. The original data is preprocessed to output keywords and their corresponding confidence scores. Samples with confidence scores higher than a preset confidence threshold are retained as valid samples.

6. The method according to claim 3, characterized in that, The method further includes: Receives a working mode switching trigger event from the smart glasses; the mode switching trigger event is triggered based on an instruction or environmental state. The working mode of the smart glasses is switched between offline and online modes according to the switching trigger event. In offline mode, the voice feature recognition, motion feature recognition, and control intent recognition are all completed locally on the smart glasses, and the raw data and intermediate feature data of the voice data and body motion data are restricted to local processing and encrypted storage, and are not uploaded to external networks. In online mode, the anonymized multimodal fusion feature vector or recognition result is uploaded to the server for model optimization or data analysis of the multimodal fusion neural network model.

7. A type of smart glasses, characterized in that, The smart glasses are equipped with a first microphone and a first radar; the first microphone includes a bone conduction microphone for detecting head vibration data when the wearer of the smart glasses outputs speech; the first radar includes a millimeter-wave radar for identifying body movement data of the wearer; the smart glasses include: The acquisition module is used to acquire voice data of the wearer of the smart glasses through the first microphone, and to acquire body movement data of the wearer through the first radar; The first recognition module is used to perform voice feature recognition on the voice data to obtain the voice keywords output by the wearer; The second recognition module is used to perform motion feature recognition on the body motion data to obtain the gesture trajectory of the wearer; The third recognition module is used to recognize the control intention of the wearer based on the voice keywords and the gesture trajectory, and obtain the target control intention of the wearer for the smart glasses; the third recognition module is also used to obtain the recognition confidence of the voice keywords and the trajectory type of the gesture trajectory; The system determines whether the recognition confidence of the voice keyword is higher than a first preset threshold, and whether the trajectory type of the gesture trajectory belongs to a predefined set of valid gestures; wherein, the set of valid gestures includes gesture categories with instruction semantics; the set of valid gestures is predefined according to the function of the smart glasses; If the recognition confidence of the voice keyword is higher than the first preset threshold and the trajectory type of the gesture trajectory belongs to the set of valid gestures, the target control intention of the wearer for the smart glasses is determined based on the currently obtained voice keyword and gesture trajectory; otherwise, the currently collected voice data and body movement data are discarded. A control module is used to control the smart glasses according to the target control intention.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images