Intelligent interaction control method for AI glasses

Abstract

The invention discloses an intelligent interaction control method for AI glasses, and relates to the technical field of AI glasses, and the method comprises the following steps: synchronously collecting user input signals and environmental parameters through a multi-source sensor group built in the glasses, the sensor group at least comprising an IMU, a binocular camera, a microphone array and a physiological sensor; carrying out real-time fusion processing on the multi-modal input data by adopting a lightweight neural network to generate an interaction intention feature vector; based on real-time detection results of environmental noise intensity and illumination conditions, dynamically distributing weight coefficients of all interaction modes; generating a hierarchical response instruction according to the weight coefficient and a confidence threshold; and continuously optimizing the personalized interaction strategy of the user through a federal learning framework. According to the AI glasses intelligent interaction control method, through multi-modal weighted fusion, edge AI acceleration and federated learning optimization, the interaction success rate in a complex environment is improved to 93.6%; the end-to-end delay is controlled within 28ms; and a user-defined interaction strategy is supported.

Description

Technical Field

[0001] The present invention relates to the technical field of AI glasses, and specifically to an intelligent interaction control method for AI glasses. Background Art

[0002] Defects existing in the existing intelligent glasses interaction methods: Touch operations are inconvenient in mobile scenarios; Speech interaction has a low recognition rate in noisy environments; There is a lack of adaptability to user states (such as fatigue level, attention). Summary of the Invention

[0003] The purpose of the present invention is to provide an intelligent interaction control method for AI glasses, so as to solve the problems in the above-mentioned background art that the existing intelligent interaction control method for AI glasses has inconvenient touch operations in mobile scenarios; low recognition rate of speech interaction in noisy environments; and lack of adaptability to user states (such as fatigue level, attention).

[0004] To achieve the above purpose, the present invention provides the following technical solution: An intelligent interaction control method for AI glasses, the steps are as follows:

[0005] Synchronously collect user input signals and environmental parameters through a multi-source sensor group built into the glasses, and the sensor group at least includes an IMU, a binocular camera, a microphone array, and a physiological sensor;

[0006] Use a lightweight neural network to perform real-time fusion processing on multi-modal input data to generate an interaction intention feature vector;

[0007] Based on the real-time detection results of environmental noise intensity and light conditions, dynamically allocate the weight coefficients of each interaction mode;

[0008] Generate a hierarchical response instruction according to the weight coefficient and the confidence threshold;

[0009] Continuously optimize the user's personalized interaction strategy through a federated learning framework.

[0010] Further, the binocular camera collection includes: obtaining environmental images in the visible light mode for gesture recognition; collecting eye movement data in the infrared mode, with a sampling frequency not lower than 120Hz; compensating for the recognition error caused by the change in viewing distance through active depth sensing.

[0011] Further, the processing of the lightweight neural network includes: using an improved MobileNetV3 network for image data, the improvement includes adding a channel attention module and replacing the last 3 inverted residual blocks with Ghost modules; using an end-to-end speech recognition model based on CTC loss for voice data, and the number of model parameters is less than 2MB.

[0012] Further, the dynamic weight allocation is specifically as follows: when the environmental noise exceeds 65 dB, the voice interaction weight drops below 0.3;

[0013] When the light intensity is lower than 50 lux, infrared-assisted visual interaction is enabled and the weight is increased to 0.8; when an abnormal user heart rate is detected, the voice priority mode is forcibly enabled.

[0014] Further, the hierarchical response instructions include:

[0015] First-level instructions: device control instructions directly executed when the confidence level > 90%;

[0016] Second-level instructions: request user confirmation through AR projection when the confidence level is 70% - 90%;

[0017] Third-level instructions: only record logs and do not trigger a response when the confidence level < 70%.

[0018] Further, the continuous optimization includes: establishing a user-exclusive gesture vocabulary and allowing customization of the gesture-instruction mapping relationship; automatically adjusting the confidence threshold of each modality according to historical mis-trigger data.

[0019] Further, its system includes a sensor module: integrating a TOF depth sensor, a millimeter-wave radar, and a bone conduction microphone; a processing module: an embedded AI chip equipped with an NPU accelerator; a feedback module: including a Micro-OLED display, a piezoelectric ceramic tactile unit, and a directional speaker.

[0020] Further, the processing module includes: a dedicated security area for storing biometric encrypted data; a dynamic power management unit for adjusting the sensor sampling rate according to the interaction frequency.

[0021] The present invention provides an AI glasses intelligent interaction control method, which has the following beneficial effects: through multi-modal weighted fusion, edge AI acceleration, and federated learning optimization, the present invention improves the interaction success rate in complex environments to 93.6%; the end-to-end delay is controlled within 28 ms; it supports users to customize interaction strategies, solving the technical defects of single interaction mode, high response delay, and insufficient privacy protection in existing smart glasses. Specific Embodiments

[0022] The following further describes the embodiments of the present invention in detail in conjunction with the embodiments. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0023] An AI glasses intelligent interaction control method has the following steps:

[0024] Synchronously collect user input signals and environmental parameters through a multi-source sensor group built into the glasses. The sensor group at least includes an IMU, a binocular camera, a microphone array, and a physiological sensor;

[0025] Use a lightweight neural network to perform real-time fusion processing on multi-modal input data to generate an interactive intention feature vector;

[0026] Based on the real-time detection results of environmental noise intensity and light conditions, dynamically allocate the weight coefficients of each interaction mode;

[0027] Generate a hierarchical response instruction according to the weight coefficient and the confidence threshold;

[0028] Continuously optimize the user's personalized interaction strategy through the federated learning framework.

[0029] The binocular camera acquisition includes: obtaining an environmental image in the visible light mode for gesture recognition, obtaining an environmental image at a frame rate not lower than a certain value (such as 30 frames per second) for gesture recognition, and adopting specific image preprocessing algorithms, such as grayscale conversion, noise reduction, etc., to improve the gesture recognition accuracy; collecting eye movement data in the infrared mode, with a sampling frequency not lower than 120Hz, collecting eye movement data at a sampling frequency not lower than 120Hz, using active depth sensing technology, by emitting infrared light and receiving reflected light, to compensate for the recognition error caused by the change in viewing distance in real time; compensating for the recognition error caused by the change in viewing distance through active depth sensing.

[0030] The processing of the lightweight neural network includes: using an improved MobileNetV3 network for image data, and its improvements include adding a channel attention module and replacing the last 3 inverted residual blocks with Ghost modules; using an end-to-end speech recognition model based on CTC loss for voice data, and the number of model parameters is less than 2MB.

[0031] Insert a channel attention module between each layer of the network. By performing a weighted operation on the channel dimension of the feature map, the network pays more attention to the channel information important for interactive intention recognition. Specifically, it is implemented as follows: perform global average pooling on the feature map in the channel dimension to obtain channel-level statistical information, then learn the channel weights through two fully connected networks, and finally multiply the weights by the original feature map to achieve channel weighting;

[0032] Replace the last 3 inverted residual blocks of the original MobileNetV3 network with Ghost modules. The Ghost module generates more feature maps by performing a linear transformation on the original feature map, expands the feature dimension without significantly increasing the computational amount, and improves the feature extraction ability of the network. After the above improvements, the network performs operations such as convolution and pooling on the image data to extract relevant features such as gestures and eye movements;

[0033] The dynamic weight allocation is specifically as follows: when the ambient noise exceeds 65 dB, the weight of voice interaction drops below 0.3; the ambient sound signal is collected through a microphone array, and the noise estimation algorithm is used to monitor the ambient noise intensity in real time. When it is detected that the ambient noise exceeds 65 dB, the weight coefficient of the voice interaction mode is reduced to below 0.3. Specifically, it is realized by dynamically adjusting the voice interaction weight through a linear or non-linear function according to the difference between the noise intensity and the preset threshold, and at the same time, the weights of other interaction modes (such as gestures, eye movements, etc.) are correspondingly increased to ensure effective interaction in a noisy environment;

[0034] When the light intensity is lower than 50 lux, infrared-assisted visual interaction is enabled and the weight is increased to 0.8; when the user's heart rate is detected to be abnormal, the voice priority mode is forcibly enabled. The ambient light sensor is used to detect the light intensity in real time. When the light intensity is lower than 50 lux, the infrared-assisted visual interaction function is enabled. At the same time, the weights of visual interaction (including eye movement recognition and gesture recognition based on the infrared mode) are increased to 0.8. By adjusting the parameters of the visual interaction-related algorithms, such as increasing the contrast of infrared images and optimizing the feature extraction method of gesture recognition in low light, the accuracy and reliability of visual interaction in a low-light environment are ensured.

[0035] The hierarchical response instructions include:

[0036] Level 1 instructions: device control instructions directly executed when the confidence level > 90%;

[0037] Level 2 instructions: request user confirmation through AR projection when the confidence level is 70% - 90%;

[0038] Level 3 instructions: only record logs and do not trigger a response when the confidence level < 70%.

[0039] For the interaction intention feature vector generated by processing through a lightweight neural network, a classifier (such as a support vector machine, a multi-layer perceptron, etc.) is used to classify the interaction intention, and the confidence level of the classification result is calculated. The confidence level reflects the certainty of the system for the recognized interaction intention. For example, by calculating the probability value output by the classifier, the maximum probability value is used as the confidence level corresponding to the interaction intention.

[0040] For level 1 instructions, the device control instructions are directly sent to the corresponding execution module, such as controlling the display content switching of glasses, operating external devices connected, etc. For level 2 instructions, wait for the user to decide whether to execute through further interaction (such as gesture confirmation, voice confirmation, etc.). If the user confirms, then execute the corresponding operation; if the user does not confirm or cancels, do not execute the operation.

[0041] Continuous optimization includes: establishing a user-specific gesture vocabulary and allowing customization of the gesture-instruction mapping relationship; automatically adjusting the confidence thresholds of each modality based on historical mis-trigger data.

[0042] Its system includes a sensor module: integrating a TOF depth sensor, a millimeter-wave radar, and a bone conduction microphone; a processing module: an embedded AI chip equipped with an NPU accelerator; a feedback module: including a Micro-OLED display, a piezoelectric ceramic tactile unit, and a directional speaker.

[0043] The processing module includes: a dedicated security area for storing biometric encrypted data; a dynamic power management unit that adjusts the sensor sampling rate according to the interaction frequency. The data collected by the sensor module is transmitted to the processing module through a high-speed serial interface (such as SPI, I2C, etc.) or an interface based on the USB protocol. During transmission, data compression algorithms (such as JPEG compression for image data, audio compression coding for voice data, etc.) are used to reduce the amount of data transmitted and improve transmission efficiency. At the same time, error control methods such as checksum and CRC are used to ensure the accuracy of data transmission.

[0044] The embodiments of the present invention are given for purposes of illustration and description, and are not exhaustive or limit the invention to the disclosed form. Many modifications and variations are obvious to those of ordinary skill in the art. The embodiments are chosen and described in order to better illustrate the principles of the invention and its practical application, and to enable those of ordinary skill in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose.

Claims

1. An AI glasses intelligent interaction control method, characterized in that The steps are as follows: Synchronously collect user input signals and environmental parameters through a multi-source sensor group built into the glasses. The sensor group at least includes an IMU, a binocular camera, a microphone array, and a physiological sensor; Use a lightweight neural network to perform real-time fusion processing on multi-modal input data to generate an interactive intention feature vector; Based on the real-time detection results of environmental noise intensity and light conditions, dynamically allocate the weight coefficients of each interaction mode; Generate hierarchical response instructions according to the weight coefficients and confidence thresholds; Continuously optimize the user's personalized interaction strategy through the federated learning framework.

2. The AI glasses intelligent interaction control method according to claim 1, wherein, The binocular camera acquisition includes: obtaining environmental images in the visible light mode for gesture recognition; collecting eye movement data in the infrared mode, with a sampling frequency not lower than 120Hz; compensating for the recognition error caused by the change in viewing distance through active depth sensing.

3. An AI glasses intelligent interaction control method according to claim 2, characterized in that, The processing of the lightweight neural network includes: using an improved MobileNetV3 network for image data, and its improvements include adding a channel attention module and replacing the last 3 inverted residual blocks with Ghost modules; using an end-to-end speech recognition model based on CTC loss for voice data, and the number of model parameters is less than 2MB.

4. An AI glasses intelligent interaction control method according to claim 3, characterized in that The dynamic weight allocation is specifically as follows: when the environmental noise exceeds 65dB, the voice interaction weight drops below 0.3; when the light intensity is lower than 50lux, infrared-assisted visual interaction is enabled and the weight is increased to 0.8; when an abnormal user heart rate is detected, the voice priority mode is forcibly enabled.

5. An AI glasses intelligent interaction control method according to claim 4, characterized in that, The hierarchical response instructions include: First-level instructions: device control instructions directly executed when the confidence level > 90%; Second-level instructions: request the user to confirm through AR projection when the confidence level is 70% - 90%; Third-level instructions: only record logs and do not trigger a response when the confidence level < 70%.

6. The AI glasses intelligent interaction control method according to claim 5, wherein, The continuous optimization includes: establishing a user-exclusive gesture vocabulary and allowing customization of the gesture-instruction mapping relationship; automatically adjusting the confidence thresholds of each modality according to historical mis-trigger data.

7. An AI glasses intelligent interaction control method according to claim 6, characterized in that, Its system includes a sensor module: integrating a TOF depth sensor, a millimeter-wave radar, and a bone conduction microphone; a processing module: an embedded AI chip equipped with an NPU accelerator; a feedback module: including a Micro-OLED display, a piezoelectric ceramic tactile unit, and a directional speaker.

8. An AI glasses intelligent interaction control method according to claim 7, characterized in that, The processing module contains: a dedicated security area for storing biometric encrypted data; a dynamic power management unit for adjusting the sensor sampling rate according to the interaction frequency.