Intelligent sound box based on gesture recognition and non-contact control method thereof

Abstract

The invention discloses an intelligent loudspeaker box based on gesture recognition and a non-contact control method thereof.The intelligent loudspeaker box comprises a multi-mode sensor array, an edge computing chip, an audio processing module and a wireless communication module, the multi-mode sensor array is used for collecting three-dimensional space information, motion parameters and posture data of gestures, and the edge computing chip is used for processing the three-dimensional space information, the motion parameters and the posture data of the gestures; the edge calculation chip is used for realizing gesture data processing and deep learning model reasoning, and the audio processing module is used for executing audio operation after receiving a gesture control instruction. The multi-modal sensor fusion and deep learning algorithm are combined, so that the gesture recognition accuracy is effectively improved compared with that of a traditional scheme; the stable performance is still kept under the dark light and complex background environment, local real-time processing is achieved through an edge computing architecture, the total delay from gesture collection to instruction execution is controlled within 80 ms, the real-time interaction requirement is met, a dynamic model optimization mechanism can complete personalized parameter adjustment, and the problem that gesture habits of different users are different is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of smart speaker interaction technology, specifically to a smart speaker based on gesture recognition and its contactless control method. Background Technology

[0002] Currently, smart speakers primarily rely on voice commands and touch panels for interactive control. Voice interaction is susceptible to environmental noise, dialect differences, and interference from multiple sound sources. For example, in family gatherings or noisy public places, speakers often misinterpret commands or fail to respond. Touch operation requires users to be close to the device, making it difficult to achieve convenient control in scenarios where hands are occupied, such as cooking or exercising. Existing gesture recognition technology applied to smart speakers suffers from high recognition latency, high misjudgment rates for complex actions, and poor adaptability to differences in user gesture habits. For instance, infrared sensors fail in low-light environments, and simple threshold judgment algorithms struggle to distinguish similar gestures, limiting the naturalness and reliability of smart speaker interaction. Therefore, there is an urgent need to develop a highly robust, low-latency contactless smart speaker control solution. This paper presents a gesture recognition-based smart speaker and its contactless control method to address these issues. Summary of the Invention

[0003] To address the problems mentioned in the background, the present invention provides a technical solution for a gesture recognition smart speaker, comprising a multimodal sensor array, an edge computing chip, an audio processing module, and a wireless communication module. The multimodal sensor array is used to collect three-dimensional spatial information, motion parameters, and posture data of gestures. The edge computing chip is used to perform gesture data processing and deep learning model inference. The audio processing module is used to execute audio operations after receiving gesture control commands. The wireless communication module is used to connect and communicate with external devices.

[0004] As a preferred embodiment of the present invention, the modal sensor array includes a ToF depth camera, a millimeter-wave radar, and an inertial measurement unit. The ToF depth camera is used to capture the three-dimensional spatial coordinate information of the gesture, the millimeter-wave radar is used to monitor the hand movement speed and acceleration in real time, and the inertial measurement unit records the wrist posture changes, thereby realizing omnidirectional data acquisition of the gesture.

[0005] As a preferred technical solution of the present invention, the edge computing chip is equipped with an AI acceleration chip and has a built-in deep learning inference engine, which can complete gesture data processing and recognition locally, reducing data transmission latency. The AI ​​acceleration chip is model H200 and adopts the Hopper architecture.

[0006] As a preferred embodiment of the present invention, the frequency processing module includes dual microphones and a high-fidelity speaker. The dual microphone array uses beamforming to locate the sound source and works in conjunction with gesture control to achieve directional volume adjustment.

[0007] As a preferred embodiment of the present invention, the wireless communication module supports WiFi and Bluetooth functions, and can be linked with smart home devices to realize gesture control of smart devices throughout the house.

[0008] A contactless control method for smart speakers based on gesture recognition includes the following steps: Step 1: Synchronously acquire gesture data using a multimodal sensor array and perform spatiotemporal alignment and preprocessing; Step 2: Dynamic feature extraction using a spatiotemporal Transformer network; Step 3: Construct a primary recognition model and a secondary recognition model, and achieve fast and accurate gesture determination through the hierarchical recognition model; Step 4: Establish a customizable gesture command mapping database. Users can configure personalized gesture commands through a mobile APP. The recognition results are converted into control commands by the semantic parsing module, which drive the audio processing module to perform playback control and smart home linkage operations. Step 5: Use an online learning mechanism for adaptive model optimization. Introduce an online learning mechanism so that when the system makes more than 3 consecutive misjudgments, it automatically triggers incremental learning, adds newly collected gesture data to the training set, and quickly updates the model parameters through transfer learning to adapt to the gesture habits of different users.

[0009] As a preferred technical solution of the present invention, in step one, the multimodal sensor array synchronously collects gesture data at a frequency of 200Hz, fuses different sensor data through a spatiotemporal alignment algorithm, and eliminates noise interference by median filtering and normalization processing to construct a gesture data vector containing 3D features.

[0010] As a preferred technical solution of the present invention, in step two, the spatiotemporal Transformer network introduces user behavior history data as a conditional vector, adaptively adjusts the weights of different features, and effectively distinguishes similar gestures.

[0011] As a preferred technical solution of the present invention, in step three, the primary recognition model uses a lightweight convolutional neural network to quickly filter invalid gestures, reducing the computational load, and the secondary recognition model uses a long short-term memory network to perform temporal analysis on complex continuous gestures, thereby improving the recognition accuracy.

[0012] The present invention has the following advantages: by combining multimodal sensor fusion with deep learning algorithms, the present invention effectively improves the accuracy of gesture recognition compared with traditional solutions, and maintains stable performance in low light and complex background environments. Local real-time processing is achieved through an edge computing architecture, with the total latency from gesture acquisition to command execution controlled within 80ms, meeting the requirements of real-time interaction. The dynamic model optimization mechanism can complete personalized parameter adjustments, effectively solve the problem of differences in gesture habits among different users, and realize cross-platform linkage between smart speakers and other smart home devices through gesture control, thus expanding application scenarios. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the structure of a smart speaker system with gesture recognition according to a preferred embodiment of the present invention; Figure 2 This is a schematic diagram of a non-contact control method for a smart speaker based on gesture recognition, according to a preferred embodiment of the present invention. Detailed Implementation

[0014] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0015] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Example 1 Please refer to the following: Figure 1 and Figure 2 The present invention relates to a smart speaker with gesture recognition, comprising: a multimodal sensor array, an edge computing chip, an audio processing module, and a wireless communication module. The multimodal sensor array is used to collect three-dimensional spatial information, motion parameters, and posture data of gestures. The edge computing chip is used to realize gesture data processing and deep learning model inference. The audio processing module is used to execute audio operations after receiving gesture control commands. The wireless communication module is used to connect and communicate with external devices.

[0017] The modal sensor array includes a ToF depth camera, millimeter-wave radar, and an inertial measurement unit (IMU). The ToF depth camera captures the three-dimensional spatial coordinates of gestures, the millimeter-wave radar monitors hand movement speed and acceleration in real time, and the IMU records wrist posture changes, achieving omnidirectional data acquisition of gestures. The edge computing chip, equipped with an AI acceleration chip and a built-in deep learning inference engine, can complete gesture data processing and recognition locally, reducing data transmission latency by more than 70%. The AI ​​acceleration chip is model H200, using the Hopper architecture. Hardware assembly: The ToF depth camera and millimeter-wave radar sensor are mounted at a specific angle on the top of the speaker to ensure no blind spots in gesture acquisition. The edge computing chip and sensors are connected via a high-speed SPI interface to ensure efficient data transmission.

[0018] The audio processing module includes dual microphones and a high-fidelity speaker. The dual microphone array uses beamforming for sound source localization and, in conjunction with gesture control, enables directional volume adjustment. The wireless communication module supports WiFi and Bluetooth, allowing integration with smart home devices for gesture-based control of all smart home devices. In daily use, when a user makes an upward hand gesture, the edge computing chip quickly recognizes it and sends the command to the audio processing module to increase the volume. If the system misinterprets the gesture, the user can trigger a model optimization process using a specific error-correction gesture.

[0019] Example 2 A contactless control method for smart speakers based on gesture recognition includes the following steps: Step 1: Synchronously acquire gesture data using a multimodal sensor array and perform spatiotemporal alignment and preprocessing; The multimodal sensor array synchronously acquires gesture data at a frequency of 200Hz, fuses data from different sensors using a spatiotemporal alignment algorithm, and eliminates noise interference using median filtering and normalization to construct a gesture data vector containing 3D features.

[0020] Step 2: Use a spatiotemporal Transformer network for dynamic feature extraction; the spatiotemporal Transformer network introduces historical user behavior data as a conditional vector, adaptively adjusts the weights of different features, and effectively distinguishes similar gestures (such as "swipe left and right" and "draw a circle").

[0021] Step 3: Construct a primary recognition model and a secondary recognition model. The primary recognition model uses a lightweight convolutional neural network (CNN) to quickly filter invalid gestures and reduce the computational load. The secondary recognition model uses a long short-term memory network (LSTM) to perform temporal analysis on complex continuous gestures, thereby improving the recognition accuracy.

[0022] Step 4: Establish a customizable gesture command mapping database. Users can configure personalized gesture commands via a mobile app. The recognition results are converted into control commands by the semantic parsing module, driving the audio processing module to perform playback control and smart home linkage operations. Step 5: Adaptive model optimization using an online learning mechanism. An online learning mechanism is introduced whereby incremental learning is automatically triggered when the system makes more than three consecutive misjudgments. Newly collected gesture data is added to the training set, and model parameters are quickly updated through transfer learning to adapt to the gesture habits of different users. Model Training: 200,000 sets of gesture data from 500 users of different ages and genders were collected, covering 15 basic gestures and 10 compound gestures. The spatiotemporal Transformer model was trained on the NVIDIA DGXA100 platform with a learning rate of 0.001 and a training cycle of 30 rounds.

[0023] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0024] Other parts of this invention that are not detailed herein are all prior art and will not be described further here.

[0025] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A smart speaker with gesture recognition, characterized in that, include: The system includes a multimodal sensor array, an edge computing chip, an audio processing module, and a wireless communication module. The multimodal sensor array is used to collect three-dimensional spatial information, motion parameters, and posture data of gestures. The edge computing chip is used to perform gesture data processing and deep learning model inference. The audio processing module is used to execute audio operations after receiving gesture control commands. The wireless communication module is used to connect and communicate with external devices.

2. The smart speaker with gesture recognition as described in claim 1, characterized in that, The modal sensor array includes a ToF depth camera, a millimeter-wave radar, and an inertial measurement unit. The ToF depth camera is used to capture the three-dimensional spatial coordinate information of the gesture, the millimeter-wave radar is used to monitor the hand movement speed and acceleration in real time, and the inertial measurement unit records the wrist posture changes, realizing omnidirectional data acquisition of the gesture.

3. The smart speaker with gesture recognition as described in claim 1, characterized in that, The edge computing chip is equipped with an AI acceleration chip and a built-in deep learning inference engine, which can complete gesture data processing and recognition locally, reducing data transmission latency. The AI ​​acceleration chip is model H200 and adopts the Hopper architecture.

4. The smart speaker with gesture recognition as described in claim 1, characterized in that, The audio processing module includes dual microphones and a high-fidelity speaker. The dual microphone array uses beamforming to locate the sound source and works with gesture control to achieve directional volume adjustment.

5. The smart speaker with gesture recognition as described in claim 1, characterized in that, The wireless communication module supports WiFi and Bluetooth, enabling it to connect with smart home devices.

6. The non-contact control method for a smart speaker based on gesture recognition as described in any one of claims 1-5, characterized in that, Includes the following steps: Step 1: Synchronously acquire gesture data using a multimodal sensor array and perform spatiotemporal alignment and preprocessing; Step 2: Dynamic feature extraction using a spatiotemporal Transformer network; Step 3: Construct a primary recognition model and a secondary recognition model, and achieve fast and accurate gesture determination through the hierarchical recognition model; Step 4: Establish a customizable gesture command mapping database. Users can configure personalized gesture commands through a mobile APP. The recognition results are converted into control commands by the semantic parsing module, which drive the audio processing module to perform playback control and smart home linkage operations. Step 5: Use an online learning mechanism for adaptive model optimization. Introduce an online learning mechanism so that when the system makes more than 3 consecutive misjudgments, it automatically triggers incremental learning, adds newly collected gesture data to the training set, and quickly updates the model parameters through transfer learning to adapt to the gesture habits of different users.

7. The non-contact control method for a smart speaker based on gesture recognition as described in claim 6, characterized in that, In step one, the multimodal sensor array synchronously collects gesture data at a frequency of 200Hz. The data from different sensors are fused through a spatiotemporal alignment algorithm, and noise interference is eliminated by median filtering and normalization to construct a gesture data vector containing 3D features.

8. The non-contact control method for a smart speaker based on gesture recognition as described in claim 6, characterized in that, In step two, the spatiotemporal Transformer network introduces historical user behavior data as a conditional vector, adaptively adjusting the weights of different features to effectively distinguish similar gestures.

9. The non-contact control method for a smart speaker based on gesture recognition as described in claim 6, characterized in that, In step three, the primary recognition model uses a lightweight convolutional neural network to quickly filter invalid gestures, reducing the computational load. The secondary recognition model uses a long short-term memory network to perform temporal analysis on complex continuous gestures, thereby improving the recognition accuracy.

Images