Baton action recognition system for smart home control

Abstract

The invention relates to a baton action recognition system for intelligent home control, which comprises a hardware part and a software part, and is characterized in that the hardware part comprises a baton circuit and a receiver, and the baton circuit is internally provided with a multi-axis motion sensor, a processing chip, a key and an integrated chip; the multi-axis motion sensor is used for collecting acceleration and angular velocity data of the baton in a three-dimensional space, the processing chip is used for carrying out preprocessing and action recognition processing on the collected sensor data, the key is used for starting action collection, the integrated chip is used for sending an action recognition result to the receiver, and the receiver is used for receiving the action recognition result. A brand-new interaction mode is provided, a user can complete tasks needing complex operation in the past only through gesture actions, the interaction process of the smart home is greatly simplified, the system combines innovative design of hardware and software, especially efficient operation of the data preprocessing module and the action recognition algorithm module, and the interaction efficiency is greatly improved. And convenient experience is brought to the user.

Description

Technical Field

[0001] The present invention relates to the field of smart home control, and in particular to a baton motion recognition system for smart home control. Background Art

[0002] Smart home control technology has developed rapidly in recent years. Traditional control methods mainly include the following categories:

[0003] 1. Voice control: Home devices can be operated through voice assistants (such as Alexa, Google Assistant, Siri, etc.). However, voice control has limitations in noisy environments, voice recognition accuracy, and privacy protection.

[0004] 2. Mobile device control: Users operate home devices through apps on smartphones or tablets. This method requires real-time operation of the device, which is not always convenient, especially when the user does not have the device in his hand or the device is powered off.

[0005] 3. Physical buttons or remote controls: Although simple to operate, they require users to touch the device or find the remote control, which is not flexible enough.

[0006] In recent years, sensor-based somatosensory control has gradually emerged. Controlling devices by capturing human movements or gestures is a more natural way of human-computer interaction. However, gesture recognition devices (such as cameras or radars) currently on the market have the following problems:

[0007] (1) Camera equipment requires a clear field of view, and users may need to face the camera to operate it, which is greatly affected by light and angle.

[0008] (2) Radar-type gesture devices are mainly directional and can only effectively detect gesture movements in a specific direction. This directionality limits the scope of use of the device, making it less suitable in certain application scenarios.

[0009] To solve the above problems, the present invention proposes a smart home control system based on baton motion recognition. The baton collects motion data through a built-in six-axis motion sensor (accelerometer and gyroscope), and combines it with a deep learning algorithm to achieve accurate command recognition. This method does not require a camera or specific sensors, has a wide range of applications, a low threshold for use, and strong adaptability. Users only need to hold the baton to operate quickly, which reduces device dependence and usage thresholds, and has high flexibility and adaptability. Summary of the invention

[0010] The purpose of the present invention is to overcome the shortcomings of the prior art, adapt to actual needs, and provide a baton motion recognition system for smart home control to solve the problem that current camera devices require a clear line of sight, and users may need to face the camera to operate, which is greatly affected by light and angle; although radar-type gesture devices have certain advantages, their implementation cost is high and the technical problem of limited ability to recognize fine movements is solved.

[0011] In order to achieve the purpose of the present invention, the technical solution adopted by the present invention is as follows: a baton motion recognition system for smart home control is designed, including a hardware part and a software part. The hardware part includes a baton circuit and a receiver. The baton circuit is built with a six-axis motion sensor, a processing chip, a button and an integrated chip. The six-axis motion sensor is used to collect acceleration and angular velocity data of the baton in three-dimensional space. The processing chip is used to pre-process the collected sensor data and process the motion recognition. The button is used to start motion collection. The integrated chip is used to send the motion recognition result to the receiver;

[0012] The software part includes a data preprocessing module and a motion recognition algorithm module; the data preprocessing module is used to filter, extract features and normalize the raw data collected by the six-axis motion sensor. Data filtering is used to eliminate high-frequency noise, feature extraction is used to extract features in the time domain and frequency domain, and data standardization is used to normalize the data. The motion recognition algorithm module uses a convolutional neural network to perform motion recognition on the preprocessed data and generate control instructions.

[0013] Preferably, the six-axis motion sensor is composed of a three-axis accelerometer and a three-axis gyroscope. After pressing the button, the six-axis sensor samples 128 points at a frequency of 80 Hz, and each point includes six data, representing acceleration and angular velocity in three directions respectively.

[0014] Preferably, the data preprocessing module further comprises an angle conversion step, which converts the measured motion data with the baton itself as the reference system into horizontal and vertical motion components with the ground as the reference system.

[0015] Preferably, the one-dimensional convolutional neural network structure adopted by the action recognition algorithm module includes a data input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer, which converts the preprocessed sensor data into a time series feature vector to form an input tensor, extracts time series features through the convolution layer, reduces the dimension through the pooling layer, and classifies through the fully connected layer, classifies the actions into a predefined action set, and outputs the recognition result.

[0016] Preferably, the predefined action set includes but is not limited to lighting control, curtain control, air conditioning control, music playback control, TV / media device control, security monitoring, and other home appliance control.

[0017] Preferably, the processing chip model in the baton circuit is ESP32-C3 or ESP32-S3, which is an embedded processor with a default main frequency of 160MHz, capable of collecting six-axis sensor signals, signal processing and neural network reasoning, and finally sending the results to the receiver via Bluetooth, Zigbee protocol or WiFi protocol.

[0018] Preferably, the receiver is responsible for receiving the control signal emitted by the baton and forwarding it to the smart home system. The receiver is an independent hardware device or a plug-in / software integrated into the smart home platform, which is directly integrated into the existing smart home gateway and supports compatibility with common smart home protocols, thereby realizing control of home appliances.

[0019] Preferably, the time domain and frequency domain extracted features include but are not limited to the mean, standard deviation, and power spectral density of acceleration.

[0020] Compared with the prior art, the present invention has the following beneficial effects:

[0021] 1. The present invention provides a new interaction mode, allowing users to complete tasks that previously required complex operations with gestures alone, greatly simplifying the interaction process of smart homes. The system combines the innovative design of hardware and software, especially the efficient operation of the data preprocessing module and the motion recognition algorithm module, bringing a convenient experience to users.

[0022] 2. The motion recognition algorithm module of the present invention adopts convolutional neural network (CNN) technology to achieve high-precision motion recognition. The deep learning algorithm can accurately capture and recognize the user's gestures, greatly reducing the error rate and improving the overall efficiency of the system and user experience. The user only needs to hold the baton and draw specific movements in the air to easily control smart home devices without the need for additional cameras or sensor arrays, truly realizing seamless control of the entire scene.

[0023] 3. The present invention, through its precise motion recognition capability and convenient interaction method, makes the technology not only suitable for the smart home field, but also can be widely used in multiple fields such as industrial control and educational interaction. With the continuous development of the Internet of Things and artificial intelligence technology, the market application prospect of the present invention will become more and more broad, bringing users a more intelligent and convenient life and work experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the process of the present invention; DETAILED DESCRIPTION

[0025] The present invention is further described below in conjunction with the accompanying drawings and embodiments:

[0026] A baton motion recognition system for smart home control, see Figure 1 , including hardware and software parts. The hardware part includes a baton circuit and a receiver. The baton circuit has a built-in six-axis motion sensor, a processing chip, a button and an integrated chip. The six-axis motion sensor is used to collect acceleration and angular velocity data of the baton in three-dimensional space. The processing chip is used to pre-process the collected sensor data and process motion recognition. The button is used to start motion collection. The integrated chip is used to send the motion recognition result to the receiver;

[0027] The software part includes a data preprocessing module and a motion recognition algorithm module; the data preprocessing module is used to filter, extract features and normalize the raw data collected by the six-axis motion sensor. Data filtering is used to eliminate high-frequency noise, feature extraction is used to extract features in the time domain and frequency domain, and data standardization is used to normalize the data. The motion recognition algorithm module uses a convolutional neural network to perform motion recognition on the preprocessed data and generate control instructions.

[0028] For details, see Figure 1 The six-axis motion sensor consists of a three-axis accelerometer and a three-axis gyroscope. After pressing the button, the six-axis sensor samples 128 points at a frequency of 80Hz. Each point includes six data, representing the acceleration and angular velocity in three directions respectively.

[0029] For more details, see Figure 1 ,The data preprocessing module also includes an angle transformation step, which converts the measured ,motion data with the baton itself as the reference system into the horizontal and vertical ,motion components with the ground as the reference system.

[0030] For further information, see Figure 1 ,The one-dimensional convolutional neural network structure adopted by the ,action recognition algorithm module includes data input layer, ,convolution layer, pooling layer, fully connected layer and output layer, ,which converts the preprocessed sensor data into time series feature ,vectors to form the input tensor, extracts the time series features through the ,convolution layer, reduces the dimension through the pooling layer, ,classifies the actions into predefined action sets, and outputs the ,recognition result.

[0031] Further, see Figure 1 ,The predefined action set includes but is not limited to lighting control, curtain control, air conditioning control, music ,playing control, TV / media device control, security monitoring, and other home ,appliances control.

[0032] It is worth noting that see Figure 1,The processing chip model in the baton circuit is ESP32-C3 or ESP32-S3, which is an embedded processor with a default 160MHz main frequency, which can collect six-axis sensor signals, signal processing and neural network reasoning, and finally send the results to the receiver via Bluetooth, Zigbee protocol or WiFi protocol.

[0033] It is worth noting that see Figure 1 The receiver is responsible for receiving the control signal sent by the baton and forwarding it to the smart home system. The receiver is an independent hardware device or a plug-in / software integrated in the smart home platform. It is directly integrated into the existing smart home gateway and supports compatibility with common smart home protocols, thereby realizing control of home devices.

[0034] It is worth mentioning that see Figure 1 ,The features extracted in the time domain and frequency domain include but are not limited to the mean, standard deviation, and power spectral density of acceleration.

[0035] A baton motion recognition system for smart home control includes the following workflow:

[0036] S1. The user presses a button on the baton to start action recording: The baton is a core interactive device, and a button for starting action collection is set on it. When the user intends to control the smart home device, the action recording process is started by pressing a button on the baton;

[0037] S2. The six-axis sensor of the baton collects motion data and completes preprocessing through the processing chip: a six-axis motion sensor is set inside the baton, which consists of a three-axis accelerometer and a three-axis gyroscope, and is used to collect acceleration and angular velocity data of the baton in three-dimensional space; when the user presses a button to start recording the action, the six-axis sensor collects data points at a predetermined frequency (such as 80Hz) to form a data set with a dimension of (128, 6); after receiving the sensor data, the processing chip performs data preprocessing operations, including data filtering, feature extraction and data standardization, to eliminate noise, extract effective features and prepare data input format for subsequent neural network processing;

[0038] S3. The processed data is used for action recognition through a convolutional neural network to generate control instructions: The preprocessed data is input into a convolutional neural network (CNN) for action recognition; the convolutional neural network extracts time series features and classifies actions through multi-layer convolution, pooling, and full connection operations to generate corresponding control instructions; the control instructions are matched with a predefined action set, such as "turn on the light", "turn off the light", etc.;

[0039] S4. The command is sent to the receiver via the wireless module: the baton is provided with an integrated chip (such as Wi-Fi, Bluetooth or Zigbee) for wirelessly sending the generated control command to the receiver; the receiver can be an independent device or integrated into a smart home gateway, such as the Mijia gateway, which is responsible for receiving the command from the baton;

[0040] S5. After the receiver receives the command, the corresponding device is controlled through the smart home system: After the receiver receives the control command, it parses the command and controls the corresponding home device to perform corresponding operations through the smart home system (such as Mijia) according to the command content; supported home devices include turning on and off lights, adjusting brightness, electric curtains, audio, etc.

[0041] S6, the processor on the baton can complete the collection of six-axis sensor signals, signal processing and neural network reasoning, and finally send the results to the receiver via Bluetooth, Zigbee protocol or WiFi protocol. The receiver is connected to smart home systems such as Mi Home via Bluetooth or WiFi, and the home appliances can be controlled through Mi Home.

[0042] Embodiment 1

[0043] Data preprocessing

[0044] Since the measured motion data is based on the baton itself as the reference system, and the user's waving motion needs to be recognized with the ground as the reference system, we refer to the paper, "A Feature Extraction Method for Realtime Human Activity Recognition on Cell Phones", use some angle transformations to obtain the horizontal and vertical motion components with the ground as the reference system, and obtain data with a dimension of (128, 4)

[0045] Data collection: When the user presses the start button on the baton, the six-axis motion sensor begins to collect acceleration and angular velocity data, forming a data set with a dimension of (128, 6), where each data point contains acceleration data in three directions and angular velocity data in three directions.

[0046] Angle transformation: In order to convert the collected data from the baton's own reference system to the ground reference system, the system uses the following steps to perform angle transformation: Calculate the baton's pitch and yaw angles, which can be calculated from the data of the six-axis sensor.

[0047] The acceleration data and angular velocity data are transformed from the baton's own reference system to the ground reference system using the tilt angle and yaw angle. The transformed data includes the horizontal and vertical acceleration components (Ax, Ay) and the horizontal and vertical angular velocity components (Gx, Gy).

[0048] After the transformation, the data dimension is reduced from (128, 6) to (128, 4), that is, each data point contains acceleration data in two directions and two angular velocity data.

[0049] Data standardization: Filter, extract features and standardize the transformed data to provide input for the action recognition algorithm module.

[0050] Numerical example:

[0051] Original data (with the baton itself as the reference system):

[0052] Acceleration: Ax = 1g, Ay = 0g, Az = 0g (indicates that the baton is horizontal to the right)

[0053] Angular velocity: Gx = 0.2 rad / s, Gy = 0 rad / s, Gz = -0.1 rad / s (indicates that the baton rotates clockwise around the X-axis at a rate of 0.2 rad / s, and counterclockwise around the Z-axis at a rate of 0.1 rad / s, and there is no rotation along the Y-axis)

[0054] Assume that the baton's tilt angle pitch=30° and its yaw angle yaw=0° (indicating that the baton is tilted 30° to the right and has no horizontal deflection).

[0055] Transformed data (with the ground as the reference system):

[0056] Acceleration data:

[0057] Horizontal acceleration component (Ax'): 0.866g (cos(30°))

[0058] Vertical acceleration component (Ay'): 0.5g (sin(30°))

[0059] Angular velocity data:

[0060] Transformed X-axis angular velocity component (Gx'): ≈0.173 rad / s (decreased by the inclination angle)

[0061] Angular velocity component in the Y-axis direction due to the tilt angle (Gy'): ≈-0.058 rad / s (indicating a slight rotation of the baton around the Y-axis in the ground reference frame).

[0062] Embodiment 2

[0063] Time series action recognition and quantization deployment based on multi-channel deep convolutional neural network

[0064] A time series classification method based on a multi-channel deep convolutional neural network (CNN) is used to identify the actions performed by the user through the baton. This method refers to the network structure design in the paper "Time Series Classification UsingMulti-Channels Deep Convolutional Neural Networks", but is modified and optimized according to actual needs. After training, the network model is quantized to int8 precision and deployed on an embedded processor (such as ESP32-C3) to achieve low power consumption and high efficiency real-time action recognition.

[0065] Network structure:

[0066] Input layer: accepts preprocessed time series data;

[0067] Multi-channel convolutional layer:

[0068] The first convolutional layer: contains multiple convolution kernels, each of which is responsible for extracting different features;

[0069] Pooling layer: Perform a maximum pooling operation on the output of the first convolutional layer to reduce the size of the feature map while retaining important features;

[0070] Subsequent convolutional layers: More convolutional and pooling layers can be added to extract deeper features. The output dimension of each layer varies depending on the kernel size, stride, and padding.

[0071] Fully connected layer: Flatten the output of the convolutional layer into a one-dimensional vector and perform feature fusion and classification through a fully connected layer. The number of output nodes of the last fully connected layer matches the number of action categories.

[0072] Output layer: Use the softmax function to normalize the output of the fully connected layer to obtain the probability distribution of each action category.

[0073] Quantification and deployment:

[0074] Training: Use a large amount of labeled action data to train the network until a satisfactory accuracy is achieved;

[0075] Quantization: After training is completed, use quantization tools (such as TensorFlow Lite's quantization function) to quantize the network model from floating-point precision (such as float32) to int8 precision. The quantization process includes the quantization of weights and activation values ​​to reduce the model size and computational complexity while maintaining model performance as much as possible.

[0076] Deployment: Deploy the quantized model on an embedded processor (such as ESP32-C3), write code using an embedded development environment (such as Arduino IDE or ESP-IDF), load the quantized model into the processor, and write interface functions to receive the preprocessed time series data, perform model inference, and output action recognition results.

[0077] Numerical example:

[0078] Training data: contains 1000 samples, each sample is a time series data with a dimension of (128, 4), corresponding to 5 different action categories;

[0079] Training process: Use cross-validation and early stopping to prevent overfitting, and record performance indicators such as accuracy and loss during training;

[0080] Model performance before quantization: 96% accuracy on the validation set, model size is 97KB (floating point precision);

[0081] Model performance after quantization: After quantization to int8 precision, the model size is reduced to 33.5KB, while the accuracy on the validation set decreases by about 0.5%;

[0082] Post-deployment performance: When deployed on ESP32-C3, the inference time is less than 200ms; when deployed on ESP32-S3, the inference time is less than 100ms, meeting the requirements of real-time action recognition.

[0083] Note: The above values ​​are only examples. The network structure, training process, quantization effect, and deployment performance in the actual system will vary based on the specific implementation and hardware conditions.

[0084] By implementing this embodiment, the system can efficiently recognize the actions performed by the user through the baton, and transmit the recognition results to the smart home system in real time to perform corresponding control operations. At the same time, the model size and computational complexity are reduced through quantization technology, so that the system can run on a low-power embedded processor.

[0085] In summary, the present invention realizes precise control of smart home devices by recognizing the action of the user waving a baton in the air. Through simple and intuitive gesture operations, the user's interactive experience can be improved, and the dependence on traditional touch or voice control can be reduced, providing a novel, convenient and efficient way for smart home control.

[0086] In addition, the components designed in the present invention are all universal standard parts or components known to technical personnel in this field. Their structures and principles can be known to technical personnel through technical manuals or through conventional experimental methods. They can be fully implemented by technical personnel in this field. Needless to say, the content protected by the present invention does not involve improvements to internal structures and methods.

[0087] The embodiments of the present invention disclose preferred embodiments, but are not limited thereto. A person skilled in the art can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. However, as long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A baton motion recognition system for smart home control, comprising a hardware part and a software part, characterized in that: The hardware part includes a baton circuit and a receiver. The baton circuit has a built-in multi-axis motion sensor, a processing chip, a button, and an integrated chip. The multi-axis motion sensor is used to collect acceleration and angular velocity data of the baton in three-dimensional space. The processing chip is used to process the collected sensor data and process motion recognition. The button is used to start motion collection, and the integrated chip is used to send the motion recognition results to the receiver. The software part includes a data preprocessing module and an action recognition algorithm module; the data preprocessing module is used to filter, extract features and normalize the raw data collected by the multi-axis motion sensor. Data filtering is used to eliminate high-frequency noise, feature extraction is used to extract features in the time domain and frequency domain, and data standardization is used to normalize the data. The action recognition algorithm module uses a convolutional neural network to perform action recognition on the preprocessed data and generate control instructions.

2. The baton motion recognition system for smart home control according to claim 1, characterized in that: The multi-axis motion sensor is composed of a three-axis accelerometer and a three-axis gyroscope. After pressing the button, the multi-axis motion sensor samples 128 points at one of the frequencies between 20Hz-2000Hz. Each point includes six data, representing the acceleration and angular velocity in three directions respectively.

3. The baton motion recognition system for smart home control according to claim 1, characterized in that: The data preprocessing module also includes an angle conversion step, which converts the measured motion data with the baton itself as the reference system into horizontal and vertical motion components with the ground as the reference system.

4. The baton motion recognition system for smart home control according to claim 1, characterized in that: The one-dimensional convolutional neural network structure adopted by the action recognition algorithm module includes a data input layer, a convolution layer, a pooling layer, a fully connected layer and an output layer. The preprocessed sensor data is converted into a time series feature vector to form an input tensor, and the time series features are extracted through the convolution layer. The dimension is reduced through the pooling layer, and the action is classified through the fully connected layer. The action is classified into a predefined action set and the recognition result is output.

5. The baton motion recognition system for smart home control according to claim 4, characterized in that: The predefined action set includes, but is not limited to, lighting control, curtain control, air conditioning control, music playback control, TV / media device control, security monitoring, and other home appliance control.

6. The baton motion recognition system for smart home control according to claim 1, characterized in that: The processing chip in the baton circuit is an embedded processor with a main frequency of 32MHz-320MHZ, which can complete the acquisition of multi-axis motion sensor signals, signal processing and neural network reasoning, and finally send the results to the receiver via Bluetooth, Zigbee protocol or WiFi protocol.

7. The baton motion recognition system for smart home control according to claim 1, characterized in that: The receiver is responsible for receiving the control signal emitted by the baton and forwarding it to the smart home system. The receiver is an independent hardware device or a plug-in / software integrated in the smart home platform. It is directly integrated into the existing smart home gateway and supports compatibility with common smart home protocols, thereby realizing control of home devices.

8. The baton motion recognition system for smart home control according to claim 1, characterized in that: The time domain and frequency domain extracted features include but are not limited to the mean, standard deviation, and power spectrum density of acceleration.

Images