Smart home man-machine interaction system based on inertial sensor gesture recognition

Abstract

The invention discloses a smart home man-machine interaction system based on inertial sensor gesture recognition, which performs inertial navigation by reading a bracelet acceleration gyroscope sensor to obtain the actual position and posture of the bracelet acceleration gyroscope sensor in the space, and can indirectly position the positions of other Internet of Things smart homes in the environment. And digital twinning of the smart home is realized. And the bracelet transmits the calculated position and attitude data to a local microcomputer at a base station end of the Internet of Things of the user in real time, and a neural network runs on the base station to classify gestures. Meanwhile, three-dimensional space calculation is carried out on the base station, the Internet of Things smart home equipment pointed by the hand of the user is obtained, and then corresponding smart home action commands are executed according to different gesture actions. A user only needs to wear one bracelet device to complete interaction with the Internet of Things smart home.

Description

Technical Field

[0001] The present invention relates to the technical field of smart home control, and in particular to the technical field of a smart home human-computer interaction method based on inertial sensor gesture recognition. Background Art

[0002] With the popularity of IoT applications, the need to use microcontrollers to run specific machine learning projects at ultra-low power consumption is becoming increasingly apparent. Smart home control is one of them. In order to improve the convenience of life, more and more solutions for controlling smart homes through the IoT have been launched, including voice recognition control methods, camera motion capture solutions, and remote control solutions through smart terminals such as mobile phones. However, all of the above solutions have certain problems. Among them, the voice recognition control method requires dialogue control, which will make a certain sound and is easily interfered with; the camera motion capture solution requires the camera to capture user movements, which has poor privacy; and the remote control through smart terminals such as mobile phones is cumbersome and requires opening the APP to perform a series of controls. Summary of the invention

[0003] The purpose of the present invention is to solve the problems in the prior art and to propose a smart home human-computer interaction system based on inertial sensor gesture recognition, which can control the smart home through a wristband.

[0004] To achieve the above object, the present invention proposes a smart home human-computer interaction system based on inertial sensor gesture recognition, comprising: A wristband that can be worn on a human body, wherein the wristband has a built-in acceleration gyro sensor (gyroscope and accelerometer), and the built-in wristband inertial navigation system obtains the acceleration gyro sensor data to calculate the user's actual position and posture information in space; The IoT gateway is connected to the wristband and the smart home devices to be controlled. The wristband transmits the calculated position and posture data to the IoT gateway in real time, performs three-dimensional space calculation on the IoT gateway, and processes the gesture signal and converts it into a control signal to control the smart home; responsible for the reasoning of the gesture recognition neural network; A gesture recognition neural network is used to analyze and recognize user gesture signals based on the accelerometer and gyroscope sensor data; The three-dimensional space modeling and solution system takes the IoT gateway as the world coordinate origin, establishes a coordinate system for the world space through the inertial navigation of the wristband, and then locates other IoT smart home devices to obtain the digital twin of the device; By obtaining the accelerometer-gyro sensor data and the smart home position, the posture of the bracelet is calculated, so as to determine the smart home device that the arm is specifically pointing to. By obtaining the IoT smart home device that the user's hand is pointing to, the gesture recognition neural network is run at the same time to classify the gestures, and then different commands are generated for different gesture actions, and the commands are sent to the relevant smart home commands for execution.

[0005] Preferably, the gesture recognition neural network and the three-dimensional space modeling and solution system are deployed in the Internet of Things gateway, and the gesture recognition neural network and the three-dimensional space solution are run through the Internet of Things gateway.

[0006] Preferably, the wristband is equipped with a microphone module for collecting user voice, and the IoT gateway is equipped with a speech recognition neural network. After the microphone module of the wristband is triggered, the speech recognition neural network obtains the user's voice text, converts the voice into text, and inputs the result into a large language model set in the cloud. The large language model outputs a text reply according to the voice text, and forwards the output result to the wristband for display through the wristband interactive interface.

[0007] Preferably, it also includes an indoor positioning system, which is based on the built-in low-power Bluetooth of the Internet of Things gateway, and is equipped with AOA positioning technology in the low-power Bluetooth. The direction of the electromagnetic wave is calculated by calculating the phase difference generated when an electromagnetic wave reaches different antennas, and the distance is calibrated and calculated by the intensity of the electromagnetic wave to locate the position of each smart home and bracelet.

[0008] Preferably, the gesture recognition neural network is constructed and trained using PyTorch, running a three-channel one-dimensional CNN convolutional neural network to perform machine learning on the multi-axis sensor waveform signals in the time domain and frequency domain.

[0009] Preferably, the bracelet, the IoT gateway and the smart home communicate wirelessly via BLE low-power Bluetooth.

[0010] Preferably, the specific steps of the wristband inertial navigation system to obtain the user's position and posture information include: combining the posture data of the gyroscope, first using a rotation matrix to perform coordinate transformation, and rotating the three axes of acceleration to the world coordinate system; in order to reduce the error, it is also necessary to calculate the linear interpolation between the two acceleration data to compensate for the error caused by the gyroscope sampling time; and performing a second integration of the acceleration to obtain the position information.

[0011] Preferably, the wristband inertial navigation system also includes dynamic drift prediction, and dynamically performs sensor data fusion through an EKF extended Kalman filter algorithm or a Madgwick algorithm.

[0012] Preferably, the three-dimensional space modeling and solving system converts the posture data of the bracelet into a rotation matrix to obtain a hand direction vector, and then calculates the three-dimensional space line-surface intersection problem and the point-line spacing problem, and determines whether the hand direction vector intersects with the smart home in the space; When they intersect, the gesture recognition neural network performs neural network reasoning. If a corresponding control gesture is made, the IoT gateway issues a control instruction, causing the corresponding IoT device to respond accordingly. The beneficial effects of the smart home human-computer interaction system based on inertial sensor gesture recognition of the present invention: the present invention reads the accelerometer gyroscope sensor of the wristband to perform inertial navigation, obtains its actual position and posture in space, and can indirectly locate the position of other IoT smart homes in the environment, thereby realizing the digital twin of the smart home. The wristband transmits the calculated position and posture data to the user's local IoT gateway microcomputer in real time, and runs a neural network on the base station to classify the gestures. At the same time, a three-dimensional space solution is performed on the base station to obtain the IoT smart home device pointed by the user's hand, and then the corresponding smart home action commands are executed for different gesture actions. The present invention innovatively adds an IoT gateway end, and allocates the computing tasks on battery devices such as wristbands that are not suitable for large-scale computing to the base station, which greatly reduces the power consumption of battery devices and improves the intelligence of the system. Users only need to wear a wristband device to complete the interaction with the IoT smart home. Directly detecting user gestures through a wristband is a very radical solution, but it is very suitable for the convenience of users wearing it in life. Through the new system configuration of the present invention, more complex action detection can be achieved while avoiding the power consumption problem of battery devices. The smart devices in the home are spatially positioned, and combined with the IoT properties of the devices themselves, a digital twin of the smart home is constructed, which is very convenient for docking with AR applications after future technological developments. Only the wristband sensor is used for gesture detection, which greatly reduces the infringement of user privacy.

[0013] The features and advantages of the present invention will be described in detail through embodiments in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of a smart home human-computer interaction system based on inertial sensor gesture recognition of the present invention.

[0015] Figure 2 It is a schematic diagram of a wristband of a smart home human-computer interaction system based on inertial sensor gesture recognition of the present invention.

[0016] Figure 3 It is a schematic diagram of the process of calculating the linear interpolation between two acceleration data to compensate for the error caused by the gyroscope sampling time in the present invention.

[0017] Figure 4 The three-dimensional space modeling and solving system of the present invention generates a three-dimensional space modeling schematic diagram.

[0018] Figure 5 It is a schematic diagram of the screen being controlled by a mouse pointer in the three-dimensional space modeling and solving system of the present invention.

[0019] Figure 6 It is a schematic diagram of the present invention when controlling a lamp.

[0020] Figure 7 This is a schematic diagram of controlling via a mouse pointer on a TV screen.

[0021] Figure 8 This is a diagram of the bracelet’s voice recognition dialogue.

[0022] Fig. 9 It is an overall block diagram of the system in the embodiment.

[0023] in: DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below through the accompanying drawings and embodiments. However, it should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of the present invention. In addition, in the following description, the description of known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0025] In the description of the present invention, it should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it may be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it may be directly connected to the other element or indirectly connected to the other element.

[0026] In the description of the present invention, it should be noted that the terms "center", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, or the positions or positional relationships in which the invention product is usually placed when in use, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined. "Several" means one or more, unless otherwise clearly and specifically defined.

[0027] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. Embodiment 1

[0028] See also Figure 1 , Figure 2 and Fig. 9 The present invention discloses a smart home human-computer interaction system based on inertial sensor gesture recognition, comprising a wristband that can be worn on a human body, the wristband having a built-in acceleration gyroscope sensor, and the wristband having a built-in wristband inertial navigation system, which obtains the actual position and posture information of the user in space by calculating the acceleration gyroscope sensor data; the wristband has its own basic functions; The IoT gateway is connected to the communication between the wristband and the smart home devices to be controlled. The wristband transmits the calculated position and posture data to the IoT gateway in real time, performs three-dimensional space calculation on the IoT gateway, and processes the gesture signal and converts it into a control signal to control the smart home; it is responsible for the reasoning of the gesture recognition neural network; in this embodiment, the IoT gateway is a high-performance Linux microcomputer, so it is suitable for running neural networks on it and performing edge computing; The gesture recognition neural network is used to analyze and recognize user gesture signals based on the accelerometer and gyroscope sensor data. The project finally uses Pytorch for neural network training, converts to ONNX and OpenVINO IR, and uses NNCF for INT8 quantization. Finally, OpenVINO is used to call the core graphics acceleration reasoning on the IoT gateway platform. The gesture recognition deep CNN network uses a three-channel Conv1D as the input layer, with two layers of convolution and two layers of full connection, and finally uses the Softmax activation function for output. In this embodiment, gestures include gesture classifications such as "turning the wrist", "rubbing fingers three times", and "no gesture"; 3D space modeling and solving system, see Figure 4 , Figure 5 , taking the IoT gateway as the world coordinate origin, establish a coordinate system for the world space through the inertial navigation of the bracelet, and then locate other IoT smart home devices to obtain the digital twin of the device; PyTorch is used in this embodiment to run a special CNN convolutional neural network or RNN recurrent neural network to perform machine learning on the multi-axis sensor waveform signals in the time domain and frequency domain. It has good performance and is also suitable for calculation on the IoT gateway. The feasibility of using CNN convolutional neural network to classify time domain and frequency domain signals can be found in the article "Signal modulation recognition and classification method based on time-frequency distribution and CNN"; See also Figure 6 and Figure 7 By obtaining the accelerometer gyro sensor data and the smart home position, the posture of the bracelet is calculated, so as to determine the smart home device that the arm is specifically pointing to. By obtaining the IoT smart home device that the user's hand is pointing to, the gesture recognition neural network is run at the same time to classify the gestures, and then different commands are generated for different gesture actions, and the commands are sent to the relevant smart home commands for execution.

[0029] In this embodiment, the gesture recognition neural network and the three-dimensional space modeling and solving system are deployed in the IoT gateway, and the gesture recognition neural network and the three-dimensional space solving are run through the IoT gateway. This can reduce the power consumption of the wristband battery device and achieve a highly cohesive and highly coupled system architecture, improve the battery life of the wristband, and reduce the cost and size of the wristband.

[0030] Among them, in the bracelet inertial navigation system, the data obtained by the acceleration gyro sensor includes the acceleration of gravity, so the first thing to do is to calibrate the zero point. Combined with the gyro data, calculate the magnitude of the acceleration perpendicular to the ground read when stationary, and remove the acceleration of gravity in subsequent calculations.

[0031] The specific method is to measure the modulus of the three-axis components of the accelerometer when the bracelet is absolutely stationary, that is, the local gravity acceleration. In the subsequent inertial navigation calculation, the acceleration value is combined with the attitude angle through the rotation matrix to transform it to the world system, and then the difference is made in the Z-axis direction to obtain the acceleration of the bracelet in the stationary non-inertial system.

[0032] In order to obtain the position, the acceleration needs to be integrated twice. However, before starting the integration, the three axes of the acceleration need to be rotated to the world coordinate system. Therefore, it is necessary to combine the gyroscope's attitude data and use a rotation matrix for coordinate transformation. Figure 3 ,In order to reduce the error, we also need to calculate the linear interpolation between the two acceleration data to compensate for the error caused by the gyroscope sampling time.

[0033] Since the acceleration gyro sensor will drift and the drift speed is uncertain, it is necessary to dynamically predict the drift. Initially consider the adaptive EKF extended Kalman filter algorithm. The EKF algorithm can predict the drift of the gyro relatively well in both dynamic and static conditions. Or consider using the Mahony algorithm for simple compensation. The disadvantage is that it will generate additional errors in dynamic conditions. Among them, the EKF algorithm reference article: Quaternion EKF attitude update algorithm. The Mahony algorithm reference article: Detailed explanation of the Mahony attitude solution algorithm.

[0034] After the position and posture data are calculated, they are transmitted to the IoT gateway for further gesture recognition and smart home control through the gesture recognition neural network.

[0035] Regarding the three-dimensional space modeling and solution system, in order to improve the positioning accuracy, the three-dimensional space modeling and solution system of the components of this embodiment realizes the collision ball of the lamps and the control plane of the Internet of Things smart screen. The posture data of the bracelet is converted into a rotation matrix to obtain the hand direction vector, and then the three-dimensional space line-surface intersection problem and the point-line spacing problem are calculated.

[0036] When they intersect, neural network reasoning is executed. If a corresponding control gesture is made, the IoT gateway will issue a control instruction, causing the corresponding IoT device to respond.

[0037] We implemented switch control when the arm is pointing to the IoT lamp, and when the arm is pointing to the screen, we generated a mouse pointer and controlled the position of the mouse pointer to the direction the arm is pointing, and responded to left-click and ESC-key click functions according to the finger and wrist movements.

[0038] In this embodiment, the switches of each smart home in the three-dimensional space modeling and solution system can be abstracted as a collision ball in the three-dimensional space, and the direction of the wristband arm can be abstracted as a ray, and the intersection relationship between the two is calculated. When they intersect and the user's wristband makes a corresponding gesture, the Internet of Things terminal controls the lamp to make a relevant response.

[0039] The arm gesture TV remote control in the project advanced goal, in which the large screen can be abstracted as a rectangular plane in a three-dimensional space, and the three points of the plane are located through the bracelet to obtain its actual position and size in space. The coordinates of the intersection of the arm ray and the plane are calculated, and converted to the corresponding pixel coordinates on the large screen through linear transformation, and then the corresponding cursor movement operation is performed through the virtual USB mouse connected to the TV. Different gestures made by the user can correspond to the click operation of the mouse and the return operation of the remote control. The collision ball of the smart home switch in the three-dimensional space and the control plane of the IoT smart screen are realized. The hand direction vector is obtained after the posture data of the bracelet is converted into a rotation matrix, and then the line-surface intersection problem and the point-line spacing problem in the three-dimensional space are calculated.

[0040] Preferably, the bracelet, the IoT gateway and the smart home all communicate wirelessly via BLE low-power Bluetooth. Embodiment 2

[0041] Based on the first embodiment, refer to Figure 8 , the wristband is equipped with a microphone module for collecting user voices, the IoT gateway is equipped with a speech recognition neural network, and the speech recognition neural network is a Whisper speech recognition neural network. After the microphone module of the wristband is triggered, the speech recognition neural network obtains the user's voice text, converts the voice into text results and inputs them into a large language model set in the cloud or IoT gateway. The large language model outputs a text reply according to the voice text, and forwards the output result to the wristband for display through the wristband interactive interface. The large language model is a 7B parameter MistralAI large language model. In this embodiment, an IoT gateway bridges the wristband and the cloud, so that the wristband can directly communicate with the large model. At the same time, the speech recognition task is executed on the base station side, which does not occupy the user's private computing power. After the computing power of the edge side is improved in the future, the large model can be directly run on the IoT gateway, and the local private offline large model can be directly realized, which has great potential. In this project, this combination is realized by fine-tuning the large model so that it has the performance of an IoT smart butler. Specifically, the large model can further control the state of the device in the IoT through system calls or text instructions based on the understanding of the user's voice text, so as to realize the IoT voice butler.

[0042] Whisper is a general-purpose speech recognition model. It is trained on large datasets of various audios and is a multi-task model that can perform multilingual speech recognition, speech translation, and language identification. This embodiment also integrates indoor positioning technology to locate the position of the bracelet and each smart home device. The user can control the device pointed by the arm by raising his hand at any position. The indoor positioning system is based on the built-in low-power Bluetooth of the Internet of Things gateway, and is equipped with AOA positioning technology in the low-power Bluetooth. The direction of the electromagnetic wave is calculated by calculating the phase difference generated by an electromagnetic wave reaching different antennas, and then the distance is calibrated and calculated by the intensity of the electromagnetic wave to locate the position of each smart home and bracelet. The positioning accuracy can reach the sub-meter level. On this basis, combined with the 9-axis IMU inertial navigation, a Kalman filter is input to achieve an indoor navigation effect with absolute zero point calibration, and the accuracy is expected to reach the decimeter level. Integrate the AOA positioning technology into the Internet of Things gateway to reduce the size of the entire set of equipment and improve positioning accuracy.

[0043] The standard parts used in this application document can all be purchased from the market. The specific connection methods of each part adopt conventional means such as mature main control chips, computing platforms, communication protocol standards, interface standards, battery specifications, etc. in the existing technology. Workers can complete normal operation according to the existing technical manual. In addition, the circuit connection adopts the conventional connection method in the existing technology, which will not be described in detail here.

[0044] It should be noted that, although the above embodiments have been described in this article, the scope of patent protection of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields are all included in the scope of protection of the patent of the present invention.

Claims

1. A smart home human-computer interaction system based on inertial sensor gesture recognition, characterized in that: include: A wristband that can be worn on a human body, wherein the wristband has a built-in acceleration gyro sensor (gyroscope and accelerometer), and the built-in wristband inertial navigation system obtains the acceleration gyro sensor data to calculate the user's actual position and posture information in space; The IoT gateway is connected to the wristband and the smart home devices to be controlled. The wristband transmits the calculated position and posture data to the IoT gateway in real time, performs three-dimensional space calculation on the IoT gateway, and processes the gesture signal and converts it into a control signal to control the smart home; responsible for the reasoning of the gesture recognition neural network; A gesture recognition neural network is used to analyze and recognize user gesture signals based on the accelerometer and gyroscope sensor data; The three-dimensional space modeling and solution system takes the IoT gateway as the world coordinate origin, establishes a coordinate system for the world space through the inertial navigation of the wristband, and then locates other IoT smart home devices to obtain the digital twin of the device; By obtaining the accelerometer-gyro sensor data and the smart home position, the posture of the bracelet is calculated, so as to determine the smart home device that the arm is specifically pointing to. By obtaining the IoT smart home device that the user's hand is pointing to, the gesture recognition neural network is run at the same time to classify the gestures, and then different commands are generated for different gesture actions, and the commands are sent to the relevant smart home commands for execution.

2. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: The gesture recognition neural network and the three-dimensional space modeling and solution system are deployed in the Internet of Things gateway, and the gesture recognition neural network and the three-dimensional space solution are run through the Internet of Things gateway.

3. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: The wristband is equipped with a microphone module for collecting user voice, and the IoT gateway is equipped with a speech recognition neural network. After the microphone module of the wristband is triggered, the speech recognition neural network obtains the user's voice text, converts the voice into text, and inputs the result into a large language model set in the cloud or the IoT gateway. The large language model outputs a text reply according to the voice text, and forwards the output result to the wristband for display through the wristband interactive interface.

4. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: It also includes an indoor positioning system, which is based on the built-in low-power Bluetooth of the Internet of Things gateway and is equipped with AOA positioning technology. The direction of the electromagnetic wave is calculated by calculating the phase difference generated when an electromagnetic wave reaches different antennas, and the distance is calibrated and calculated by the intensity of the electromagnetic wave to locate the position of each smart home and bracelet.

5. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: The gesture recognition neural network is built and trained using PyTorch, runs a three-channel one-dimensional CNN convolutional neural network, and performs machine learning on the multi-axis sensor waveform signals of the sensor in the time domain and frequency domain.

6. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: The wristband, the IoT gateway and the smart home all communicate wirelessly via BLE low-power Bluetooth.

7. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: The specific steps of the wristband inertial navigation system to obtain the user's position and posture information include: combining the posture data of the gyroscope, first using a rotation matrix to perform coordinate transformation, and rotating the three axes of acceleration to the world coordinate system; in order to reduce the error, it is also necessary to calculate the linear interpolation between the two acceleration data to compensate for the error caused by the gyroscope sampling time; and performing a second integration of the acceleration to obtain the position information.

8. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 7, characterized in that: The wristband inertial navigation system also includes dynamic prediction drift, and dynamically performs sensor data fusion through an EKF extended Kalman filter algorithm or a Madgwick algorithm.

9. The smart home human-computer interaction system based on inertial sensor gesture recognition as claimed in claim 1, characterized in that: The three-dimensional space modeling and solving system converts the posture data of the bracelet into a rotation matrix to obtain the hand direction vector, and then calculates the three-dimensional space line-surface intersection problem and the point-line spacing problem to determine whether the hand direction vector intersects with the smart home in the space; When they intersect, the gesture recognition neural network performs neural network reasoning. If a corresponding control gesture is made, the IoT gateway issues a control instruction, causing the corresponding IoT device to make a relevant response.

Images