A True Wireless Headphone Interaction Method Based on IMU Facial Gesture Recognition

Abstract

The present invention is a true wireless headset interaction method based on IMU facial gesture recognition. The present invention trains a domain discriminator, a feature extractor based on a convolutional neural network, and a domain discriminator to distinguish different user identities based on their facial gestures; offline model training is performed to obtain a user-independent feature extractor and gesture classifier; after the offline model training is completed, the user discriminator is removed, and the feature extractor and gesture classifier are deployed to a personal device for model transfer learning and real-time gesture recognition. The present invention achieved an accuracy rate of 82.3% in user-independent tasks. User-independent tasks refer to tasks where the target user's data is not included in the offline model training data. In addition, after fine-tuning the model using a small amount of target user data, a gesture recognition accuracy rate of over 95% can be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of wireless headset interaction, and in particular to a true wireless headset interaction method based on IMU facial gesture recognition. Background Art

[0002] True wireless stereo (TWS) earbuds are now widely used. They integrate various wireless chips and sensors and are typically used with smartphones and personal computers. Their unique feature is their freedom from wires, allowing users to use them with ease. However, unlike other traditional smart devices, they are compact and lack touchscreen displays for control. Users also lack the ability to adjust volume, pause playback, and other functions through the wires of traditional earbuds. Early TWS manufacturers added physical buttons to control the earbuds, but this approach compromised their airtightness. In recent years, the most widely used wireless earbud interaction method has been developed by Apple, which incorporates an inertial measurement unit (IMU) into its AirPods to detect taps, enabling actions like changing songs and answering / ending calls. However, this interaction method currently has its challenges. Due to the small surface area of TWS earbuds, manufacturers can only offer a limited number of control functions. Furthermore, this method requires users to precisely locate tiny touch points on the earbuds, which can cause discomfort when pressing them. Similarly, other true wireless earbuds based on capacitive sensors and pressure sensors face similar challenges. Currently, some technologies have expanded the interaction surface between the human body and true wireless earbuds to the face, enriching the earbuds' control functions. For example, microphones can be used to detect the sounds produced by sliding the face with different gestures. However, using microphones to detect gestures often introduces a lot of ambient noise, resulting in reduced recognition accuracy. Furthermore, prolonged voice signal sampling consumes a significant amount of energy in the earbuds. Summary of the Invention

[0003] This invention aims to expand the interactive capabilities of existing true wireless earbuds and allow users to interact with them in a more convenient and easy way. Currently, most true wireless earbuds are equipped with an IMU to sense tapping gestures. This invention expands the scope of this universal sensing function to seven different facial gestures near the human ear. The core principle is that the vibration signals generated by the contact between the human finger and the skin are sensed by the IMU of the true wireless earbuds. Different gestures will produce different signal patterns. Using deep learning methods to classify these signals, these different gestures can be distinguished.

[0004] In addition, since gesture habits vary greatly among different users, the present invention uses Domain Adversarial Learning (DAL) to train offline models, suppressing user-specific features while enhancing gesture-related features, thereby improving the generalization ability of the gesture recognition model.

[0005] Subsequently, the present invention proposes a transfer learning method, which uses a small amount of data of the target user (i.e., the user currently using the system) to fine-tune the pre-trained user-independent model, thereby improving the accuracy of real-time gesture recognition of the target user.

[0006] Finally, the present invention designs a set of Android programs to deploy the model. The true wireless headset interaction method based on IMU facial gesture recognition can run in real time on current Android phones and only brings low system power consumption.

[0007] The present invention provides a true wireless headset interaction method based on IMU facial gesture recognition, and the present invention provides the following technical solutions:

[0008] A true wireless headset interaction method based on IMU facial gesture recognition includes the following steps:

[0009] Step 1: Train the domain discriminator. The convolutional neural network-based feature extractor and domain discriminator can distinguish different users based on their facial gestures.

[0010] Step 2: Perform offline model training to obtain a user-independent feature extractor and gesture classifier;

[0011] Step 3: After offline model training is completed, remove the user discriminator and deploy the feature extractor and gesture classifier to the personal device for model transfer learning and real-time gesture recognition.

[0012] Preferably, the step 1 is specifically:

[0013] Data was collected using UltraCore Electronics' HI226DK inertial measurement unit, which was fixed near the user's ear with a sampling rate of 200Hz. In each experiment, participants sat on a chair and performed seven different gestures on their face: click, swipe up, swipe down, swipe left, swipe right, clockwise circle, and counterclockwise circle. Three-axis accelerometer data and three-axis gyroscope data were recorded, along with user identity tags and gesture tags.

[0014] Preferably, the step 2 is specifically as follows:

[0015] Step 2.1: Training the domain discriminator, network input: The data matrix M input to the neural network consists of six independent time series of the three-axis accelerometer and the three-axis gyroscope, M = {M1, M2...M6}. Before inputting into the neural network, to remove the influence of gravity acceleration and accelerate network convergence, each axis of the data matrix M is normalized separately.

[0016] Feature Extractor: The feature extractor (C) consists of three layers of one-dimensional convolution. The output channels of the convolution layers are 32, 64, and 128, respectively, with a kernel size of 3 and a stride of 1. A batch normalization layer is added after each convolution layer to accelerate network convergence, and ReLU is used as the activation function. A maximum pooling layer is added after the convolution layer to reduce network complexity and alleviate overfitting. After the Flatten layer, the feature extractor outputs a 2048-dimensional feature vector F, which is expressed as follows:

[0017] F=C(M;θ C ) (1)

[0018] Where C represents the input mapping relationship of the feature extractor, θ C Represents the network parameters of the feature extractor;

[0019] The domain discriminator (D) is constructed by two fully connected layers with 512 nodes and K nodes respectively, where K represents the number of users participating in the training. The input of the domain discriminator is the feature vector F extracted by the feature extractor, and the output is expressed as follows:

[0020] D p =D(F;θ D ) (2)

[0021] Among them, D is the mapping relationship between the input and output of the domain discriminator, D p is the network’s prediction vector, θ D is the domain discriminator network parameter; the real user label D t is converted to a one-hot encoding and the cross entropy is used to estimate the loss between the predicted vector and the true user label:

[0022]

[0023] Among them, N is the number of samples in a batch training, and the training goal of the network is to minimize the domain loss The Adam optimizer is sampled to update parameters. After training, the convolutional neural network-based feature extractor and domain discriminator can distinguish different users based on their facial gestures.

[0024] Step 2.2: Train the gesture classifier. The gesture classifier consists of two fully connected layers with 512 nodes and I nodes, respectively, where I = 7, representing 7 gestures. When the network is trained, the parameters of the domain discriminator are fixed, and the optimizer only updates the parameters of the feature extractor and gesture classifier. The goal of network training is to maximize the accuracy of gesture classification while confusing the discrimination results of the domain discriminator. The output of the domain discriminator eventually tends to a uniform distribution U = {1 / M, 1 / M…, 1 / M}. The root mean square error is used to calculate the output D of the domain discriminator. p The loss value between the expected uniform distribution U is:

[0025]

[0026] The predicted value of the gesture classifier is expressed as follows:

[0027] G p =G(F;θ G ) (5)

[0028] Among them, G represents the input-output mapping relationship of the gesture classifier, θ G is the network parameter of the gesture classifier; the predicted value G of the gesture classifier p and real gesture tags G t Use cross-loss entropy to calculate:

[0029]

[0030] It is a combination of the above user discrimination loss and gesture classification loss function:

[0031]

[0032] The training goal of the network is to minimize We enhance gesture-related features while suppressing domain-specific features; ultimately we obtain a user-independent feature extractor and gesture classifier.

[0033] Preferably, the step 3 is specifically:

[0034] After the offline model training is completed, the user discriminator is removed and the feature extractor and gesture classifier are deployed to personal devices. When using a small number of target user samples for personalized learning, the network's underlying feature extractor parameters are fixed, while the fully connected layer is fine-tuned. The network optimization goal is to minimize the gesture classifier prediction value G p and real gesture tags G t loss

[0035]

[0036] Model deployment and fine-tuning are implemented using Pytorch Mobile and Deeplearning 4j, respectively.

[0037] Preferably, the deployment and fine-tuning of the model are implemented using Pytorch Mobile and Deeplearning 4j respectively. The specific implementation steps are as follows: the wireless headset establishes and transmits IMU data to the smartphone, and then uses a simple binary classification network to detect whether the target user performs a facial gesture. When the target user is performing a facial gesture, it is determined whether it is the first time the user performs a facial gesture or whether the system has been used for a period of time. If it has been used for a period of time, a small number of target user samples are used to fine-tune the model. Otherwise, the current IMU data is directly input into the gesture classifier for gesture judgment, and finally the corresponding operation is called on the smartphone.

[0038] Beneficial effects:

[0039] The current interaction methods of true wireless earphones have many inconveniences, such as few operating functions, easy squeezing of the ear canal, or susceptibility to environmental interference and high energy consumption. The present invention provides a true wireless earphone interaction method based on IMU facial gesture recognition to solve the above problems. In user-dependent tasks, the gesture recognition accuracy can reach more than 95%. Among them, user dependence means that the target user's data is included in the offline model training data. Afterwards, the present invention achieved an accuracy of 82.3% in user-independent tasks. User-independent tasks mean that the target user's data is not included in the offline model training data. In addition, after fine-tuning the model using a small amount of target user data, a gesture recognition accuracy of more than 95% can be achieved. Finally, the Pytorch Mobile model deployed on an existing Android phone only takes 1.3ms to infer the target user's gesture. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a typical application scenario of the present invention, where a user controls a music app on a smartphone by performing facial gestures;

[0041] Figure 2 This is the detailed process of implementing data collection, offline model training and model personalization, and online reasoning in the present invention;

[0042] Figure 3 This is a flow chart of the present invention's real-time interaction with true wireless earphones through IMU facial gesture recognition;

[0043] Figure 4 This is an example of data collected by the present invention for 7 different gestures of the same user (upper row) and the same gesture of 7 different users (lower row);

[0044] Figure 5 This is the first step of offline model training proposed in this invention, training the domain discriminator;

[0045] Figure 6 This is the second step of offline model training proposed by the present invention, training the gesture classifier;

[0046] Figure 7 This is a schematic diagram of the model performing transfer learning (fine-tuning);

[0047] Figure 8 This is a comparison of the gesture recognition accuracy of the domain adversarial training method proposed in this invention and other methods on user-dependent tasks;

[0048] Figure 9 This is a comparison of the gesture recognition accuracy of the domain adversarial training method proposed in this invention and other traditional machine learning and deep learning methods on user-independent tasks;

[0049] Figure 10 This is a comparison of the gesture recognition accuracy of the model after fine-tuning using the domain adversarial training method proposed in this invention and other methods. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to specific embodiments. Specific embodiment one:

[0052] according to Figures 1 to 10 As shown, the specific optimization technical solution adopted by the present invention to solve the above technical problems is: the present invention uses the IMU of the true wireless headset to perceive facial gestures near the human ear, Figure 1 A typical application scenario of the present invention is shown, where a user controls a music app on a smartphone by performing facial gestures. Figure 2 The detailed process of implementing data collection, offline model training, model personalization, and online inference in the present invention is described. The main steps of the invention include three: gesture data collection, offline model training, model transfer learning, and real-time gesture recognition.

[0053] Step 1: Train the domain discriminator. The convolutional neural network-based feature extractor and domain discriminator can distinguish different users based on their facial gestures.

[0054] This paper uses the HiPNUC HI226DK inertial measurement unit (IMU) to collect data. The HI226DK is fixed near the user's ear and the sampling rate is set to 200Hz. In each experiment, the participant sits in a chair and performs seven different facial gestures: tap, swipe up, swipe down, swipe left, swipe right, clockwise circle, and counterclockwise circle. The three-axis accelerometer and three-axis gyroscope data are then recorded, along with user identification and gesture labels. Figure 4 Examples of data collected are shown for seven different gestures from the same user (top row) and the same gesture from seven different users (bottom row).

[0055] Step 2: Perform offline model training to obtain a user-independent feature extractor and gesture classifier;

[0056] Offline model training consists of two steps: the first step is to train the domain discriminator, and the second step is to train the gesture classifier.

[0057] Step 2.1: The first step is to train the domain discriminator. The network structure is as follows Figure 5 As shown in Figure 1, the network input matrix M, which is composed of six independent time series from the three-axis accelerometer and three-axis gyroscope, is normalized along each axis of the matrix M before inputting it into the neural network to remove the influence of gravity and accelerate network convergence.

[0058] Feature Extractor: First, the feature extractor The network consists of three layers of one-dimensional convolutions, with output channels of 32, 64, and 128, respectively, a kernel size of 3, and a stride of 1. Batch Normalization (BN) layers are added after each convolutional layer to accelerate network convergence, using ReLU as the activation function. A Maxpooling layer is then added after the convolutional layers to reduce network complexity and mitigate overfitting. Finally, after the Flatten layer, the feature extractor outputs a 2048-dimensional feature vector F, which is expressed mathematically as follows:

[0059]

[0060] in It represents the input mapping relationship of the feature extractor. Represents the network parameters of the feature extractor.

[0061] Domain Discriminator: Domain Discriminator It consists of two fully connected layers with 512 nodes and K nodes (K represents the number of users participating in the training). The input of the domain discriminator is the feature vector F extracted by the feature extractor, and the output can be expressed as:

[0062]

[0063] in is the mapping relationship between the input and output of the domain discriminator, D p is the network’s prediction vector, is the domain discriminator network parameter. The real user label D t It is converted into a one-hot vector and the cross entropy is used to estimate the loss between the predicted vector and the true user label:

[0064]

[0065] Where N is the number of samples in a batch training (Batch size), and the training goal of the network is to minimize the domain loss The Adam optimizer is sampled to update the parameters. After training, the feature extractor and domain discriminator based on the convolutional neural network can distinguish different users based on their facial gestures.

[0066] Step 2.2: The second step is to train the gesture classifier. The network structure is as follows Figure 6 As shown, in Figure 5 Add gesture classifier based on The gesture classification is composed of two fully connected layers with 512 nodes and I (I = 7, representing 7 gestures) respectively. When the network is trained, the parameters of the domain discriminator (user discriminator) are fixed, and the optimizer only updates the parameters of the feature extractor and gesture classifier. The goal of network training is to maximize the accuracy of gesture classification while confusing the discrimination results of the domain discriminator. Therefore, the output of the domain discriminator should eventually tend to a uniform distribution U = {1 / M, 1 / M…, 1 / M}. The root mean square error (MSE) is used to calculate the output D of the domain discriminator. p The loss value between the expected uniform distribution U is:

[0067]

[0068] Meanwhile, the predicted value of the gesture classifier is:

[0069]

[0070] in, It represents the input-output mapping relationship of the gesture classifier. is the network parameter of the gesture classifier. The predicted value G of the gesture classifier p and real gesture tags G t Use cross-loss entropy to calculate:

[0071]

[0072] After that, It is a combination of the above user discrimination loss and gesture classification loss function:

[0073]

[0074] The training goal of the network is to minimize We enhance gesture-related features while suppressing domain (user)-specific features, ultimately obtaining a user-independent feature extractor and gesture classifier.

[0075] Step 3: After offline model training is completed, remove the user discriminator and deploy the feature extractor and gesture classifier to the personal device for model transfer learning and real-time gesture recognition.

[0076] Figure 7 The network structure of the model transfer learning (personalized learning) of the present invention is shown. After the offline model training is completed, the user discriminator is removed and the feature extractor and gesture classifier are deployed to the personal device. When using a small number of target user samples for personalized learning, the parameters of the underlying feature extractor of the network are fixed, while the fully connected layer is fine-tuned. The optimization goal of the network at this time is to minimize the gesture classifier prediction value G p and real gesture tags G t loss

[0077]

[0078] Figure 3 The flowchart of the present invention's real-time gesture recognition is presented. Model deployment and fine-tuning are implemented using PytorchMobile and Deeplearning 4j, respectively. The specific implementation steps are as follows: The wireless headset establishes a connection with the smartphone and transmits IMU data to the phone. A simple binary classification network is then used to detect whether the target user is performing a facial gesture. If the target user is performing a facial gesture, a determination is made as to whether this is the user's first time performing a facial gesture or whether the system has been used for a long time (due to changes in skin condition caused by personal habits, which can reduce gesture recognition accuracy). If so, a small number of target user samples are used to fine-tune the model. If not, the current IMU data is directly input into the gesture classifier for gesture judgment, and finally the corresponding operation is invoked on the smartphone.

[0079] Figure 8 This is a comparison of the gesture recognition accuracy of five different machine learning models in user-dependent tasks. The abbreviations and specific implementation methods of these five machine learning models are as follows:

[0080] DAL-CNN: Training offline CNN models using domain adversarial training.

[0081] CNN: Use CNN directly and input IMU data to obtain gesture classification inference results.

[0082] XGBoost, Randomforest, SVM: Extract time and frequency domain features from IMUs, and input the feature vectors into the machine learning classification model.

[0083] Experimental results: Given different numbers of samples of each person and each gesture, the DAL-CNN based on domain adversarial training proposed in this invention achieved the highest gesture recognition accuracy, reaching more than 95% when given 100 samples of each gesture per person.

[0084] Figure 9 This is a comparison of the gesture recognition accuracy of the domain adversarial training method DAL-CNN proposed in this invention and other traditional machine learning deep learning methods in user independent tasks.

[0085] Experimental results: DAL-CNN achieved the highest gesture recognition accuracy when various numbers of users participated in model training. When 19 users participated in training, the gesture recognition accuracy reached over 82%.

[0086] Figure 10 This figure compares the gesture recognition accuracy of several different model personalization methods using different numbers of target user samples. The abbreviations and specific implementation methods of these methods are as follows:

[0087] The model training process of DAL-CNN and CNN is the same as Figure 7 , that is, fix the pre-trained feature extractor and fine-tune the fully connected layer.

[0088] DAL-CNN-SVM and CNN-SVM use a pre-trained CNN-based feature extractor to replace the traditional feature extraction method, and then use a small number of target user samples to perform model personalized learning.

[0089] XGBoost directly uses a small amount of target user samples to train the gesture classification model.

[0090] Experimental results show that the proposed DAL-CNN based on domain adversarial training achieves the highest gesture recognition accuracy when given different numbers of samples of each gesture. When the model is personalized with 10 target user samples of each gesture, the gesture recognition accuracy exceeds 91%.

[0091] The above is only a preferred embodiment of a true wireless headset interaction method based on IMU facial gesture recognition. The scope of protection of a true wireless headset interaction method based on IMU facial gesture recognition is not limited to the above embodiment. All technical solutions under this idea belong to the scope of protection of the present invention. It should be pointed out that for those skilled in the art, several improvements and changes without departing from the principle of the present invention should also be regarded as the scope of protection of the present invention.

Claims

1. A true wireless headset interaction method based on IMU facial gesture recognition, characterized by: The following steps are involved: Step 1: Train the domain discriminator. The convolutional neural network-based feature extractor and domain discriminator can distinguish different users based on their facial gestures. Step 2: Perform offline model training to obtain a user-independent feature extractor and gesture classifier; The step 2 is specifically as follows: Step 2.1: Training the domain discriminator, network input: The data matrix M input to the neural network consists of six independent time series of the three-axis accelerometer and the three-axis gyroscope, M = {M1, M2...M6}. Before inputting into the neural network, to remove the influence of gravity acceleration and accelerate network convergence, each axis of the data matrix M is normalized separately. Feature Extractor: The feature extractor (C) consists of three layers of one-dimensional convolution. The output channels of the convolution layers are 32, 64, and 128, respectively, with a kernel size of 3 and a stride of 1. A batch normalization layer is added after each convolution layer to accelerate network convergence, and ReLU is used as the activation function. A maximum pooling layer is added after the convolution layer to reduce network complexity and alleviate overfitting. After the Flatten layer, the feature extractor outputs a 2048-dimensional feature vector F, which is expressed as follows: F=C(M;θ C ) (1) Where C represents the input mapping relationship of the feature extractor, θ C Represents the network parameters of the feature extractor; The domain discriminator (D) is constructed by two fully connected layers with 512 nodes and K nodes respectively, where K represents the number of users participating in the training. The input of the domain discriminator is the feature vector F extracted by the feature extractor, and the output is expressed as follows: D p =D(F;θ D ) (2) Among them, D is the mapping relationship between the input and output of the domain discriminator, D p is the network’s prediction vector, θ D is the domain discriminator network parameter; the real user label D t is converted to a one-hot encoding and the cross entropy is used to estimate the loss between the predicted vector and the true user label: Among them, N is the number of samples in a batch training, and the training goal of the network is to minimize the domain loss The Adam optimizer is sampled to update parameters. After training, the convolutional neural network-based feature extractor and domain discriminator can distinguish different users based on their facial gestures. Step 2.2: Train the gesture classifier. The gesture classifier consists of two fully connected layers with 512 nodes and I nodes, respectively, where I = 7, representing 7 gestures. When the network is trained, the parameters of the domain discriminator are fixed, and the optimizer only updates the parameters of the feature extractor and gesture classifier. The goal of network training is to maximize the accuracy of gesture classification while confusing the discrimination results of the domain discriminator. The output of the domain discriminator eventually tends to a uniform distribution U = {1 / M, 1 / M…, 1 / M}. The root mean square error is used to calculate the output D of the domain discriminator. p The loss value between the expected uniform distribution U is: The predicted value of the gesture classifier is expressed as follows: G p =G(F;θ G ) (5) Among them, G represents the input-output mapping relationship of the gesture classifier, θ G is the network parameter of the gesture classifier; the predicted value G of the gesture classifier p and real gesture tags G t Use cross-loss entropy to calculate: It is a combination of the above user discrimination loss and gesture classification loss function: The training goal of the network is to minimize Enhance gesture-related features while suppressing domain-specific features; ultimately obtaining a user-independent feature extractor and gesture classifier; Step 3: After offline model training is completed, remove the user discriminator and deploy the feature extractor and gesture classifier to the personal device for model transfer learning and real-time gesture recognition.

2. The true wireless headset interaction method based on IMU facial gesture recognition according to claim 1, characterized in that: The step 1 is specifically as follows: Data was collected using UltraCore Electronics' HI226DK inertial measurement unit, which was fixed near the user's ear with a sampling rate of 200Hz. In each experiment, participants sat on a chair and performed seven different gestures on their face: click, swipe up, swipe down, swipe left, swipe right, clockwise circle, and counterclockwise circle. Three-axis accelerometer data and three-axis gyroscope data were recorded, along with user identity tags and gesture tags.

3. The true wireless headset interaction method based on IMU facial gesture recognition according to claim 1, characterized in that: The step 3 is specifically as follows: After the offline model training is completed, the user discriminator is removed and the feature extractor and gesture classifier are deployed to personal devices. When using a small number of target user samples for personalized learning, the network's underlying feature extractor parameters are fixed, while the fully connected layer is fine-tuned. The network optimization goal is to minimize the gesture classifier prediction value G p and real gesture tags G t loss Model deployment and fine-tuning are implemented using Pytorch Mobile and Deeplearning 4j, respectively.

4. The true wireless headset interaction method based on IMU facial gesture recognition according to claim 3 is characterized by: The model was deployed and fine-tuned using Pytorch Mobile and Deeplearning 4j, respectively. The specific implementation steps are as follows: the wireless headset establishes a connection with the smartphone and transmits IMU data to the phone. A simple binary classification network is then used to detect whether the target user is performing a facial gesture. If the target user is performing a facial gesture, the network determines whether it is the user's first time performing a facial gesture or whether the user has been using the system for a while. If the user has been using the system for a while, the model is fine-tuned using a small number of target user samples. Otherwise, the current IMU data is directly input into the gesture classifier for gesture judgment. Finally, the corresponding operation is called on the smartphone.

Images