A human-computer interaction method and system based on a nine-axis IMU and edge AI

Abstract

The application belongs to the field of gesture recognition and intelligent remote control, and discloses a human-computer interaction method and system based on a nine-axis IMU and edge AI, which comprises the following steps: acquiring original IMU data generated by the nine-axis IMU based on user actions; performing attitude estimation and attitude solution on the original IMU data based on an extended Kalman filtering algorithm to obtain attitude data corresponding to the original IMU data; pre-processing the attitude data to obtain an attitude data matrix; inputting the attitude data matrix into a pre-deployed edge AI model to output a gesture recognition result; generating an operation instruction corresponding to the gesture recognition result and sending the operation instruction to a device to be controlled; by using the nine-axis IMU, the problem of attitude drift caused by the lack of a magnetometer in a six-axis IMU is solved; the pre-deployed edge AI model can automatically extract deep temporal features of gestures and complete high-precision judgment, thereby realizing accurate gesture recognition.

Description

Technical Field

[0001] This invention relates to the field of gesture recognition and intelligent remote control, and in particular to a human-computer interaction method and system based on a nine-axis IMU and edge AI. Background Technology

[0002] In the fields of smart home, Internet of Things and consumer electronics, the types of intelligent devices have surged. New types of intelligent interactive devices, represented by rings, have emerged in the market. These devices are mainly controlled remotely through physical buttons and cloud-based AI-based voice and gesture recognition.

[0003] However, the existing smart rings still have the following shortcomings: (1) Insufficient interactive intelligence and high latency. The physical buttons in the existing technology lack intelligence and rely on voice or gesture recognition from the source. The command needs to go through a long link of data collection, network upload, cloud analysis and command issuance, which causes significant control delay and is greatly affected by the network; (2) Insufficient perception dimension leads to limited recognition accuracy. The existing technology usually uses a six-axis IMU (Inertial Measurement Unit). This component only contains an accelerometer and a gyroscope. When the device performs attitude calculation, it cannot obtain the absolute geomagnetic direction as a reference, which easily produces cumulative gyroscope drift. As a result, it cannot accurately perceive and locate the user's absolute direction and complex gestures in three-dimensional space. The recognition rate of commands containing spatial orientation information such as rotation and pointing is low and the misjudgment rate is high; (3) Fixed function and poor scene universality. The smart ring products on the market are usually supported by the manufacturer's preset gesture commands and cannot be changed. They cannot be customized according to the user's personal habits. Moreover, the product function is limited to health monitoring or sports assistance and cannot adapt to diverse application scenarios.

[0004] Existing technologies still suffer from insufficient interactivity, perception dimensions, and versatility in functional scenarios for smart rings. Therefore, existing technologies need further improvement. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a human-computer interaction method based on a nine-axis IMU and edge AI to address the shortcomings of existing technologies in terms of interactivity, perception dimension and functional versatility of smart rings.

[0006] The technical solution adopted by this invention to solve the technical problem is as follows: In a first aspect, the present invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, including: Acquire the raw IMU data generated by the nine-axis IMU based on user actions; The attitude estimation and attitude calculation of the original IMU data are performed based on the extended Kalman filter algorithm to obtain the attitude data corresponding to the original IMU data. The attitude data is preprocessed to obtain an attitude data matrix; The gesture data matrix is ​​input into a pre-deployed edge AI model, and the gesture recognition result is output. An operation command corresponding to the gesture recognition result is generated and sent to the device to be controlled.

[0007] In one implementation, the raw IMU data includes: triaxial acceleration, triaxial angular velocity, and triaxial geomagnetic intensity.

[0008] In one implementation, the step of performing attitude estimation and attitude calculation on the original IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the original IMU data includes: Define a unit quaternion for the attitude, update the unit quaternion based on the triaxial angular velocity, and determine the state vector based on the updated unit quaternion; The first observation residual is constructed based on the difference between the triaxial acceleration and the direction of gravity. The second observation residual is constructed based on the difference between the triaxial geomagnetic intensity and the theoretical direction of geomagnetic north. Calculate the Kalman gain based on the first observation residual and the second observation residual, update the state vector based on the Kalman gain, and output the attitude estimation result; The attitude estimation results are used to calculate the attitude and output the attitude data.

[0009] In one implementation, the preprocessing of the attitude data to obtain an attitude data matrix includes: Based on a preset window length, sliding step size, and sampling rate, a sliding window truncation operation is performed on the attitude data to generate continuous window data; wherein each window data is a two-dimensional data matrix. The pose data matrix is ​​obtained by normalizing all window data using the Z-Score normalization method.

[0010] In one implementation, the pre-deployed edge AI model includes a convolutional neural network model and a classification decision model. The step of inputting the pose data matrix into the pre-deployed edge AI model and outputting the gesture recognition result includes: The pose data matrix is ​​input into the convolutional neural network model to obtain the feature vector; The feature vector is input into the classification decision model to obtain the gesture recognition result; Output the gesture recognition result.

[0011] In one implementation, the convolutional neural network model includes convolutional layers, ReLU activation layers, and pooling layers. The step of inputting the pose data matrix into the convolutional neural network model to obtain a feature vector includes: The temporal dynamic features of the pose data matrix are extracted through the convolutional layer to generate a temporal feature map; The temporal feature map is nonlinearly mapped using the ReLU activation layer to obtain higher-order semantic features; The pooling layer removes redundant information from the higher-order semantic features and outputs a feature vector.

[0012] In one implementation, inputting the feature vector into the classification decision model to obtain the gesture recognition result includes: The radial basis function kernel method maps the input feature vector to a preset high-dimensional space; wherein, the preset high-dimensional space is a gesture category sample space with clear boundaries; Calculate the kernel distance weighted sum of the feature vectors and support vectors; Based on the distance-weighted sum of the kernel function, the gesture recognition result corresponding to the feature vector is output.

[0013] Secondly, the present invention provides a human-computer interaction system based on a nine-axis IMU and edge AI, comprising: A nine-axis IMU sensor is used to acquire raw IMU data based on user actions; The main control module includes a filtering submodule, a preprocessing submodule, an edge AI submodule, and a communication submodule. The filtering submodule is used to perform attitude estimation and attitude calculation on the original IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the original IMU data. The preprocessing submodule is used to preprocess the attitude data to obtain an attitude data matrix; The edge AI submodule is used to input the pose data matrix into a pre-deployed edge AI model and output gesture recognition results; The communication submodule is used to generate operation instructions corresponding to the gesture recognition results and send the operation instructions to the device to be controlled.

[0014] Thirdly, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a human-computer interaction program based on a nine-axis IMU and edge AI, and the human-computer interaction program based on a nine-axis IMU and edge AI, when executed by the processor, is used to implement the operation of the human-computer interaction method based on a nine-axis IMU and edge AI as described in the first aspect.

[0015] Fourthly, the present invention also provides a computer-readable storage medium storing a human-computer interaction program based on a nine-axis IMU and edge AI, which, when executed by a processor, is used to implement the operation of the human-computer interaction method based on a nine-axis IMU and edge AI as described in the first aspect.

[0016] The present invention, by employing the above technical solution, has the following effects: This invention provides a human-computer interaction method and system based on a nine-axis IMU and edge AI, comprising: acquiring raw IMU data generated by the nine-axis IMU based on user actions; performing attitude estimation and attitude calculation on the raw IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the raw IMU data; preprocessing the attitude data to obtain an attitude data matrix; inputting the attitude data matrix into a pre-deployed edge AI model to output gesture recognition results; generating operation instructions corresponding to the gesture recognition results and sending the operation instructions to the device to be controlled; this invention solves the attitude drift problem caused by the lack of a magnetometer in a six-axis IMU by using a nine-axis IMU; the pre-deployed edge AI model can automatically extract deep temporal features of gestures and complete high-precision decisions, achieving accurate gesture recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the human-computer interaction method based on nine-axis IMU and edge AI in this invention.

[0019] Figure 2 This is a schematic diagram of the structure of a human-computer interaction system based on a nine-axis IMU and edge AI in one implementation of the present invention.

[0020] Figure 3 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0021] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] Exemplary methods In the fields of smart home, Internet of Things and consumer electronics, the types of intelligent devices have surged. New types of intelligent interactive devices, represented by rings, have emerged in the market. These devices are mainly controlled remotely through physical buttons and cloud-based AI-based voice and gesture recognition.

[0024] However, the existing smart rings still have the following shortcomings: (1) Insufficient interactive intelligence and high latency. The physical buttons in the existing technology lack intelligence and rely on voice or gesture recognition from the source. The command needs to go through a long link of data collection, network upload, cloud analysis and command issuance, which causes significant control delay and is greatly affected by the network; (2) Insufficient perception dimension leads to limited recognition accuracy. The existing technology usually uses a six-axis IMU (Inertial Measurement Unit). This component only contains an accelerometer and a gyroscope. When the device performs attitude calculation, it cannot obtain the absolute geomagnetic direction as a reference, which easily produces cumulative gyroscope drift. As a result, it cannot accurately perceive and locate the user's absolute direction and complex gestures in three-dimensional space. The recognition rate of commands containing spatial orientation information such as rotation and pointing is low and the misjudgment rate is high; (3) Fixed function and poor scene universality. The smart ring products on the market are usually supported by the manufacturer's preset gesture commands and cannot be changed. They cannot be customized according to the user's personal habits. Moreover, the product function is limited to health monitoring or sports assistance and cannot adapt to diverse application scenarios.

[0025] Existing technologies still suffer from insufficient interactivity, perception dimensions, and versatility in functional scenarios for smart rings. Therefore, existing technologies need further improvement.

[0026] To address the above technical problems, this invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, comprising: acquiring raw IMU data generated by the nine-axis IMU based on user actions; performing attitude estimation and attitude calculation on the raw IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the raw IMU data; preprocessing the attitude data to obtain an attitude data matrix; inputting the attitude data matrix into a pre-deployed edge AI model to output gesture recognition results; generating operation instructions corresponding to the gesture recognition results and sending the operation instructions to the device to be controlled; this invention solves the attitude drift problem caused by the lack of a magnetometer in a six-axis IMU by using a nine-axis IMU; the pre-deployed edge AI model can automatically extract deep temporal features of gestures and complete high-precision decisions, achieving accurate gesture recognition.

[0027] like Figure 1 As shown, this embodiment of the invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, including the following steps: Step S100: Obtain the raw IMU data generated by the nine-axis IMU based on user actions.

[0028] In this embodiment, a high-precision nine-axis IMU sensor is used to fuse and collect data from the accelerometer, gyroscope, and magnetometer. The magnetometer provides an absolute geomagnetic direction reference, effectively suppressing the cumulative drift of the gyroscope. Combined with full-dimensional data acquisition, it enables accurate capture of complex spatial gestures, improving the dimensionality and accuracy of gesture recognition from the source of perception, and breaking the limitation of traditional devices that can only adapt to simple planar movements.

[0029] In this embodiment, raw IMU data generated by a nine-axis IMU based on user actions is acquired. The raw IMU data includes three-axis acceleration, three-axis angular velocity, and three-axis geomagnetic intensity, meaning the raw IMU data contains three types of raw physical quantities with a total of nine degrees of freedom. Specifically, the three-axis acceleration is three-axis linear acceleration, measured in g, along the X, Y, and Z axes respectively; the three-axis angular velocity is measured in ° / s, measured around the X, Y, and Z axes respectively; and the three-axis geomagnetic intensity is measured in μT, along the north, east, and vertical directions of the geographic coordinate system respectively. The data from these nine channels together constitute a complete description of the hand's translation, rotation, and absolute orientation in three-dimensional space.

[0030] In this embodiment, the nine-axis IMU sensor used is ICM-20948. Before acquiring the raw IMU data generated by the nine-axis IMU sensor based on user actions, the method further includes: configuring the sampling frequency and range of the nine-axis IMU, wherein the sampling frequency can be configured to 100Hz.

[0031] In this embodiment, triaxial acceleration is read synchronously. Triaxial angular velocity and triaxial geomagnetic intensity In this system, the X-axis of the three-axis geomagnetic intensity represents the north of the geographic coordinate system, the Y-axis represents the east of the geographic coordinate system, and the Z-axis represents the vertical direction of the geographic coordinate system.

[0032] It should be noted that the original triaxial geomagnetic intensity output by the magnetometer is the geomagnetic intensity in the sensor coordinate system, which needs to be converted to the geographic coordinate system after attitude transformation.

[0033] like Figure 1 As shown, this embodiment of the invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, including the following steps: Step S200: Based on the extended Kalman filter algorithm, attitude estimation and attitude calculation are performed on the original IMU data to obtain attitude data corresponding to the original IMU data.

[0034] In this embodiment, after acquiring the original IMU data, the original IMU data is fused from multiple sources using the Extended Kalman Filter (EKF) algorithm to achieve attitude estimation and attitude calculation, so as to obtain the attitude data corresponding to the original IMU data.

[0035] Specifically, in one implementation of this embodiment, step S200 includes the following steps: Step S201: Define the unit quaternion of the attitude, update the unit quaternion based on the three-axis angular velocity, and determine the state vector based on the updated unit quaternion.

[0036] In this embodiment, a unit quaternion for attitude is defined, the unit quaternion is updated based on the three-axis angular velocities, and the state vector is determined based on the updated unit quaternion. That is, prior attitude estimation is achieved by integrating the three-axis angular velocities, as detailed below: The initial unit quaternion for attitude is defined as: Used to indicate the rotation of a smart bracelet in a geographic coordinate system; Obtaining the zero bias of a three-axis gyroscope That is, the non-zero deviation value of the output of each axis of the three-axis gyroscope in the state of no angular velocity input (stationary); Based on the initial unit quaternion and the zero bias of the three-axis gyroscope The state vector is defined as follows: ; After defining the state vector, attitude recursion can be achieved by integrating the unit quaternion.

[0037] Furthermore, through the obtained triaxial angular velocity measurements To update a unit quaternion, the following steps are required: The three-axis angular velocities obtained from the three-axis gyroscope are combined into a single angular velocity vector. The time interval is Calculate the rotation vector corresponding to the triaxial angular velocity measurements. : ; Rotate vector Transform to a geographic coordinate system, corresponding axis of rotation for: ; Based on rotation axis Determine the rotation angle According to the rotation angle Update the unit quaternion as follows: ; ; ; ; Based on the updated unit quaternion Determine the state vector However, due to the zero-bias instability and measurement noise of the gyroscope, the error accumulates over time after integrating the angular velocity, causing the attitude to gradually deviate from the true value, i.e., there is integral drift.

[0038] Step S202: Construct the first observation residual based on the difference between the triaxial acceleration and the direction of gravity.

[0039] In this embodiment, the direction of gravity is defined as follows: The obtained triaxial acceleration measurement values ​​are The acquired triaxial acceleration measurements are normalized. Based on the difference between the normalized triaxial acceleration measurements and the direction of gravity, the first observation residual is constructed to correct the pitch and roll angles, thereby achieving the calibration of the accelerometer in the nine-axis IMU sensor.

[0040] Attitude correction is performed using accelerometer measurements. First, the triaxial acceleration measurements are... Normalization, the corresponding formula is as follows: ; Using the current predicted attitude quaternion Gravity vector in geographic coordinate system Transform to the carrier coordinate system to obtain the predicted gravity vector. The corresponding formula is as follows: ; Further construct the accelerometer observation residuals That is, the first observation residual: ; The first observation residual reflects the error of the current attitude in the pitch and roll directions, and is used to correct the state vector in the EKF update step, thereby suppressing the drift caused by gyroscope integration.

[0041] Step S203: Construct a second observation residual based on the theoretical difference between the triaxial geomagnetic intensity and the geomagnetic northward direction.

[0042] In this embodiment, the obtained triaxial geomagnetic intensity measurement value is The acquired triaxial geomagnetic intensity measurements are normalized. Based on the difference between the normalized triaxial geomagnetic intensity measurements and the theoretical direction of the local geomagnetic northward, a second observation residual is constructed to correct the heading angle. This suppresses yaw drift and enables the correction of the magnetometer in the nine-axis IMU sensor.

[0043] Similarly, the original measurements of the triaxial magnetometer Normalization yields: ; Using the current predicted attitude quaternion The normalized magnetometer vector is transformed from the sensor coordinate system to the geographic coordinate system to obtain the magnetic field vector in the geographic coordinate system: ; in, This represents the geomagnetic field vector estimated based on the current attitude.

[0044] Then, construct the magnetometer observation residuals. That is, the second observation residual: ; Theoretical geomagnetic reference vector It can be pre-calibrated or set according to the local geomagnetic inclination, and is usually expressed as .

[0045] The second observation residual reflects the error of the current attitude in the heading direction (Yaw) and is used to correct the state vector in the EKF update step, thereby suppressing yaw drift caused by gyroscope integration.

[0046] Step S204: Calculate the Kalman gain based on the first observation residual and the second observation residual, update the state vector based on the Kalman gain, and output the attitude estimation result.

[0047] In this embodiment, the Kalman gain is calculated based on the first observation residual and the second observation residual, and the state vector is updated based on the Kalman gain, specifically as follows: In the EKF step, the quaternion is updated discretely in time using gyroscope measurements. The current angular velocity vector is then... The gyroscope's zero bias estimate from the previous moment The difference We obtain the rotation vector, and update the quaternion through quaternion exponent mapping, with the corresponding formula as follows: ; in, To represent discrete-time indexes, This represents the sampling time interval.

[0048] Further, based on the first observation residual calculated in steps S202 and S203... Second observation residual Calculate Kalman gain And update the state vector and covariance matrix, including: ; ; ; in, Let be the state covariance matrix. To observe the noise covariance matrix, For the observation matrix, It is the identity matrix. For the observation vector, The observation equation is defined as follows. Through the above update steps, the predicted attitude is corrected using the observation residuals of the accelerometer and magnetometer, thus achieving the optimal estimate of EKF.

[0049] Step S205: Perform attitude calculation on the attitude estimation results and output attitude data.

[0050] In this embodiment, the final output attitude data can be in the form of a unit quaternion. Alternatively, a unit quaternion can be converted to Euler angles using the following formula: ; ; ; in, For roll angle, The pitch angle, This is the heading angle.

[0051] In this implementation example, due to the introduction of external references from the accelerometer and magnetometer, the integral drift of the gyroscope is effectively suppressed in all three axes, especially the heading angle, which exhibits long-term stability. The final output attitude data has high dynamic response and low noise characteristics, making it suitable as input for subsequent edge AI models for gesture recognition.

[0052] like Figure 1 As shown, this embodiment of the invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, including the following steps: Step S300: Preprocess the attitude data to obtain an attitude data matrix.

[0053] Specifically, in one implementation of this embodiment, step S300 includes the following steps: Step S301: Based on the preset window length, sliding step size and sampling rate, perform a sliding window truncation operation on the attitude data to generate continuous window data; wherein, each window data is a two-dimensional data matrix.

[0054] In this embodiment, the number of poses to be processed can be in the form of a unit quaternion output in step S200. It can also be Euler angles, or the raw IMU data after Kalman filtering and noise reduction can be used as input.

[0055] In this embodiment, the preset window length is set to 200ms, corresponding to 20 data points; the sliding step size is set to 10, i.e., 50% overlap rate; the sampling rate is set to 100Hz; the above settings can cover the complete gesture action cycle and ensure the action continuity of adjacent frames, preventing key features from being truncated.

[0056] The pose data is subjected to sliding window extraction using a preset window, with each window generating a separate set of data, resulting in continuous window data. Each window generates a single set of data. The matrix is ​​a two-dimensional data matrix, where the row indices correspond to sensor channels and the column indices correspond to 20 consecutive sampling times within the window. Each element in the matrix represents the value of the corresponding channel at the corresponding time. For example, the value of the acceleration X-axis at the 5th sampling point is represented by the element in the 1st row and 5th column.

[0057] Step S302: Normalize all window data using the Z-Score normalization method to obtain the pose data matrix.

[0058] Specifically, the Z-Score standardization method is adopted, and the mean and variance of the pre-stored training set are used to perform a linear transformation on the current window data to eliminate the inconsistency of dimensions caused by the tightness of the user's clothing and the difference in the gesture amplitude, so as to ensure the stability of the input distribution and finally obtain the posture data matrix; among which, the main content of the pre-stored training set is gesture sample data.

[0059] In this embodiment, the mean and variance of the pre-stored training set are used to perform a linear transformation to obtain a standardized 9×20 matrix, which is the pose data matrix for the final input edge AI model.

[0060] like Figure 1 As shown, this embodiment of the invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, including the following steps: Step S400: Input the gesture data matrix into a pre-deployed edge AI model and output the gesture recognition result.

[0061] In this embodiment, the pre-deployed edge AI model includes a convolutional neural network model and a classification decision model, namely lightweight CNN and SVM. Since the smart ring is small in size, the algorithm architecture corresponding to the embedded edge AI model can run efficiently and with low power consumption on the processing chip of the smart ring.

[0062] Specifically, in one implementation of this embodiment, step S400 includes the following steps: Step S401: Input the pose data matrix into the convolutional neural network model to obtain the feature vector.

[0063] In this embodiment, the convolutional neural network model is a lightweight 1D-CNN network model, which consists of three cascaded one-dimensional convolutional modules. Each one-dimensional convolutional neural network model includes a convolutional layer, a ReLU activation layer, and a pooling layer.

[0064] The pose data matrix is ​​input into the convolutional neural network model to obtain feature vectors, including the following steps: Step S401a: Extract the temporal dynamic features of the pose data matrix through the convolutional layer to generate a temporal feature map; Step S401b: The temporal feature map is nonlinearly mapped through the ReLU activation layer to obtain higher-order semantic features; Step S401c: The pooling layer removes redundant information from the higher-order semantic features and outputs a feature vector.

[0065] In this embodiment, a convolutional neural network model is used to achieve temporal dynamic capture of pose data, multi-channel coupling analysis, and feature compression and abstraction, as detailed below: Temporal dynamic capture directly responds to temporal waveform features such as sudden acceleration changes and peak angular velocity by sliding convolutional kernels on the time axis. For example, for the "swinging" action, the convolutional neural network model can keenly capture the specific waveform envelope of the angular velocity rising rapidly from zero and then falling back, and is not affected by the specific time point of the action (translation invariance). Multi-channel coupling analysis, with convolutional kernel depth covering all sensor channels, allows the convolutional neural network model to automatically learn the correlation weights between data of each axis during the operation. For example, in the "rotate" gesture, the network will focus on extracting the synchronization characteristics of the changes in the Z-axis of the gyroscope and the X / Y-axis of the magnetometer, thereby distinguishing between simple planar movement and spatial rotation. Feature compression and abstraction: through multiple layers of convolution and nonlinear mapping, the original time-series waveform is gradually abstracted into high-order semantic features; the subsequent pooling layer removes redundant information and retains only the most significant response values.

[0066] In this embodiment, the final output of the convolutional neural network model is a one-dimensional high-dimensional feature vector with a length of 64 or 128. This feature vector condenses the core discrimination information of the current gesture. Among them, the 64-dimensional one-dimensional feature vector consumes less resources and has faster inference, making it suitable for simple gestures; the 128-dimensional one-dimensional feature vector has stronger expressive power and can improve the discrimination of similar gestures.

[0067] Step S402: Input the feature vector into the classification decision model to obtain the gesture recognition result.

[0068] In this embodiment, the classification decision model is specifically a pre-trained SVM classifier, which is used to determine the gesture category corresponding to the pose data, i.e., the gesture recognition result.

[0069] In this embodiment, the SVM model is pre-trained offline on a PC, and the optimal support vectors and hyperplane parameters are solved using the sequential minimal optimization (SMO) algorithm. These parameters (support vector matrix, Lagrange multipliers, bias terms) are fixed-point quantized and stored as a constant array in the Flash area of ​​the smart ring MCU. During actual inference, only algebraic operations are performed, without iterative training.

[0070] Support Vector Matrix: Stores the support vector samples selected after training. Each support vector is a 64- or 128-dimensional vector extracted by the CNN.

[0071] Lagrange multipliers: the weight coefficients corresponding to each support vector. This indicates the degree of contribution of the support vector to the classification decision. Non-zero values ​​indicate the contribution of the support vector to the classification decision. The corresponding samples are support vectors.

[0072] Bias term: The constant term in the classification hyperplane This is used to adjust the position of the decision boundary, and the final classification decision function is... ,in For kernel function, The category labels for the support vectors.

[0073] In this embodiment, the feature vector is input into the classification decision model to obtain the gesture recognition result, including the following steps: Step S402a: Map the input feature vector to a preset high-dimensional space based on the radial basis function kernel method; wherein, the preset high-dimensional space is a gesture category sample space with clear boundaries.

[0074] In this embodiment, the preset high-dimensional space is a gesture category sample space with obvious boundaries. The input feature vector is mapped to the preset high-dimensional space based on the radial basis function kernel method. In this space, the sample distribution of different gesture categories has obvious interval boundaries.

[0075] Based on the Radial Basis Function (RBF) method, the input feature vector is mapped to a predefined high-dimensional space, as follows: ; in, The current input feature vector, For the first Support vectors, For kernel parameters.

[0076] Step S402b: Calculate the kernel function distance weighted sum of the feature vector and support vector.

[0077] ; in, The total number of support vectors, For the corresponding Lagrange multipliers, For the class labels of support vectors, This is a bias term.

[0078] Step S402c: Based on the distance weighted sum of the kernel function, output the gesture recognition result corresponding to the feature vector.

[0079] ; Here, sign() is the sign function, which outputs +1 or -1. 1 represents the binary classification result. For multi-class gestures (such as clenching a fist, swiping, pointing, etc.), a one-to-many or many-to-many strategy is used to combine multiple binary classification SVMs to achieve the final class decision.

[0080] In this embodiment, the determination result of the gesture corresponding to the feature vector is output by using a sign function to calculate the weighted sum of the distance between the kernel function of the feature vector and the support vector.

[0081] A one-to-many strategy is used to combine multiple binary SVMs to achieve the final classification. Each gesture category... A separate SVM classifier is used, and during training, this class of samples is treated as the positive class. ), and all other class samples are treated as negative classes ( Output decision value: .

[0082] During inference, calculate the decision values ​​of all KSVMs, and select the category with the largest decision value as the recognition result: .

[0083] Step S403: Output the gesture recognition result.

[0084] In this embodiment, compared to the traditional method of directly connecting a Softmax layer to the end of the CNN, this embodiment introduces an SVM as the classification head, which can effectively suppress overfitting in small sample scenarios, significantly improve the discrimination of similar gestures (such as "index finger" and "middle finger"), and reduce the false alarm rate.

[0085] In one implementation of this embodiment, the lightweight CNN / SVM dual model used in the edge AI model can also be replaced with other neural network models suitable for time series data processing.

[0086] like Figure 1 As shown, this embodiment of the invention provides a human-computer interaction method based on a nine-axis IMU and edge AI, including the following steps: Step S500: Generate an operation command corresponding to the gesture recognition result and send the operation command to the device to be controlled.

[0087] In this embodiment, the output gesture recognition result is used to generate operation instructions to control the device to be controlled. The gesture recognition result is based on the gesture dataset used in the pre-training process of the preset SVM classifier, i.e., the classification decision model. The gesture dataset contains preset gesture types.

[0088] In one implementation of this embodiment, the user can also add custom gesture types to the gesture dataset. For example, the user can update the preset gesture dataset via a mobile phone or computer, obtain the updated SVM classifier based on the updated gesture dataset, and write it into the main control chip of the smart bracelet, or update the SVM classifier via the transmission module to realize custom gestures.

[0089] In one implementation of this embodiment, an alternative solution combining a six-axis IMU with a miniature optical sensor or barometer can be used to compensate for the attitude drift caused by the absence of a magnetometer through multimodal data input, so as to achieve similar recognition accuracy.

[0090] This embodiment achieves the following technical effects through the above technical solution: 1. A high-precision nine-axis IMU sensor (including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer) is used to collect full-dimensional data on the user's hand movements. Through sensor fusion algorithms such as Extended Kalman Filter (EKF), drift-free, high-precision three-dimensional attitude information, including the absolute yaw angle, is calculated. This step fundamentally solves the attitude drift problem caused by the lack of a magnetometer in six-axis IMUs, providing a reliable perceptual foundation for recognizing complex spatial gestures.

[0091] 2. Deploy an edge AI model on a low-power MCU (such as STM32L4) local to the ring, using a lightweight CNN / SVM dual-model architecture. First, Kalman filtering is used to preprocess the raw or fused IMU data for noise reduction. Then, a CNN model automatically extracts deep temporal features of the gestures. Finally, an SVM classifier performs high-precision judgment on the features to complete real-time gesture recognition. By placing the core AI computation on the edge, the high latency and network dependency issues of cloud solutions are completely avoided, achieving millisecond-level local response.

[0092] 3. Build an open, user-customizable ecosystem. Through the accompanying mobile app, users can record, train, and bind any new gesture. The ring device incorporates the new commands into the recognition system through an incremental learning mechanism (such as updating SVM support vectors). This step breaks down the functional barriers of traditional products, giving the device strong scene adaptability and personalization potential.

[0093] Exemplary device Based on the above embodiments, the present invention also provides a human-computer interaction system based on a nine-axis IMU and edge AI, such as... Figure 2 As shown, the human-computer interaction system based on a nine-axis IMU and edge AI includes: A nine-axis IMU sensor is used to acquire raw IMU data based on user actions; The main control module includes a filtering submodule, a preprocessing submodule, an edge AI submodule, and a communication submodule. The filtering submodule is used to perform attitude estimation and attitude calculation on the original IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the original IMU data. The preprocessing submodule is used to preprocess the attitude data to obtain an attitude data matrix; The edge AI submodule is used to input the pose data matrix into a pre-deployed edge AI model and output gesture recognition results; The communication submodule is used to generate operation instructions corresponding to the gesture recognition results and send the operation instructions to the device to be controlled.

[0094] In this embodiment, the nine-axis IMU sensor includes a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, and its corresponding hardware model is ICM-20948.

[0095] In this embodiment, the main control module uses an STM32L4 low-power MCU, and the communication submodule uses a Bluetooth 5.2 communication module.

[0096] In this embodiment, the human-computer interaction system based on a nine-axis IMU and edge AI also includes: External Flash memory is used to store a constant array of the classification decision model; The power management unit (PMU) includes a wireless charging coil and an 80mAh LI-Po battery.

[0097] This embodiment uses a human-computer interaction system based on a nine-axis IMU and edge AI, integrating an ultra-low-power MCU and a Bluetooth 5.2 communication module, and implementing a refined power management strategy. The MCU is configured to enter nA-level deep sleep mode when idle, and is woken up by an IMU interrupt; the Bluetooth module dynamically adjusts its transmission power based on the communication distance. Working together, these two components minimize overall power consumption while ensuring connection reliability, achieving a relatively long battery life.

[0098] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 3 As shown.

[0099] The terminal includes: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor of the terminal provides computing and control capabilities; the memory of the terminal includes a computer-readable storage medium and internal memory; the computer-readable storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the computer-readable storage medium; the interface is used to connect to external devices; the display screen is used to display relevant information; and the communication module is used to communicate with a cloud server or other devices.

[0100] When executed by the processor, this computer program is used to implement a human-computer interaction method based on a nine-axis IMU and edge AI.

[0101] It will be understood by those skilled in the art that Figure 3 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0102] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a human-computer interaction program based on a nine-axis IMU and edge AI, which, when executed by the processor, is used to implement the operation of the human-computer interaction method based on the nine-axis IMU and edge AI as described above.

[0103] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a human-computer interaction program based on a nine-axis IMU and edge AI, which, when executed by a processor, is used to implement the operation of the human-computer interaction method based on a nine-axis IMU and edge AI as described above.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, database, or other media used in the embodiments provided by this invention can include both non-volatile and volatile memory.

[0105] In summary, this invention provides a human-computer interaction method and system based on a nine-axis IMU and edge AI, comprising: acquiring raw IMU data generated by the nine-axis IMU based on user actions; performing attitude estimation and attitude calculation on the raw IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the raw IMU data; preprocessing the attitude data to obtain an attitude data matrix; inputting the attitude data matrix into a pre-deployed edge AI model to output gesture recognition results; generating operation instructions corresponding to the gesture recognition results and sending the operation instructions to the device to be controlled; by using a nine-axis IMU, the attitude drift problem caused by the lack of a magnetometer in a six-axis IMU is solved; the pre-deployed edge AI model can automatically extract deep temporal features of gestures and complete high-precision decision-making, achieving accurate gesture recognition.

[0106] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A human-computer interaction method based on a nine-axis IMU and edge AI, characterized in that, include: Acquire the raw IMU data generated by the nine-axis IMU based on user actions; The attitude estimation and attitude calculation of the original IMU data are performed based on the extended Kalman filter algorithm to obtain the attitude data corresponding to the original IMU data. The attitude data is preprocessed to obtain an attitude data matrix; The gesture data matrix is ​​input into a pre-deployed edge AI model, and the gesture recognition result is output. An operation command corresponding to the gesture recognition result is generated and sent to the device to be controlled.

2. The human-computer interaction method based on a nine-axis IMU and edge AI according to claim 1, characterized in that, The raw IMU data includes: triaxial acceleration, triaxial angular velocity, and triaxial geomagnetic intensity.

3. The human-computer interaction method based on a nine-axis IMU and edge AI according to claim 2, characterized in that, The attitude estimation and attitude calculation based on the extended Kalman filter algorithm are performed on the original IMU data to obtain attitude data corresponding to the original IMU data, including: Define a unit quaternion for the attitude, update the unit quaternion based on the triaxial angular velocity, and determine the state vector based on the updated unit quaternion; The first observation residual is constructed based on the difference between the triaxial acceleration and the direction of gravity. The second observation residual is constructed based on the difference between the triaxial geomagnetic intensity and the theoretical direction of geomagnetic north. Calculate the Kalman gain based on the first observation residual and the second observation residual, update the state vector based on the Kalman gain, and output the attitude estimation result; The attitude estimation results are used to calculate the attitude and output the attitude data.

4. The human-computer interaction method based on a nine-axis IMU and edge AI according to claim 1, characterized in that, The preprocessing of the attitude data to obtain an attitude data matrix includes: Based on a preset window length, sliding step size, and sampling rate, a sliding window truncation operation is performed on the pose data to generate continuous window data; wherein each window data is a two-dimensional data matrix. The pose data matrix is ​​obtained by normalizing all window data using the Z-Score normalization method.

5. The human-computer interaction method based on a nine-axis IMU and edge AI according to claim 1, characterized in that, The pre-deployed edge AI model includes a convolutional neural network model and a classification decision model. The step of inputting the pose data matrix into the pre-deployed edge AI model and outputting gesture recognition results includes: The pose data matrix is ​​input into the convolutional neural network model to obtain the feature vector; The feature vector is input into the classification decision model to obtain the gesture recognition result; Output the gesture recognition result.

6. The human-computer interaction method based on a nine-axis IMU and edge AI according to claim 5, characterized in that, The convolutional neural network model includes convolutional layers, ReLU activation layers, and pooling layers. The step of inputting the pose data matrix into the convolutional neural network model to obtain feature vectors includes: The temporal dynamic features of the pose data matrix are extracted through the convolutional layer to generate a temporal feature map; The temporal feature map is nonlinearly mapped using the ReLU activation layer to obtain higher-order semantic features; The pooling layer removes redundant information from the higher-order semantic features and outputs a feature vector.

7. The human-computer interaction method based on a nine-axis IMU and edge AI according to claim 5, characterized in that, The step of inputting the feature vector into the classification decision model to obtain the gesture recognition result includes: The radial basis function kernel method maps the input feature vector to a preset high-dimensional space; wherein, the preset high-dimensional space is a gesture category sample space with clear boundaries; Calculate the kernel distance weighted sum of the feature vectors and support vectors; Based on the distance-weighted sum of the kernel function, the gesture recognition result corresponding to the feature vector is output.

8. A human-computer interaction system based on a nine-axis IMU and edge AI, characterized in that, include: A nine-axis IMU sensor is used to acquire raw IMU data based on user actions; The main control module includes a filtering submodule, a preprocessing submodule, an edge AI submodule, and a communication submodule. The filtering submodule is used to perform attitude estimation and attitude calculation on the original IMU data based on the extended Kalman filter algorithm to obtain attitude data corresponding to the original IMU data. The preprocessing submodule is used to preprocess the attitude data to obtain an attitude data matrix; The edge AI submodule is used to input the pose data matrix into a pre-deployed edge AI model and output gesture recognition results; The communication submodule is used to generate operation instructions corresponding to the gesture recognition results and send the operation instructions to the device to be controlled.

9. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a human-computer interaction program based on a nine-axis IMU and edge AI, which, when executed by the processor, is used to implement the operation of the human-computer interaction method based on a nine-axis IMU and edge AI as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a human-computer interaction program based on a nine-axis IMU and edge AI, which, when executed by a processor, is used to implement the operation of the human-computer interaction method based on a nine-axis IMU and edge AI as described in any one of claims 1-7.

Images